The Spin Stops Here! Decision-Making Under Severe Uncertainty	Faqs | Help | @ | Contact |
voodoostan info-gap decision theory info-gap economics severe uncertainty mighty maximin robust decisions responsible decisions

---

Contending with uncertainty in conservation management decisions (McCarthy 2014)

---

In this post I respond to comments made about my criticism of info-gap decision theory (IGDT) in the article

• McCarthy, M. 2014. Contending with uncertainty in conservation management decisions
  Annals of the New York Academy of Science, 1332:77-91

Before I turn to its content, I should point out that, to the best of my knowledge, this is the first peer-reviewed article by an Australian scholar and a former follower of IGDT, in which the author explains why he altered his position with regard to IGDT.

The significance of this is discussed in the post: The End of the IGDT Affair (2003-2014) in the Land of the Black Swan.

About the article

The abstract of the article reads as follows:

Efficient conservation management is particularly important because current spending is estimated to be insufficient to conserve the world's biodiversity. However, efficient management is confounded by uncertainty that pervades conservation management decisions. Uncertainties exist in objectives, dynamics of systems, the set of management options available, the influence of these management options, and the constraints on these options. Probabilistic and nonprobabilistic quantitative methods can help contend with these uncertainties. The vast majority of these account for known epistemic uncertainties, with methods optimizing the expected performance or finding solutions that achieve minimum performance requirements. Ignorance and indeterminacy continue to confound environmental management problems.While quantitative methods to account for uncertainty must aid decisions if the underlying models are sufficient approximations of reality, whether such models are sufficiently accurate has not yet been examined.

The article examines two types of methods for dealing with uncertainty: probabilistic methods and nonprobabilistic methods. The latter are classified by the author as follows:

• Maximin
• Minimum [sic]^* regret
• Scenarios
• Info-gap decision theory

^* Should be "minimax regret".

In my response I focus on the article's assessment of IGDT.

My response

Although the article does discuss some aspects of my criticism of IGDT, it does not give a coherent account of this criticism, which inevitably means that it fails to give an adequate account of the fundamental problems afflicting this theory and its "mainstream literature". By "mainstream literature" I mean IGDT's basic texts and the articles written by its founder and its followers.

So, before I take up some of the issues that the article does address, it is important that we consider first the "big picture'' which, regrettably, is not addressed by it. Consider then the following two questions whose clarification is crucial for a proper assessment of IGDT:

• Does the IGDT literature present a correct view of this theory's role and place in the state of the art?
• Does IGDT deliver the "goods" (tools) that, according to its founder, it was designed to provide?

The answers to these important questions are crystal clear:

• IGDT, or rather its "mainstream" literature, grossly misrepresents the role and place of the theory in the state of the art.
• IGDT's robustness model cannot possibly deal properly with the uncertainty that it claims to address, which means that it is incapable of performing the very task for which it was professedly designed.

To explain the first point, the issue here is that while the "mainstream" literature on IGDT (i.e. Ben-Haim 2001, 2006) claims that IGDT is radically different from all current theories of decision under uncertainty, the fact of the matter is that IGDT's robustness model is a reinvention of a simple maximin model that is known universally as radius of stability (circa 1960).

As for the second point, the issue is that the profound incongruity between the inherently local orientation of IGDT's robustness analysis and the inherent likelihood-free nature of its uncertainty model, makes a mockery of IGDT's principal claim that it provides a method for dealing with the type of uncertainty whose characteristics are spelled out below, an uncertainty that IGDT is most definitely justified to proclaim severe.

It is important to note therefore that the article does not spell out these basic characteristics clearly and as a result, fails to give its readers a correct picture of the type of uncertainty that IGDT in fact claims to address.

The point here is that it is essential to have an accurate idea of the uncertainty that IGDT claims to address to be able to appreciate that, the inherently local robustness analysis prescribed by IGDT's robustness model is utterly incapable of taking on an uncertainty of this nature. Hence, to appreciate that this claim in fact renders IGDT a prime example of a voodoo decision theory, especially in cases where the uncertainty is unbounded. It is important to note then that, according to Ben-Haim (2001, 2010), most of the commonly encountered IGDT models are characterized by an unbounded uncertainty.

Those readers who may feel uneasy about the term voodoo, are reminded that its function in the phrase "voodoo decision theory" is precisely identical to its function in the commonly used phrases voodoo science, voodoo economics, voodoo mathematics, voodoo statistics, voodoo accounting, and so on. To see that this label in fact gives an utterly illuminating description of IGDT consider Hayes et al.'s (2013, p. 9; bold face added) following assessment of IGDT (emphasis added):

Plausibility is being evoked within IGDT in an ad hoc manner, and it is incompatible with the theory's core premise, hence any subsequent claims about the wisdom of a particular analysis have no logical foundation. It is therefore difficult to see how they could survive significant scrutiny in real-world problems. In addition, cluttering the discussion of uncertainty analysis techniques with ad hoc methods should be resisted.

Thus, for the purposes of this discussion, readers should feel free to interpret the term voodoo as connoting devoid of a logical foundation.

These points, as well as other problematic aspects of IGDT, are explained in detail in Sniedovich (2014).

Comments

In what follows I focus on the following Statements (McCarthy 2014, pp. 87-88):

While use of info-gap decision theory has increased, so have criticisms. I have published papers using info-gap decision theory,^71-74 but now I agree with critics that it overstates the level of uncertainty that it accommodates. Sniedovich^75-78 is a vocal critic, although some turns of phrase (e.g., referring to "voodoo decision making") and his mathematical treatment might obscure the case in the eyes of many ecologists. Simultaneously, Sniedovich⁷⁸ argues that defenders of info-gap do not address his major criticisms, which essentially dispute the form of uncertainty analyzed by info-gap decision theory.⁷⁰

1. Illusionary gap

The article discusses, separately, the maximin paradigm and IGDT without pointing out that IGDT's robustness model and IGDT's robust-satisficing decision models are in fact maximin models. This is a gross misrepresentation of the basic facts about IGDT. Because, by placing IGDT alongside the maximin paradigm, without making it clear that IGDT's robustness model and IGDT's robust-satisficing decision model are in fact simple maximin models, the article creates the false impression that IGDT and the maximin paradigm are two distinct methods, indeed, that IGDT's robustness model is not a maximin model.

This portrayal of IGDT is not only misleading, it is in fact inexcusable. Because, given that the article itself acknowledges that "info-gap robustness" is in fact a reinvention of the concept "radius of stability" (circa 1960), and given that Sniedovich had demonstrated, time and again, that "radius of stability" model is in fact a simple maximin model, hence that IGDT's robustness model is a simple maximin model, how is it then that the article does not state loud and clear that IGDT's robustness model is a maximin model?!

But more than this, considering that the article acknowledges that "info-gap robustness" is in fact a reinvention of the concept "radius of stability" (circa 1960), it is surprising that it does not discuss what David Fox terms the "illusionary gap" (see my post at http://info-gap.moshe-online.com/illusionary_gap.html). I take it that McCarthy (2014) is aware that Fox's coinage "illusionary gap" refers to Burgman and Regan's (2014) attempt to make a case for IGDT as a method that "fills a gap in ecological applications". Fox's term, as well as his discussion of this matter, were triggered by Sniedovich (2014) which explains in detail that this attempt by Burgman and Regan's (2014) effectively creates a spurious gap in the state of the art designed to be filled with IGDT.

And what all this goes to show is that it is high time that scholars using IGDT came to terms with the fact that IGDT is a reinvention of the wheel and a square one at that, and that they began to acknowledge this fact in their publications about this theory.

2. Ecologists

Sniedovich's^[75-78] mathematical treatment of the flaws in IGDT is hardly more complex than the mathematical treatment of IGDT itself in Ben-Haim^[68-69]. Indeed, Sniedovich^[75-78], by necessity, use the same mathematical models that are used in Ben-Haim^[68-69]. The inference therefore is that ecologists who cannot cope with the technical complexity of Sniedovich's^[75-78] criticism of IGDT are hardly in a position to cope with the complexity of the exposition of IGDT in Ben-Haim^[68-69]. Surely, such ecologists should steer clear of a theory whose technical aspects they are unable to follow.

The fact of the matter is, however, that, the leading IGDT scholars who were actively engaged in the promotion and use of this theory in Australia clearly did not appear to have any difficulties in comprehending the obvious flaws in IGDT identified in Sniedovich^[75-78]. Their decision to continue to promote the use of IGDT in Australia, despite Sniedovich's^[75-78] criticism, was not due to a lack of understanding of Sniedovich's^[75-78] criticism of IGDT.

I do not wish to speculate on this decision as the author of the article is in a much better position to address this interesting aspect of the info-gap affair. It is a pity therefore that the article does not explain why it took so long, about 8 years, for its author to publicly alter his position on IGDT.

3. Level of uncertainty

The problem with IGDT is not, as claimed in the article, that it overstates the level of uncertainty that it accommodates. In fact, IGDT as such does not distinguish between different levels of uncertainty. Furthermore, Sniedovich^[78] does not dispute, as claimed in the article, the form of uncertainty analyzed by IGDT.

Rather, Sniedovich^[75-78] points out the obvious, which is that IGDT's local, "likelihood-free" robustness analysis is incapable of tackling the uncertainty postulated by IGDT. I remind the reader that, according to Ben-Haim^[68-69], the uncertainty that IGDT was designed to deal with is characterized by

• a vast (e.g. unbounded) uncertainty space
• a poor point estimate that can be significantly wrong (sometimes it is just a wild guess)
• a "likelihood-free", "chance-free", "plausibility-free" quantification of uncertainty.

Surely, one need not be a risk analyst to realize that the local robustness analysis prescribed by IGDT's robustness is utterly unsuitable for the treatment of this type of uncertainty.

And one need not be a risk analyst to figure out that in cases where the uncertainty space is unbounded, IGDT's treatment of a "likelihood-free" uncertainty is a classic example of voodoo decision-making.

4. Voodoo

A detailed explanation of why IGDT merits the title "voodoo decision theory par excellence" can be found in Sniedovich^[78]. Scholars who do not appreciate this point cannot possibly appreciate how fundamentally flawed IGDT in fact is, nor can they appreciate Hayes et al.'s^[70] verdict that risk analysis based on this theory has no logical foundation.

The reference in McCarty (2014) to this issue is an indication that its author apparently continues to misjudge/underestimate the fundamental flaws in IGDT.

Because, the picture is this:

Voodoo Decision-Making The large square represents the uncertainty space, namely the set of all possible/plausible values of the uncertainty parameter, and the small yellow square represents the values of the uncertainty parameter that are actually involved in the robustness analysis. The values of the uncertainty parameter in the large black square take no part whatsoever in the robustness analysis. This means that the value of the info-gap robustness yielded by this analysis is determined in total disregard to the values of the uncertainty parameter within the black area. In other words, IGDT's robustness analysis completely ignores the black area. It is important to note that it is repeatedly emphasized in Ben-Haim (2001, 2006, 2010) that, the uncertainty space of IGDT models is typically unbounded. This means that the small yellow square is typicallyinfinitesimally small. In other words: the inference is that IGDT's robustness analysis typically ignores completely most of the possible/plausible values of the uncertainty parameter. Differently put, IGDT's robustness analysis is typically conducted on a minute neighborhood of the uncertainty space. Add to this the fact that, given that the uncertainty is claimed to be likelihood-free, "plausibility-free", and so on, the question arising is: on what grounds does/can one determine the location and size of the yellow square in the big square?! In sum, what logical foundation is there to the proposition to use this type of local robustness analysis as a means for identifying robust decisions given this type of uncertainty? Isn't this type of robustness analysis a voodoo analysis par excellence?

5. Unknown Unknowns

The article refers on page 82 to Wintle et al.^[40]. For the benefit of readers who are not familiar with Wintle et al.^[40], it should be pointed out that the main thesis of Wintle et al. [40] is that IGDT, which, as made vivid by the above picture, prescribes no more than a local robustness analysis, is a suitable tool for dealing with the non-probabilistic uncertainty of no less than Unknown Unknowns and Black Swans.

Surely, it is hard to envisage a more striking example of a (peer-reviewed) proposition outlining voodoo decision making. And yet, there is nothing in McCarthy (2014) to indicate that the author is aware of the ramifications of the proposition to use IGDT to tackle no less than Unknown Unknowns and Black Swans.

6. The maximin connection and local robustness

It should also be pointed out that the article does not mention the fact that some IGDT scholars, including the Father of IGDT, continue to maintain, despite the incontestable proof to the contrary, that IGDT's robustness analysis is not a maximin analysis and that this analysis is not local in nature.

~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ ! ~ !

As a final word.

Nearly nine years ago, in November 2005, a "coffee session" was held at University House where I explained in plain language to a number of IGDT scholars, including the author of the article under review here, some of the obvious flaws in IGDT. Since then I repeatedly pointed out to IGDT scholars in Australia the need to reassess the use and promotion of IGDT in Australia.

The article under consideration is of course a positive development, even if it took its author 8 years or so to reach the inevitable conclusion about IGDT. Still, as indicated above, this is too little, too late and too politically correct.

In view of this, and as we seem to reach the "End of the Affair", it might be a good idea to arrange another "coffee session" to reflect on the IGDT "affair" and to draw the necessary conclusions from it. I can assure IGDT scholars that they might benefit greatly from such an informal session.

Also, in view of the history of this "affair", I suggest that it is important to communicate the assessment of IGDT in this article to Decision Point. See

http://info-gap.moshe-online.com/McCarthy_14.html

and at

http://info-gap.moshe-online.com/end_of_affair.html

Stay tuned for further comments on this article.

Moshe Sniedovich
November 3, 2014
Melbourne, Australia
The Land of the Black Swan

---

Bibliography

[40] = Wintle et al. (2010).
[68] = Ben-Haim (2001).
[69] = Ben-Haim (2006).
[70] = Hayes et al. (2013).
[75] = Sniedovich (2012b).
[76] = Sniedovich 2102).
[77] = Sniedovich (2012a).
[78] = Sniedovich (2014).
[79] = Burgman and Regan (2014).

• Bedow, B. 1977. The ten natural laws of operations analysis. Interfaces 7(3):122.
• Ben-Haim, Y. 2001. Information Gap Decision Theory. Academic Press, Amsterdam, The Netherlands.
• Ben-Haim, Y. 2006. Info-Gap Decision Theory. Academic Press, Amsterdam, The Netherlands.
• Ben-Haim, Y. 2010. Info-gap Economics: An Operational Introduction. Palgrave, New York, USA.
• Ben-Haim, Y. 2010a. Frequently asked questions about info-gap. URL http:// www.technion.ac.il/yakov/IGT/faqs01.pdf .
• Ben-Haim, Y. 2012. Doing Our Best: Optimization and the Management of Risk. Risk Analysis 32(8):1326-1332.
• Ben-Haim, Y. 2012a. Why Risk Analysis is Difficult, and Some Thoughts on How to Proceed. Risk Analysis 32(10):1638-1646.
• Ben-Haim, Y. 2012b. Modeling and design of a Hertzian contact: an info-gap approach. The Journal of Strain Analysis for Engineering Design 47(2):153-162.
• Ben-Haim, Y (2014). Robust satisfying and probability of survival. {\it International Journal of Systems Science 45(1):3-19.
• Ben-Tal, A., Golany, B., and Shtern, S. 2009. Robust multi-echelon multi-period inventory control. European Journal of Operational Research 199:922-935.
• Burgman, M.A., Wintle, B.A., Thompson, C.A., Moilanen, A., Runge, M.C. and Ben-Haim, Y. 2010. Reconciling uncertain costs and benefits in Bayes nets for invasive species management. Risk Analysis 30:277-284.
• Burgman, M.A., Wintle, B.A., Thompson, C.A., Moilanen, A., Runge, M.C. and Ben-Haim, Y. 2008. Reconciling uncertain costs and benefits in Bayes nets for invasive species management. Final Report, Project 0601-0611, Australian Centre of Excellence for Risk Analysis. URL http://www.acera.unimelb.edu.au/materials/endorsed/0601_0611.pdf .
• Burgman, M.A., and Regan, H.M. 2014. Information-gap decision theory fills a gap in ecological applications. Ecological Applications 24(1):227-228.
• Carmel, Y., and Ben-Haim, Y. 2005. Info-gap robust-satisficing model of foraging behavior: do foragers optimize or satisfice? The American Naturalist 166:633-641.
• Fox, D.R. 2008. To IG or not to IG?-- that is the question. Decision Point 24:10-11.
• Fox, D. 2014. Information-gap decision theory creates a gap in ecological applications and then fills it. URL: http://www.environmetrics.net.au/index.php?news&nid=73
• French, S.D. 1988. Decision Theory. Ellis Horwood, NY.
• Halpern, B.S., Regan, H.M., Possingham, H.P. and McCarthy, M.A. 2006. Accounting for uncertainty in marine reserve design. Ecology Letters 9:2-11.
• Hayes, K.R. 2011. Uncertainty and Uncertainty Analysis Methods. Final report for the Australian Centre of Excellence for Risk Analysis (ACERA), CSIRO Division of Mathematics, Informatics and Statistics, Hobart, Australia, 130 pp.
• Hayes, K.R., Barry, S.C., Hosack, G.R., and Peters, G.W. 2013. Severe uncertainty and info-gap decision theory. Methods in Ecology and Evolution 4:601-611.
• Hillson, D. 2007. `Unknown-Unknowns' and risk. Project Management Today 19(12):22-27.
• Hodgson, D.J., and Townley, S. 2004. Linking management changes to population dynamic responses: the transfer function of a projection matrix perturbation. Journal of Applied Ecology 41:1155-1161.
• McCarthy, M. 2014. Contending with uncertainty in conservation management decisions. Annals of the New York Academy of Science, 1332:77-91.
• Milne, W.E., and Reynolds, R.R. 1962. Fifth-order methods for the numerical solution of ordinary differential equations. Journal of the ACM 9(1):64-70.
• Nicholson, E., and Possingham, H.P. 2007. Making conservation decisions under uncertainty for the persistence of multiple species. Ecological Applications 17:251-265.
• Peterson, M. 2009. An Introduction to Decision Theory. Cambridge University Press.
• Rawls, J. 1971. Theory of Justice. Belknap Press, Cambridge, MA.
• Regan, H.H., Ben-Haim, Y., Langford, B., Wilson, W.G., Lundberg, P., Andelman, S.J. and Burgman, M.A. 2005. Robust decision-making under severe uncertainty for conservation management. Ecological Applications 15:1471-1477.
• Resnik, M.D. 1987. Choices: an Introduction to Decision Theory. University of Minnesota Press, Minneapolis, MN.
• Sniedovich, M. 2007. The art and science of modeling decision-making under severe uncertainty. Decision Making in Manufacturing and Services 1(1-2):111-136.
• Sniedovich, M. 2008. From Shakespeare to Wald: modeling worst-case analysis in the face of severe uncertainty. Decision Point 22:8-9.
• Sniedovich, M. 2008a. Wald's Maximin Model: A Treasure in Disguise! Journal of Risk Finance 9(3):278-291.
• Sniedovich M. 2009. A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 2071-2079, Taylor and Francis Group, London, 2009.
• Sniedovich M. 2009a. Modeling of robustness against severe uncertainty, pp. 33- 42, Proceedings of the 10th International Symposium on Operational Research, SOR'09, Nova Gorica, Slovenia, September 23-25, 2009.
• Sniedovich, M. 2010. A bird's view of info-gap decision theory. Journal of Risk Finance 11(3):268-283.
• Sniedovich, M. 2011. Info-gap decision theory: a perspective from the Land of the Black Swan. URL: http://info-gap.moshe-online.com/acera.html
• Sniedovich, M. 2011a. A classic decision theoretic perspective on worst-case analysis, Applications of Mathematics 56(5):499-509, 2011.
• Sniedovich, M. 2012. Fooled by Local Robustness. Risk Analysis 32(10):1630-1637.
• Sniedovich, M. 2012a. Fooled by local robustness: an applied ecology perspective. Ecological Applications 22(5):1421-1427.
• Sniedovich, M. 2012b. Black Swans, new Nostradamuses, voodoo decision theories and the science of decision-making in the face of severe uncertainty. International Transactions in Operational Research, 19:253-281.
• Sniedovich, M. 2013. From local robustness to irresponsible decision-making, Proceedings of the BALCOR 2013 Conference, September 7-9, 2013, Belgrade, Serbia, pp. 695-704.
• Sniedovich, M. 2013a. The many faces of worst-case analysis. Proceedings of the SYM-OP-IS 2013 Conference, September 9-11 2013, Belgrade, Serbia, pp. 887-882.
• Sniedovich, M. 2014. The elephant in the rhetoric on info-gap decision theory. Ecological Applications 24(1):229-233.
• Wald, A. 1939. Contributions to the theory of statistical estimation and testing hypotheses. Annals of Mathematical Statistics 10(4):299-326.
• Wald, A. 1945. Statistical decision functions which minimize the maximum risk. The Annals of Mathematics 46(2):265-280.
• Wald, A. 1947. An essentially complete class of admissible decision functions. The Annals of Mathematical Statistics 18:549-555.
• Wald, A. 1950. Statistical Decision Functions. John Wiley, NY.
• Wilf, H.S. 1960. Maximally stable numerical integration. Journal of the Society for Industrial and Applied Mathematics 8(3):537-540.
• Wintle, B.A. 2010. On the look out for black swans: monitoring for unknown unknowns. Decision Point 43:4-7.
• Wintle, B.A., Runge, M.C., and Bekessy, S.A. 2010. Allocating monitoring effort in the face of unknown unknowns. Ecology Letters 13(11):1325-1337.
• Yokomizo, H., Coutts, S.R., and Possingham, H.P. 2014. Decision science for effective management of populations subject to stochasticity and imperfect knowledge. Population Ecology 56:41-53.
• Zlobec, S. 1987. Survey of input optimization. Optimization 18:309-348.

---

Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.

Last modified: