Reviews of publications on Info-Gap decision theory
Review # 2 (Posted: April 11, 2009; Last update: November 28, 2011)
Reference: HELEN M. REGAN, YAKOV BEN-HAIM, BILL LANGFORD, WILLIAM G. WILSON, PER LUNDBERG, SANDY J. ANDELMAN, AND MARK A. BURGMAN
Robust decision-making under severe uncertainty for conservation management.
Ecological Applications, 15(4), 1471-1477, 2005.
Abstract In conservation biology it is necessary to make management decisions for endangered and threatened species under severe uncertainty. Failure to acknowledge and treat uncertainty can lead to poor decisions. To illustrate the importance of considering uncertainty, we reanalyze a decision problem for the Sumatran rhino, Dicerorhinus sumatrensis, using information-gap theory to propagate uncertainties and to rank management options. Rather than requiring information about the extent of parameter uncertainty at the outset, information-gap theory addresses the question of how much uncertainty can be tolerated before our decision would change. It assesses the robustness of decisions in the face of severe uncertainty. We show that different management decisions may result when uncertainty in utilities and probabilities are considered in decision-making problems. We highlight the importance of a full assessment of uncertainty in conservation management decisions to avoid, as much as possible, undesirable outcomes. Scores TUIGF:100%
Acknowledgement: We thank Dan Berleant, Mark Colyvan, Martin Drechsler, John Harwood, Tom Hobbs, and an anonymous reviewer for useful discussions and comments that assisted in the preparation of this paper. This work was conducted as part of the Setting priorities and making decisions for conservation risk management Working Group supported by the National Center for Ecological Analysis and Synthesis, a Center funded by NSF (Grant #DEB-0072909), the University of California, and the Santa Barbara campus.
This is a relatively old paper. Still, it is on my list because, to the best of my knowledge, it is the first publication discussing the application of Info-Gap in applied ecology. So, it may shed some light on how the uncritical use of Info-Gap decision theory evolved in this discipline.
The rather high SNHNSNDN score reflects the authors' distorted portrayal of the tools provided by classical decision theory for the treatment of severe uncertainty. Consider this (page 1471, emphasis is mine):For decision-making under uncertainty, the usual procedure is to assign probabilities to each of the relevant states and utilities to each of the outcomes. The approach usually taken is to maximize expected utility.
Clearly, the implication is that classical decision theory has no other means to offer for the treatment of "unusual" non-probabilistic cases. Or, that using other procedures for this purpose is not common practice in classical decision theory, or something to this effect. The inference, therefore, is that given classical decision theory's manifest shortcomings, one turns to Info-Gap decision theory to fill in the gap!
But the point is that other than making this sweeping statement the authors do not bother to show this to be the case. This is unfortunate because a quick look at one of the main references cited by the authors, namely Resnik (1987), reveals that Chapter 2 is entitled Decisions Under Ignorance. It is divided into the following sections:
Chapter 2: Decisions Under Ignorance
- 2-1: Preference ordering
- 2-2: The Maximin Rule
- 2-3: The Optimism-Pessimism Rule
- 2-4: The Principle of Insufficient Reason
- 2-5: Too many Rules?
- 2-6: An Application in Social Philosophy:
Rawls vs Harsanyi
The probabilistic methods of classical decision theory are discussed in Chapter 3: Decisions Under Risk: Probability.
Apparently, the authors did not realize that relevant to their discussion in Resnik (1987) is not Chapter 3: Decisions Under Risk: Probability, but rather Chapter 2: Decisions Under Ignorance!?
For, had they studied this chapter, namely Chapter 2, they would have realized that Info-Gap's robustness model is simply a Maximin model.
Then, on page 1473 we read this (emphasis is mine):The power and novelty of the info-gap approach is in the ability to explore the sensitivity of the decision to a wide range of different types of parameter, functional, and structural errors and uncertainties simultaneously, given that we do not know the extent of uncertainty in the system at the outset.
Of course, the precise opposite is the case. The purported "power and novelty" attributed to Info-Gap decision theory are in fact its two major failings.
- Firstly, the robustness model is not novel: it is a simple Maximin model (circa 1940).
- Secondly, Info-Gap's purported ability to "cope" with an unknown level of uncertainty in effect turns out to be a blatant disregard of the severity of the uncertainty under consideration. For this purported special power of Info-gap in effect amounts to a fundamentally flawed "procedure": Instead of evaluating the robustness of decisions on the basis of their performance relative to the entire stipulated region of uncertainty, Info-Gap examines the immediate neighborhood of the estimate first, to then explore away from this estimate, no further than allowed by the performance requirement. The exploration further away from the estimate terminates once the performance requirement is violated.
Conceptually, it is instructive to think about Info-Gap's definition of robustness as a play in three acts:
- Act I: Initialization:
Position yourself at the estimate, û. Equivalently: set α=0.
- Act II: Movement:
Slowly move away from the estimate. Equivalently: slowly increase the value of α.
- Act III: Termination:
Stop immediately when any further movement away from the estimate will violate the performance requirement for some u in the increased region of uncertainty.
Equivalently: stop when α=α* where α* is such that for any ε>0 there is a u∈U(α*+ε,û) such that r(d,u)< r*.
Note that in this context α* represents the robustness of decision d, namely α*=α(d,û).
Here is a schematic representation of the situation:
û <-------- U --- Complete Region of Uncertainty --- U -------->
The black area represents the complete region of uncertainty U and the white dot represents the estimate û of the parameter of interest.
Now, with this representation in front of us, I should call attention to the following: No assumption whatsoever must be made as to whether any particular value of u is more/less likely than any other value of this parameter. This is totally in line with the fact that Info-Gap decision theory not only makes no assumptions about "likelihood", it bans any talk of likelihood.
Furthermore, given that, as emphatically pointed out in this paper, the uncertainty is severe, we have not the slightest clue which value of u is the true value.
This means that to determine the robustness of a decision it is essential to establish how well the decision performs relative to the entire region of uncertainty U, or a proper approximation thereof.
To see then what Info-Gap's purported "powerful" approach accomplishes, consider the following schematic representation of the results generated by Info-Gap's robustness analysis for decision d. The red area around the estimate, denoted by U(α(d,û),û), represents the largest safe region around the estimate as determined by Info-Gap's robustness model.
No Man's Land û No Man's Land
Take special note of Info-Gap's No Man's Land. This is the subset of the complete region of uncertainty that Info-Gap's robustness model does not explore. Roughly, this is the collection of points in U that are not in the safe region U(α(d,û),û) around the estimate û.
This schematic representation vividly illustrates why I regard Info-Gap decision theory as a classic example of voodoo decision-making: rather than tackling the severity of the uncertainty under consideration, one is instructed to ignore it altogether.
In my lectures/seminars I often describe this fundamental flaw in Info-Gap decision theory as a "treasure hunt":
- The island represents the complete region of uncertainty under consideration (the region where the treasure is located).
- The tiny black dot represents the estimate of the parameter of interest (estimate of the location of the treasure).
- The large white circle represents the region of uncertainty pertaining to info-gap's robustness analysis.
- The small white square represents the true (unknown) value of the parameter of interest (true location of the treasure).
So, basing our search plan on Info-Gap Decision Theory, we may zero in on the neighborhood of downtown Melbourne, while for all we know, the true location of the treasure may well be in the Middle of the Simpson desert, or perhaps just north of Brisbane.
Note that in this picture the estimate is placed at a considerable distance from the true value of the parameter of interest. This, of course has a point and purpose. It depicts the contention (Ben-Haim 2006, pp. 280-281 ) that under severe uncertainty the estimate is a poor indication of the true value of the parameter and is likely to be substantially wrong.
I have been accused by Info-Gap scholars for exaggerating the size of the No Man's Land relative to the size of the "safe" area around the estimate.
So let me explain, yet again, why — from a methodological point of view — the No Man's Land must be depicted as much larger than the "safe" area, in the schematic representations of the working of Info-Gap's robustness model.
According to Ben-Haim (2006, p. 210, color is mine):Most of the commonly encountered info-gap models are unbounded.
The inference to be drawn then from this statement is that in most of the commonly encountered info-gap models, the "safe" area around the estimate is infinitely minuscule — in comparison to the complete region of uncertainty. Consequently, not only is it the case that my sketches do not exaggerate the flaw in Info-Gap decision theory. This flaw is so severe that it cannot possibly be exaggerated: the complete region of uncertainty is unbounded yet the robustness analysis is a priori confined to the neighborhood of a poor estimate that is likely to be substantially wrong.
More on this can be found in FAQ # 26 and FAQ # 71.
And last but not least, no reference whatsoever is made in the paper to the thriving area of optimization theory that deals specifically with robust decisions, namely Robust Optimization. For those who are unfamiliar with this field of optimization theory I should point out that Robust Optimization concerns itself with -- among other things -- non-probabilistic methods for robust decision-making under severe uncertainty. Not surprisingly, Wald's Maximin paradigm is used extensively in this field.
- Ben-Haim (2001, 2006): Info-Gap Decision Theory: decisions under severe uncertainty.
- Regan et al (2005): Robust decision-making under severe uncertainty for conservation management.
- Moilanen et al (2006): Planning for robust reserve networks using uncertainty analysis.
- Burgman (2008): Shakespeare, Wald and decision making under severe uncertainty.
- Ben-Haim and Demertzis (2008): Confidence in monetary policy.
- Hall and Harvey (2009): Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.
- Ben-Haim (2009): Info-gap forecasting and the advantage of sub-optimal models.
- Yokomizo et al (2009): Managing the impact of invasive species: the value of knowing the density-impact curve.
- Davidovitch et al (2009): Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island.
- Ben-Haim et al (2009): Do we know how to set decision thresholds for diabetes?
- Beresford and Thompson (2009): An info-gap approach to managing portfolios of assets with uncertain returns
- Ben-Haim, Dacso, Carrasco, and Rajan (2009): Heterogeneous uncertainties in cholesterol management
- Rout, Thompson, and McCarthy (2009): Robust decisions for declaring eradication of invasive species
- Ben-Haim (2010): Info-Gap Economics: An Operational Introduction
- Hine and Hall (2010): Information gap analysis of flood model uncertainties and regional frequency analysis
- Ben-Haim (2010): Interpreting Null Results from Measurements with Uncertain Correlations: An Info-Gap Approach
- Wintle et al. (2010): Allocating monitoring effort in the face of unknown unknowns
- Moffitt et al. (2010): Securing the Border from Invasives: Robust Inspections under Severe Uncertainty
- Yemshanov et al. (2010): Robustness of Risk Maps and Survey Networks to Knowledge Gaps About a New Invasive Pest
- Davidovitch and Ben-Haim (2010): Robust satisficing voting: why are uncertain voters biased towards sincerity?
- Schwartz et al. (2010): What Makes a Good Decision? Robust Satisficing as a Normative Standard of Rational Decision Making
- Arkadeb Ghosal et al. (2010): Computing Robustness of FlexRay Schedules to Uncertainties in Design Parameters
- Hemez et al. (2002): Info-gap robustness for the correlation of tests and simulations of a non-linear transient
- Hemez et al. (2003): Applying information-gap reasoning to the predictive accuracy assessment of transient dynamics simulations
- Hemez, F.M. and Ben-Haim, Y. (2004): Info-gap robustness for the correlation of tests and simulations of a non-linear transient
- Ben-Haim, Y. (2007): Frequently asked questions about info-gap decision theory
- Sprenger, J. (2011): The Precautionary Approach and the Role of Scientists in Environmental Decision-Making
- Sprenger, J. (2011): Precaution with the Precautionary Principle: How does it help in making decisions
- Hall et al. (2011): Robust climate policies under uncertainty: A comparison of Info--Gap and RDM methods
- Ben-Haim and Cogan (2011) : Linear bounds on an uncertain non-linear oscillator: an info-gap approach
- Van der Burg and Tyre (2011) : Integrating info-gap decision theory with robust population management: a case study using the Mountain Plover
- Hildebrandt and Knoke (2011) : Investment decisions under uncertainty --- A methodological review on forest science studies.
- Wintle et al. (2011) : Ecological-economic optimization of biodiversity conservation under climate change.
- Ranger et al. (2011) : Adaptation in the UK: a decision-making process.
Recent Articles, Working Papers, Notes
Also, see my complete list of articles
Moshe's new book!
- Sniedovich, M. (2012) Fooled by local robustness, Risk Analysis, in press.
- Sniedovich, M. (2012) Black swans, new Nostradamuses, voodoo decision theories and the science of decision-making in the face of severe uncertainty, International Transactions in Operational Research, in press.
- Sniedovich, M. (2011) A classic decision theoretic perspective on worst-case analysis, Applications of Mathematics, 56(5), 499-509.
- Sniedovich, M. (2011) Dynamic programming: introductory concepts, in Wiley Encyclopedia of Operations Research and Management Science (EORMS), Wiley.
- Caserta, M., Voss, S., Sniedovich, M. (2011) Applying the corridor method to a blocks relocation problem, OR Spectrum, 33(4), 815-929, 2011.
- Sniedovich, M. (2011) Dynamic Programming: Foundations and Principles, Second Edition, Taylor & Francis.
- Sniedovich, M. (2010) A bird's view of Info-Gap decision theory, Journal of Risk Finance, 11(3), 268-283.
- Sniedovich M. (2009) Modeling of robustness against severe uncertainty, pp. 33- 42, Proceedings of the 10th International Symposium on Operational Research, SOR'09, Nova Gorica, Slovenia, September 23-25, 2009.
- Sniedovich M. (2009) A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 2071-2079, Taylor and Francis Group, London..
- Sniedovich M. (2009) A Classical Decision Theoretic Perspective on Worst-Case Analysis, Working Paper No. MS-03-09, Department of Mathematics and Statistics, The University of Melbourne.(PDF File)
- Caserta, M., Voss, S., Sniedovich, M. (2008) The corridor method - A general solution concept with application to the blocks relocation problem. In: A. Bruzzone, F. Longo, Y. Merkuriev, G. Mirabelli and M.A. Piera (eds.), 11th International Workshop on Harbour, Maritime and Multimodal Logistics Modeling and Simulation, DIPTEM, Genova, 89-94.
- Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)
- Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)
- Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08
This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file )
- Sniedovich, M. (2008) A call for the reassessment of Info-Gap decision theory, Decision Point, 24, 10.
- Sniedovich, M. (2008) From Shakespeare to Wald: modeling wors-case analysis in the face of severe uncertainty, Decision Point, 22, 8-9.
- Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!, Journal of Risk Finance, 9(3), 287-291.
- Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)
In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model.
- Sniedovich. M. (2008) The Mighty Maximin! (PDF File)
This paper is dedicated to the modeling aspects of Maximin and robust optimization.
- Sniedovich, M. (2007) The art and science of modeling decision-making under severe uncertainty, Decision Making in Manufacturing and Services, 1-2, 111-136. (PDF File) .
- Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)
In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory.
- My reply (PDF File) to Ben-Haim's response to one of my papers. (April 22, 2007)
This is an exciting development!
- Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.
- Ben-Haim is not familiar with the fundamental concept point estimate. He does not realize that a function can be a point estimate of another function.
So when you read my papers make sure that you do not misinterpret the notion point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.
Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!
- A formal proof that Info-Gap is Wald's Maximin Principle in disguise. (December 31, 2006)
This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)
It shows that Info-Gap is not a new theory but rather a simple instance of Wald's famous Maximin Principle dating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context of Game Theory (1928).
- A proof that Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)
This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File) .
It shows that because Info-Gap deploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust.
- A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)
This is a very short article entitled The GAP in Info-Gap (PDF File) .
It is a math-free version of the paper above. Read it if you are allergic to math.
- A long essay entitled What's Wrong with Info-Gap? An Operations Research Perspective (PDF File) (December 31, 2006).
This is a paper that I presented at the ASOR Recent Advances in Operations Research (PDF File) mini-conference (December 1, 2006, Melbourne, Australia).
Recent Lectures, Seminars, Presentations
If your organization is promoting Info-Gap, I suggest that you invite me for a seminar at your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use — let alone promote — Info-Gap as a decision-making tool.
Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.
ASOR Recent Advances, 2011, Melbourne, Australia, November 16 2011. Presentation: The Power of the (peer-reviewed) Word. (PDF file).
- Alex Rubinov Memorial Lecture The Art, Science, and Joy of (mathematical) Decision-Making, November 7, 2011, The University of Ballarat. (PDF file).
- Black Swans, Modern Nostradamuses, Voodoo Decision Theories, and the Science of Decision-Making in the Face of Severe Uncertainty (PDF File) .
(Invited tutorial, ALIO/INFORMS Conference, Buenos Aires, Argentina, July 6-9, 2010).
- A Critique of Info-Gap Decision theory: From Voodoo Decision-Making to Voodoo Economics(PDF File) .
(Recent Advances in OR, RMIT, Melbourne, Australia, November 25, 2009)
- Robust decision-making in the face of severe uncertainty(PDF File) .
(GRIPS, Tokyo, Japan, October 16, 2009)
- Decision-making in the face of severe uncertainty(PDF File) .
(KORDS'09 Conference, Vilnius, Lithuania, September 30 -- OCtober 3, 2009)
- Modeling robustness against severe uncertainty (PDF File) .
(SOR'09 Conference, Nova Gorica, Slovenia, September 23-25, 2009)
- How do you recognize a Voodoo decision theory?(PDF File) .
(School of Mathematical and Geospatial Sciences, RMIT, June 26, 2009).
- Black Swans, Modern Nostradamuses, Voodoo Decision Theories, Info-Gaps, and the Science of Decision-Making in the Face of Severe Uncertainty (PDF File) .
(Department of Econometrics and Business Statistics, Monash University, May 8, 2009).
- The Rise and Rise of Voodoo Decision Theory.
ASOR Recent Advances, Deakin University, November 26, 2008. This presentation was based on the pages on my website (voodoo.moshe-online.com).
- Responsible Decision-Making in the face of Severe Uncertainty (PDF File) .
(Singapore Management University, Singapore, September 29, 2008)
- A Critique of Info-Gap's Robustness Model (PDF File) .
(ESREL/SRA 2008 Conference, Valencia, Spain, September 22-25, 2008)
- Robust Decision-Making in the Face of Severe Uncertainty (PDF File) .
(Technion, Haifa, Israel, September 15, 2008)
- The Art and Science of Robust Decision-Making (PDF File) .
(AIRO 2008 Conference, Ischia, Italy, September 8-11, 2008 )
- The Fundamental Flaws in Info-Gap Decision Theory (PDF File) .
(CSIRO, Canberra, July 9, 2008 )
- Responsible Decision-Making in the Face of Severe Uncertainty (PDF File) .
(OR Conference, ADFA, Canberra, July 7-8, 2008 )
- Responsible Decision-Making in the Face of Severe Uncertainty (PDF File) .
(University of Sydney Seminar, May 16, 2008 )
- Decision-Making Under Severe Uncertainty: An Australian, Operational Research Perspective (PDF File) .
(ASOR National Conference, Melbourne, December 3-5, 2007 )
- A Critique of Info-Gap (PDF File) .
(SRA 2007 Conference, Hobart, August 20, 2007)
- What exactly is wrong with Info-Gap? A Decision Theoretic Perspective (PDF File) .
(MS Colloquium, University of Melbourne, August 1, 2007)
- A Formal Look at Info-Gap Theory (PDF File) .
(ORSUM Seminar , University of Melbourne, May 21, 2007)
- The Art and Science of Decision-Making Under Severe Uncertainty (PDF File) .
(ACERA seminar, University of Melbourne, May 4, 2007)
- What exactly is Info-Gap? An OR perspective. (PDF File)
ASOR Recent Advances in Operations Research mini-conference (December 1, 2006, Melbourne, Australia).
Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.