The Spin Stops Here!
Decision-Making Under Severe Uncertainty  
Faqs | Help | @ | Contact | home  
voodoostan info-gap decision theory info-gap economics severe uncertainty mighty maximin robust decisions responsible decisions

Reviews of publications on Info-Gap decision theory

Review # 19 (Posted: September 12, 2010)

Reference: Denys Yemshanov, Frank H. Koch, Yakov Ben-Haim, and William D. Smith
Robustness of Risk Maps and Survey Networks to Knowledge Gaps About a New Invasive Pest
Risk Analysis, Vol. 30, No. 2, 261-276, 2010
Abstract In pest risk assessment it is frequently necessary to make management decisions regarding emerging threats under severe uncertainty. Although risk maps provide useful decision support for invasive alien species, they rarely address knowledge gaps associated with the underlying risk model or how they may change the risk estimates. Failure to recognize uncertainty leads to risk-ignorant decisions and miscalculation of expected impacts as well as the costs required to minimize these impacts. Here we use the information gap concept to evaluate the robustness of risk maps to uncertainties in key assumptions about an invading organism. We generate risk maps with a spatial model of invasion that simulates potential entries of an invasive pest via international marine shipments, their spread through a landscape, and establishment on a susceptible host. In particular, we focus on the question of how much uncertainty in risk model assumptions can be tolerated before the risk map loses its value. We outline this approach with an example of a forest pest recently detected in North America, Sirex noctilio Fabricius. The results provide a spatial representation of the robustness of predictions of S. noctilio invasion risk to uncertainty and show major geographic hotspots where the consideration of uncertainty in model parameters may change management decisions about a new invasive pest. We then illustrate how the dependency between the extent of uncertainties and the degree of robustness of a risk map can be used to select a surveillance network design that is most robust to knowledge gaps about the pest.
Keywords Decision theory; info-gap; robustness to uncertainty; Sirex noctilio; survey network.
Acknowledgments The authors extend their gratitude and thanks to Anne Bostelaar and Daniel Sdao (Natural Resources Canada, Canadian Forest Service) and Kurt Riitters (USDA Forest Service, Southern Research Station) for technical support with large-scale numeric simulations. The participation of Denys Yemshanov was supported by Natural Resources Canada, Canadian Forest Service. The participation of Frank Koch was supported by Research Joint Venture Agreements #06-JV-11330146–123 and #08-JV-11330146–078 between the USDA Forest Service, Southern Research Station, Asheville, NC and North Carolina State University.
Scores TUIGF:100%
SNHNSNDN:100%
GIGO:100%

Overview

There is hardly anything new in my critical comments about this article, as most of what needs to be said about it has already been covered in my reviews of other papers. All the same I want to call attention to the fact that this is a typical info-gap article. It is typical in the sense that it continues to propound the same "old" myths about info-gap decision theory that by now have become the accepted wisdom in the info-gap literature.

So, as I have already discussed all the major issues pertaining to these myths in my reviews of other papers (see list), I shall only touch very briefly on these four points:

Relation to the state of the art

The paper exhibits the customary disregard for the state of the art that is typical of info-gap publications. In particular, although the decision-making model proposed in this article is a robust optimization model, not a single reference is made in this article to the thriving field of robust optimization.

And, it goes without saying, that not a single reference is made to the fact that info-gap decision theory has come under sustained severe criticism -- a criticism that is documented in a number of publications.

As a result, the paper gives a thoroughly distorted picture of info-gap decision theory, of the state of the art in robust optimization, and more generally in decision theory, and of info-gap decision theory's place and role in decision-making under severe uncertainty.

More on this in the remaining sections of this short review.

The Maximin saga

The authors' cryptic reference to Wald's(70) 1945 paper is most telling (page 274):
Assessing model uncertainty represents a challenge because alternative models often have different parameter spaces, for example, a relatively simple traveling wave model(56) versus a complex population model.(69) Potentially, this issue can be addressed with the idea of Wald’s worst-case scenario(70) or ensemble analysis.(33,34) The latter approach, however, has been criticized for its inability to identify a "correct" model.(31) In addition, under severe uncertainty it may not be possible to reliably identify a worst case.

Take special note of the fact that the authors do not refer to Wald's Maximin model. Instead, they talk about "Wald’s worst-case scenario". Also note that we are advised that it may not be possible to reliably identify a worst case, presumably to indicate that the application of Wald's Maximin model under these conditions may be unreliable.

Before I take up these points, I pose this question to the authors. If the severity of the uncertainty may impede a reliable identification of a worst-case, how reliable is the estimate (nominal value) that is required by info-gap decision theory as the fulcrum of the info-gap analysis, under conditions of severe uncertainty? Indeed how reliable are the results yielded by such an analysis?

It is therefore important to point out to the readers that these claims about Wald's Maximin model are groundless and misleading. To begin with, the authors neglect to mention that .... info-gap's robustness model is in fact a simple instance of Wald's Maximin model, of course the most important robustness model in decision theory.

Secondly, I call the readers attention to the fact that the theorems stating that info-gap's robustness model is a simple instance of Wald's famous Maximin model are highly constructive. Namely, they spell out in detail how to obtain that particular instance expressing info-gap's robustness model from the generic Maximin model.

To see for yourself, click on the show/hide for the mobile version of the theorem and its proof.

So, what you can immediately gather from this theorem is that as a simple instance of Wald's maximin model, there is nothing that info-gap's robustness model can do that Wald's maximin model cannot do.

Furthermore, as a simple instance of the most famous model in decision theory for robust decision making under severe uncertainty, info-gap's robustness model is neither distinct, nor new in the sense of offering a new or different (or what have you) approach to robust decision-making under severe uncertainty.

In fact, the wider implication of this theorem is that as a methodology, info-gap decision theory has little, if anything, to contribute to the state of the art in robust optimization and decision making under severe uncertainty.

Because, as explained in the next section, info-gap's robustness model is not just a simple instance of Wald's Maximin model. It is a simple instance that is known universally as Radius of Stability model. As such a model, it is not designed to handle severe uncertainty. Meaning that info-gap decision theory is utterly unsuitable for robust decision making under severe uncertainty.

The Radius of Stability perspective.

This theorem and its proof show that info-gap's robustness model is not just a simple instance of Wald's Maximin model, but that this model is an instance of what is universally known as ... Radius of Stability model. The proof itself is elementary.

To see for yourself click here to show/hide the mobile version of the theorem and its proof.

The implications are of course grave.

Because, what this theorem does is to reinforce the claim that what we have here is not just a case of a simple instance of a well established model masquerading as a new and radically different model of robustness. But, a case of a reinvention of a square wheel.

To explain.

Radius of stability models are designed to handle small perturbations in a given nominal value of the parameter of interest. This means that by definition they have an inherently local orientation. They model/analyze yield only local robustness in the neighborhood of the given nominal value of the parameter. The inference therefore is that they are utterly unsuitable for the treatment of situations subject to severe uncertainty, where the uncertainty space is vast, the nominal value is a poor estimate of the true value of the parameter, and the uncertainty model is likelihood-free.

But the whole point is that these are precisely the type of situations that info-gap decision theory is claimed to take on!

The implication is then that info-gap decision theory's prescription for the treatment of severe uncertainty amounts to a methodic misapplication of the Radius of Stability model.

So, by advocating the use of the info-gap methodology for problems of the type described in the abstract, the authors exhibit a total lack of appreciation of the difference between local and global robustness, hence a total lack of appreciation of how unsuitable the info-gap methodology is for this purpose.

Relation to Voodoo decision theories

And to give you a more vivid illustration of why the info-gap methodology is not only utterly unsuitable, but in fact the wrong recipe for robust decision making under severe uncertainty, consider the picture below. The info-gap methodology is claimed to be particularly suitable for situations where the uncertainty space is unbounded. So, as indicated by the authors, the uncertainty space they consider is unbounded.

But what does the author's claim effectively amount to? The authors effectively claim that an analysis of an infinitesimally small section of a vast (unbounded) uncertainty space in the neighborhood of a poor estimate of the true value of the parameter of interest constitutes a sound approach to the treatment of severe uncertainty.

The picture is this:

No Man's LandûNo Man's Land
-∞ <-------------- Complete region of uncertainty under consideration --------------> ∞

where û denotes the estimate of the parameter of interest, the black area represents the complete region of uncertainty under consideration, the red area around û represents the region of uncertainty that actually affects the results generated by info-gap's robustness analysis, and the vast No Man's Land represents that part of the complete region of uncertainty that has no impact whatsoever on the results generated by info-gap's robustness model.

If this does not amount to voodoo decision-making, what does?

Fooled by (local) robustness

Since the authors give not the slightest indication that info-gap decision theory is based on a model of local robustness, it is important to point out that statements such as this (page 266)

We used the info-gap robustness function as a decision tool to select the pest survey network that is most immune to uncertainties about the pest.

are grossly misleading.

In fact, not only are they misleading, they may even be construed as irresponsible because of the unfounded (indeed, false) sense of confidence they may give about the results generated by the proposed robustness analysis. The point is that the results yielded by info-gap's robustness analysis have only local significance. They have no global significance whatsoever. To appreciate this point think about the distinction between a local optimum and a global optimum, local and global weather, local and global economy, local and global anesthetic, etc.

The distinction between local and global robustness is of the first importance because a decision that is locally robust/fragile in the neighborhood of a given estimate is not necessarily globally robust/fragile against the severe uncertainty in the true value of the parameter of interest, and vice versa. This distinction is particularly important in cases where the uncertainty model is likelihood-free, the estimate is poor and the uncertainty space is vast. The inference therefore is that the above statement must be read as indicating the following: "We used the info-gap robustness function as a decision tool to select the pest survey network that is most immune to uncertainties (in the neighborhood of the (poor) estimate) about the pest".

Constructing examples illustrating this point is "a walk in the park". Click here show/hide for such an example.

It is important that the readers realize that Info-gap decision theory has the dubious distinction of being the only decision theory that proposes the use of a local, radius of stability model, to analyze and manage severe uncertainly characterized by

Decision making under severe uncertainty is an extremely challenging task - one that cannot be met by means of rhetoric. It most definitely cannot be accomplished by the use of a radius of stability model of the type proposed in this article.

Readers who are not familiar with the topic "decision-making under severe uncertainty," are urged to read my recent critique of info-gap decision theory (see Sniedovich (2010) below), my FAQs about info-gap decision theory and my reviews of other papers on info-gap decision theory (see list below).

Remark:

I should stress that the statements made above should not be interpreted as suggesting that there are no cases where local robustness implies global robustness. There are certainty cases where this is true. In particular, there are many trivial cases where by inspection one can immediately verify that local robustness can be used to deduce the global robustness of a system. For a concrete example, take a look at the problem featured in Review 13 and the problem featured in Review 16.

My warning is that such implications cannot be taken for granted. They must be established on a case by case basis, the default assumption being that a priori there is no reason to believe that this is the case in general.

For example, consider the opening paragraph of the article entitled Global stability of population models (Cull 1981, p.47):
It is well known that local and global stability are not equivalent and that it is much easier to test for local stability than for global stability (see LaSalle, 1976). The point of this paper is to show that for the usual one-dimensional population models, local and global stability are equivalent.
As I indicate in my recent article (Sniedovich 2010, JRF, 11(3), p. 280):
As it is eminently clear that info-gap decision theory does not, indeed cannot -- methodologically speaking -- provide such evidence or proof in support of its generic robustness model, the duty to find a way out of this impasse is left to the users of this theory, who must do this on a case-by-case basis. Thus, it is not surprising that, once made aware of this situation, some analysts have attempted to provide some sort of evidence or proof to justify their use of info-gap's robustness model.

See for example Review 6.

Other Reviews

  1. Ben-Haim (2001, 2006): Info-Gap Decision Theory: decisions under severe uncertainty.

  2. Regan et al (2005): Robust decision-making under severe uncertainty for conservation management.

  3. Moilanen et al (2006): Planning for robust reserve networks using uncertainty analysis.

  4. Burgman (2008): Shakespeare, Wald and decision making under severe uncertainty.

  5. Ben-Haim and Demertzis (2008): Confidence in monetary policy.

  6. Hall and Harvey (2009): Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.

  7. Ben-Haim (2009): Info-gap forecasting and the advantage of sub-optimal models.

  8. Yokomizo et al (2009): Managing the impact of invasive species: the value of knowing the density-impact curve.

  9. Davidovitch et al (2009): Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island.

  10. Ben-Haim et al (2009): Do we know how to set decision thresholds for diabetes?

  11. Beresford and Thompson (2009): An info-gap approach to managing portfolios of assets with uncertain returns

  12. Ben-Haim, Dacso, Carrasco, and Rajan (2009): Heterogeneous uncertainties in cholesterol management

  13. Rout, Thompson, and McCarthy (2009): Robust decisions for declaring eradication of invasive species

  14. Ben-Haim (2010): Info-Gap Economics: An Operational Introduction

  15. Hine and Hall (2010): Information gap analysis of flood model uncertainties and regional frequency analysis

  16. Ben-Haim (2010): Interpreting Null Results from Measurements with Uncertain Correlations: An Info-Gap Approach

  17. Wintle et al. (2010): Allocating monitoring effort in the face of unknown unknowns

  18. Moffitt et al. (2010): Securing the Border from Invasives: Robust Inspections under Severe Uncertainty

  19. Yemshanov et al. (2010): Robustness of Risk Maps and Survey Networks to Knowledge Gaps About a New Invasive Pest

  20. Davidovitch and Ben-Haim (2010): Robust satisficing voting: why are uncertain voters biased towards sincerity?

  21. Schwartz et al. (2010): What Makes a Good Decision? Robust Satisficing as a Normative Standard of Rational Decision Making

  22. Arkadeb Ghosal et al. (2010): Computing Robustness of FlexRay Schedules to Uncertainties in Design Parameters

  23. Hemez et al. (2002): Info-gap robustness for the correlation of tests and simulations of a non-linear transient

  24. Hemez et al. (2003): Applying information-gap reasoning to the predictive accuracy assessment of transient dynamics simulations

  25. Hemez, F.M. and Ben-Haim, Y. (2004): Info-gap robustness for the correlation of tests and simulations of a non-linear transient

  26. Ben-Haim, Y. (2007): Frequently asked questions about info-gap decision theory

  27. Sprenger, J. (2011): The Precautionary Approach and the Role of Scientists in Environmental Decision-Making

  28. Sprenger, J. (2011): Precaution with the Precautionary Principle: How does it help in making decisions

  29. Hall et al. (2011): Robust climate policies under uncertainty: A comparison of Info-­-Gap and RDM methods

  30. Ben-Haim and Cogan (2011) : Linear bounds on an uncertain non-linear oscillator: an info-gap approach

  31. Van der Burg and Tyre (2011) : Integrating info-gap decision theory with robust population management: a case study using the Mountain Plover

  32. Hildebrandt and Knoke (2011) : Investment decisions under uncertainty --- A methodological review on forest science studies.

  33. Wintle et al. (2011) : Ecological-economic optimization of biodiversity conservation under climate change.

  34. Ranger et al. (2011) : Adaptation in the UK: a decision-making process.

Recent Articles, Working Papers, Notes

Also, see my complete list of articles
    Moshe's new book!
  • Sniedovich, M. (2012) Fooled by local robustness, Risk Analysis, in press.

  • Sniedovich, M. (2012) Black swans, new Nostradamuses, voodoo decision theories and the science of decision-making in the face of severe uncertainty, International Transactions in Operational Research, in press.

  • Sniedovich, M. (2011) A classic decision theoretic perspective on worst-case analysis, Applications of Mathematics, 56(5), 499-509.

  • Sniedovich, M. (2011) Dynamic programming: introductory concepts, in Wiley Encyclopedia of Operations Research and Management Science (EORMS), Wiley.

  • Caserta, M., Voss, S., Sniedovich, M. (2011) Applying the corridor method to a blocks relocation problem, OR Spectrum, 33(4), 815-929, 2011.

  • Sniedovich, M. (2011) Dynamic Programming: Foundations and Principles, Second Edition, Taylor & Francis.

  • Sniedovich, M. (2010) A bird's view of Info-Gap decision theory, Journal of Risk Finance, 11(3), 268-283.

  • Sniedovich M. (2009) Modeling of robustness against severe uncertainty, pp. 33- 42, Proceedings of the 10th International Symposium on Operational Research, SOR'09, Nova Gorica, Slovenia, September 23-25, 2009.

  • Sniedovich M. (2009) A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 2071-2079, Taylor and Francis Group, London.
    • .
  • Sniedovich M. (2009) A Classical Decision Theoretic Perspective on Worst-Case Analysis, Working Paper No. MS-03-09, Department of Mathematics and Statistics, The University of Melbourne.(PDF File)

  • Caserta, M., Voss, S., Sniedovich, M. (2008) The corridor method - A general solution concept with application to the blocks relocation problem. In: A. Bruzzone, F. Longo, Y. Merkuriev, G. Mirabelli and M.A. Piera (eds.), 11th International Workshop on Harbour, Maritime and Multimodal Logistics Modeling and Simulation, DIPTEM, Genova, 89-94.

  • Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)

  • Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)

  • Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08
    This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file )

  • Sniedovich, M. (2008) A call for the reassessment of Info-Gap decision theory, Decision Point, 24, 10.

  • Sniedovich, M. (2008) From Shakespeare to Wald: modeling wors-case analysis in the face of severe uncertainty, Decision Point, 22, 8-9.

  • Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!, Journal of Risk Finance, 9(3), 287-291.

  • Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)
    In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model.

  • Sniedovich. M. (2008) The Mighty Maximin! (PDF File)
    This paper is dedicated to the modeling aspects of Maximin and robust optimization.

  • Sniedovich, M. (2007) The art and science of modeling decision-making under severe uncertainty, Decision Making in Manufacturing and Services, 1-2, 111-136. (PDF File) .

  • Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)
    In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory.

  • My reply (PDF File) to Ben-Haim's response to one of my papers. (April 22, 2007)

    This is an exciting development!

    • Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.

    • Ben-Haim is not familiar with the fundamental concept point estimate. He does not realize that a function can be a point estimate of another function.

      So when you read my papers make sure that you do not misinterpret the notion point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.

      Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!


  • A formal proof that Info-Gap is Wald's Maximin Principle in disguise. (December 31, 2006)
    This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)
    It shows that Info-Gap is not a new theory but rather a simple instance of Wald's famous Maximin Principle dating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context of Game Theory (1928).

  • A proof that Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)
    This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File) .
    It shows that because Info-Gap deploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust.

  • A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)
    This is a very short article entitled The GAP in Info-Gap (PDF File) .
    It is a math-free version of the paper above. Read it if you are allergic to math.

  • A long essay entitled What's Wrong with Info-Gap? An Operations Research Perspective (PDF File) (December 31, 2006).
    This is a paper that I presented at the ASOR Recent Advances in Operations Research (PDF File) mini-conference (December 1, 2006, Melbourne, Australia).

Recent Lectures, Seminars, Presentations

If your organization is promoting Info-Gap, I suggest that you invite me for a seminar at your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use — let alone promote — Info-Gap as a decision-making tool.

Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.

Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.


Last modified: