The Spin Stops Here!
Decision-Making Under Severe Uncertainty  
Faqs | Help | @ | Contact | home  
voodoostan info-gap decision theory info-gap economics severe uncertainty mighty maximin robust decisions responsible decisions


Reviews of publications on Info-Gap decision theory

Review # 13 (Posted: February 19, 2010; Last update: February 25, 2010)

Reference: Tracy M. Rout, Colin J. Thompson, and Michael A. McCarthy
Robust decisions for declaring eradication of invasive species
Journal of Applied Ecology 46, 782–786, 2009.
Summary

1. Invasive species threaten biodiversity, and their eradication is desirable whenever possible. Deciding whether an invasive species has been successfully eradicated is difficult because of imperfect detection. Two previous studies [Regan et al., Ecology Letters, 9 (2006), 759; Rout et al., Journal of Applied Ecology, 46 (2009), 110] have used a decision theory framework to minimize the total expected cost by finding the number of consecutive surveys without detection (absent surveys) after which a species should be declared eradicated. These two studies used different methods to calculate the probability that the invasive species is present when it has not been detected for a number of surveys. However, neither acknowledged uncertainty in this probability, which can lead to suboptimal solutions.

2. We use info-gap theory to examine the effect of uncertainty in the probability of presence on decision-making. Instead of optimizing performance for an assumed systemmodel, info-gap theory finds the decision among the alternatives considered that is most robust to model uncertainty while meeting a set performance requirement. This is the first application of info-gap theory to invasive species management.

3. We find the number of absent surveys after which eradication should be declared to be relatively robust to uncertainty in the probability of presence. This solution depends on the nominal estimate of the probability of presence, the performance requirement and the cost of surveying, but not the cost of falsely declaring eradication.

4. More generally, to be robust to uncertainty in the probability of presence, managers should conduct at least as many surveys as the number that minimizes the total expected cost. This holds for any nominal model of the probability of presence.

5. Synthesis and applications. Uncertainty is pervasive in ecology and conservation biology. It is therefore crucial to consider its impact on decision-making; info-gap theory provides a way to do this. We find a simple expression for the info-gap solution, which could be applied by eradication managers to make decisions that are robust to uncertainty in the probability of presence.

Acknowledgement Many thanks to Yakov Ben-Haim and Dane Panetta for helpful advice and discussions. Also thanks to Michael Bode, Mark Burgman, Peter Baxter and three anonymous reviewers for comments on this manuscript. This research was supported by an Australian Postgraduate Award, the Commonwealth Environment Research Facility (AEDA), the Australian Centre of Excellence for Risk Analysis and an Australian Research Council Linkage Grant to MMcC (LP0884052).
Scores TUIGF:100%
SNHNSNDN:100%
GIGO:90%


Overview

My decision to review this article was triggered by the discovery that the info-gap model proposed in the paper as a framework for the modeling, analysis, and solution of the problem under consideration is not in fact a "proper" info-gap model. Or, to put it more accurately, strictly speaking the proposed model is not an info-gap model because it violates the Nesting Axiom of info-gap decision theory.

But as I began to examine the paper more carefully, it turned out that there are other matters that require critical comment.

I wish to point out, though, that at present I am working on a number of more urgent projects, so the amount of time I can devote to this review is quite limited. I shall therefore have to come back to it from time to time to enlarge on the current discussion.

This is a draft of Work in Progress (Feb 25, 2010)

At this stage I focus on the following points:


I now turn to a more detailed discussion of these points.

Errata

The mathematical analysis in the paper is not sufficiently careful. As a consequence, some of the results are incorrect. Some of the assertions may eventually prove to be correct, but the arguments supporting them are incorrect and/or incomplete.

For the purposes of this discussion it is sufficient to mention the following:

In a word, the results presented in this paper require corrections. This can be done either by imposing additional conditions on the problem so as to validate the results. Or by re-working the mathematical analysis properly to obtain results that are valid under the assumptions postulated in the paper.


The problem

It is important to be clear on what I mean when I say that the problem studied in this paper is a trivial problem.

Obviously, by this I do not want to suggest that the problem is unimportant in the disciplines of applied ecology, conservation biology, and so on. Indeed, the problem may have enormous consequences for applied ecology and conservation biology.

The point I am making is entirely different. My point is that when the "real world" situation that is investigated in this paper is cast as a decision-making problem and given a mathematical formulation, it becomes immediately clear that the mathematical problem is trivial in the sense that its solution flows (more or less) directly from its mathematical statement. As a matter of fact, the essential part of the problem can be solved by inspection.

Granted, you may contend that this does not argue against using info-gap for this purpose. Indeed, why not use info-gap to solve a problem that readily lends itself to solution by inspection? To which I would say, paraphrasing a reply once given by the renowned mathematician Richard Bellman (founder of dynamic programming): you may just as well tie a chair to your leg and attempt to swim across the river in this state.

My point is then that not only is the use of info-gap as a framework for modeling analysis and solution totally superfluous in this case. It is in fact counter-productive, if not harmful, because it gives a distorted picture of the problem in question. Worse, it gives a totally distorted picture of the real issues that we face in robust decision-making in the face of uncertainty.

To set the stage for the discussion, recall that the elements of the eradication problem under consideration are as follows:

To simplify the notation, we assume (with no loss of generality) that Cs=1, and we let E=Ce, and C=Nc. Also assume that C is an integer and let k=C+1.

So,

Statement of the problem:

Find the most robust decision d in D={1,2,3,...,k} against the uncertainty in the true value of u ∈[0,1] with respect to the performance requirement

(1)           d + uE ≤ C+1

where E and C are given positive numeric scalars, typically much greater than 1. The value of k is relatively small, say much smaller than 1000.

Note that because D is a discrete set, this problem is a discrete optimization problem.

Finally, it is important to stress that the uncertainty in the true value of u is assumed to be "considerable" (page 783) and that no probabilistic or likelihood assessments of this uncertainty are provided.

The essence of the problem

Roughly, in this context, decision d is robust if it satisfices the performance requirement d + uE ≤ C+1 for a large number of values of u∈[0,1]. The larger the number, the more robust d is. That is, we are searching for a decision whose feasible region of u, namely

(2)           F(d):= {u∈[0,1]: d + uE ≤ C+1} , d∈D
is large. The larger this region is the more robust the decision.

So, the point to note here is that the robustness that this problem aims to determine is not merely that of robustness against the uncertainty in the probability of presence. Rather, the idea here is to identify decisions that are robust against the impact of the uncertainty in the true value of this parameter on the budgetary constraint stipulated in (1).

Admittedly, as things stand, the problem is not sufficiently defined. That is, we still lack a formal definition of 'robustness'. Also, information is required regarding the nature of the uncertainty in the true value of u. Note that the latter affects the former.

And yet, despite all this, it is clear that this problem is trivial. To wit, it involves:

Thus, from the perspective of robust optimization, this is a "trivial problem".

But more than this, one need not even be a "professional" mathematician or an expert in robust optimization to figure out that

(3)           F(d) = [0,u*(d)] , d∈D
where
(4)           u*(d):= (C+1-d)/E

This follows directly (by inspection) from the definition of the performance requirement (1). For obvious reasons, we shall refer to u*(d) as the critical value of u associated with decision d.

It is immediately clear that the critical value of u would be pivotal in the definition of robustness against the uncertainty in the true value of u.

So:

In short, as can be gathered from the above, the formal definition of robustness would express "adjustments" made in these critical values in relation to what we know about the uncertainty in the true value of u.

The solution

Having clarified the issues that bear on the solution of the problem under consideration, we can now outline its solution procedure. To this end let ρ(d) denote the robustness of decision d against the uncertainty in the true value of u.

Solution Procedure

Note that the first step is easy: we compute the critical values according to the formula given in (4). The third step is also easy because the set of feasible decisions D is small, hence we can conduct the maximization of ρ(d) over D by enumeration. This means that a spreadsheet can be easily set up to solve this problem.

Of course, in cases where the maximization of ρ(d) over D={1,...,k} is also amenable to solution by analytic methods, it might be possible to obtain a closed-form solution for the continuous counterpart of the (discrete) problem that is under consideration. This would depend on the definition of robustness used.

What emerges therefore is that the real issue is in Step 2: the definition of the robustness function ρ=ρ(d).

Robustness against uncertainty

Let us examine a number of cases, representing various degrees of uncertainty in the true value of u. These cases refer to an estimate of the true value of u. Since this estimate may depend on d, we denote it û(d), d∈D.

Case 1: Certainty.

In this case we assume that the estimates û(d),d∈D are "perfect" and are equal to the respective true values of u. Thus, there is no uncertainty in the true value of the parameter.

This means that any decision is either thoroughly robust or utterly fragile. That is, d is thoroughly robust iff û(d) ≤ u*(d). Otherwise it is utterly fragile (infeasible).

Conclusion:

d is robust iff û(d) ≤ (c+1-d)/E

d is fragile iff û(d) > (c+1-d)/E

Note that in this extreme situation -- a perfect estimate -- there is even no need to use the solution procedure outlined above. We simply do as follows:

(5)           min {d + û(d)E: d∈D}

If the estimates û(d) are given by a nice smooth formula, it may be possible to identify robust decisions analytically by minimizing, over [1,k] the expression d + û(d)E, namely

(6)           min {d + û(d)E: d∈[1,k]}

If the optimal value of this expression is not greater than C+1, then the optimal value of d is the most robust decision. If this value is not a positive integer, then the nearest integer neighbors will have to be compared. And if the optimal value of this expression is greater than C+1, then all the decision are fragile.

Case 2: The estimates are very good

If the estimates û(d),d ∈ D, are very good, we would argue that it is in fact unnecessary to explore values of u that are substantially different from (much smaller or larger than) these estimates.

So, the following definition of robustness would be appropriate

(7)           ρ(d):= u*(d) - û(d) , d∈D

That is, the robustness of decision d is the "distance" between the estimate and the sub-region of [0,1] where u violates the performance requirement for this decision. For d to be robust, this distance should be large. The larger this distance the greater the robustness.

Thus, to find the optimal decision, we solve the following optimization problem:

(8)           max { ρ(d): d∈ D} = max {[C+1-d - û(d)E]/E: d∈D}

which is equivalent to

(9)           min {d + û(d)E: d∈D}

which is equivalent to the problem in Case 1: Certainty.

Case 3: The Estimates are very poor

In this case the estimates can be substantially wrong, namely no more than wild guesses, or perhaps just rumors. It may therefore be best to ignore them altogether.

Under the circumstances we may let the robustness ρ(d) be defined as follows

(10)           ρ(d):= u*(d) , d∈D

By inspection, in this case the most robust decision is d=1. Its robustness is equal to C/E. Note that if E≤C, then d=1 is super-robust: it satisfies the performance constraint for all u∈ [0,1].

Again, the situation is so simple that we need not even use the procedure outlined above.

Remarks:

The Moral of the Story

We can continue in this vein: formulating "sensible" definitions of robustness for our small problem ad infinitum ...

For example, we may want to "scale" the critical values u*(d),d∈D by dividing them by the corresponding estimates û(d),d∈D; or we may even want to "normalize" the critical values and consider [u*(d)-û(d)]/û(d) as the measure of robustness of decision d -- as done in the paper. Note that these two alternatives are equivalent.

More generally, we can incorporate "weights", call them w(d), d∈D, to refine the robustness function ρ. In particular, in cases where the estimates are very good, we can let

(11)           ρ(d) := [u*(d) - û(d)]/w(d) = [C+1-d - û(d)E]/w(d)E

where w(d) >0 is the weight associated with decision d.

The point is then that the simplicity of the performance constraint implies that the critical values u*(d),d∈D can be easily determined by inspection, and then be invoked in the definition of robustness.

Remarks:

Let us now examine how robustness is defined in the article.

Proposed Info-Gap Robustness Models:

In sharp contrast to the effortless manner in which the definitions of robustness are derived above - directly from the statement of the problem, the derivations of robustness in the article get unnecessarily complicated by the requirement to be expressed in terms of an info-gap robustness model.

In fact, the derivations are suffiently complicated that they are not explained in full in the body of the article. You may plough through the article but you will not find the expression (formula) used to measure the robustness of decisions in the context of the problem under consideration. For this you'll have to read Supplement S1.

But more than this, the derivations of robustness in the article and in the two supplements make no reference to the intuitive notion critical value of u(d). As we have seen above, this concept in fact brings out what robustness is all about in the problem studied in the article.

When you finally figure out what it is (eqn A5 in Appendix S1), you'll discover that it is the instance of the robustness model specified by (11) that corresponds to

(12)           w(d) = û(d)     ---->     ρ(d) = [C+1-d]/[û(d)E] - 1

The info-gap robstness model specified in eqn A2 in Appendix S2 corresponds to

(13)           w(d) = 1 - û(d)     ---->     ρ(d) = [C+1-d - û(d)E]/[E(1-û(d))]

The point to note about these two info-gap models is that they correspond to the definitions of robustness falling under what we refer to above as Case 2: The Estimates are Very Good.

Indeed, this is what is so interesting about these two info-gap models: the fact they that they correspond to the definitions falling under what we refer to above as Case 2: The Estimates are Very Good. Namely, for the models to make sense the estimates should be asssumed to be good, but ... the authors assume that the true value of u " ... is subject to considerable uncertainty and ignoring this uncertainty may lead to suboptimal solutions ..." (p. 783; emphasis is mine).

No comment whatsoever is made to explain the blatant incongruity between the fact that (as assumed in the paper) the estimates are poor (due to the considerable uncertainty in the true values of the parameters) and the fact that they are pivotal in the determination of robustness. Moreover, no sensitivity analysis is conducted on the estimates themselves!

Stay tuned ...


The 64K$ question

Given then the utter simplicity with which the problem under consideration can be analyzed and solved, I repeat the question raised above: why use info-gap decision theory to solve this problem?

Surely the authors should explain the rational, the point, the merit of using info-gap decision theory as a framework for the modeling, analysis and solution of a problem whose solution is simplicity itself.

I might add in this regard that over the past five years I have written a number of letters to authors of papers on info-gap decision theory. Occasionally, the letter is about the triviality of the problem under consideration.

So I prepared a generic letter, which I modify according to the circumstances. Here is a version intended for cases where the problem is "trivial":

Dear ??????:

I read with interest your paper entitled ??????.

Note that the essence of the problem investigated in this paper boils down to this:

Determine the critical value of ?????, namely the worst (largest) value of ?????? that satisfies the performance constraint ????? or equivalently ?????.

Therefore, my immediate reaction to the analysis in your paper was sheer amazement!

After all, by inspection, the answer is obviously ??????. Therefore, one cannot help but wonder how Info-Gap suddenly appears on the scene ?!

This is yet another example of what can happen when instead of trying to model and solve a given problem, one tries -- by hook or by crook -- to use a given methodology to model and solve this problem.

On a number of occasions I alluded to this danger. But here I must be more forthright.

This article is a good example of how easily one can end up focusing almost exclusively on manipulating the formulation of a given problem so as to fit it into the paradigm of a Beloved methodology. So much so that one may fail to see that the problem is actually so trivial that it can be easily solved by inspection.

Isn't it time, ??????, that we asked ourselves:
Are we in the business of developing, using and promoting scientific methods for decision-making under uncertainty in the area of ??????, or are we in the Info-Gap business? I cannot see how we can make progress on the important issues that we identified if we keep ourselves busy trying to fix conceptual and technical Info-Gap bugs.

Best wishes

    Moshe
Melbourne (date:??????)

This generic letter applies to the paper under review.

Stay tuned for more ...


The Continuing Maximin Saga

As indicated above, now that Bryan Beresford-Smith and Colin J. Thompson (2009) have conceded that Info-Gap's robustness model is a Maximin model (see Review 11), what is the point of withholding this fact from scientists in the field of applied ecology?

Stay tuned more ...


The Ongoing Severe Uncertainty Saga

The authors concede that info-gap decision theory is unsuitable for situations where the estimate is likely to be substantially wrong. For consider this:

Although info-gap theory is relevant for many management problems, two components must be carefully selected: the nominal estimate of the uncertain parameter, and the model of uncertainty in that parameter. If the nominal estimate is radically different from the unknown true parameter value, then the horizon of uncertainty around the nominal estimate may not encompass the true value, even at low performance requirements.
Rout et al (2009, p. 785)

However, their explanation of this fact is totally wrong.

To begin with, a distinction must be drawn between:

Insofar as the problem statement is concerned, the uncertainty is described by the uncertainty space under consideration, call it U, and the estimate, call it û. Needless to say, the estimate û and the true value of u are assumed to be elements of U. Obviously, if the estimate is poor the complete uncertainty space can be vast. This explains why, according to Ben-Haim (2006, p. 210), the most commonly encountered info-gap uncertainty models are unbounded.

Enter info-gap.

Given this, the info-gap model of uncertainty is constructed so that its regions of uncertainty, call them U(α,û),α≥0, are centered at û, and at least one of them contains U. Thus, if the uncertainty space U is vast, so would be the regions of uncertainty U(α,û) for large values of α.

That said, it is clear that the real trouble with info-gap's robustness analysis is not that the true value of u may not be contained in the uncertainty space of U.

Comment

I should add that I have yet to come across an info-gap publication where it is not immediately obvious that the uncertainty space U contains the (unknown) true value of u. So, for our purpose here it would be best to leave it at that: the true value of u is unknown, but it is contained in the complete uncertainty space. This means that it is contained in at least one of the regions of uncertainty centered at the estimate.

Indeed, it is ironic that the authors should raise this issue at all in this paper. After all, in the case of the problem they investigate, the unknown parameter under consideration is a probability, hence the (unknown) true value of the parameter of interest is definitely in the bounded interval [0,1].

The real trouble with info-gap's analysis lies elsewhere. It lies in info-gap's localized robustness analysis.

That is, info-gap's robustness model conducts the robustness analysis, in the first instance, in the immeidate neighborhood of the given estimate. This means that a decision that violates the performance constraint at a u near the estimate is deemed fragile regardless of its performance in neighborhoods of the uncertainty space that are further way from the estimate.

By definition, therefore, info-gap decision theory does not seek decisions that are robust against uncertainty over the given uncertainty space U. It seeks decisions that are robust in the neighborhood of the given estimate û.

So, the difficulties that info-gap's analysis would run into would remain even if the (unknown) true value of the parameter would be contained in the uncertainty space stipulated by the problem statement. In a word, the trouble is with info-gap's lame "search methodology" which a-priori undermines its ability to properly explore the uncertainty space especially under conditions of --- what the authors term --- "considerable" uncertainty.

The following picture illustrates this point.

It shows the rewards R(q,u) generated by two decisions, q' and q'', as a function of some parameter u. The estimate of the true value of u is û = 0, the uncertainty space is U=(-∞,∞) and the performance requirement is R(q,u) ≥ 0.

According to Info-Gap's robustness model, q'' is more robust than q' with respect to R(q,u) ≥ 0 because the closest u to û that violates the constraint R(q',u) ≥ 0 is at a distance α'=1.08 from û, whereas the closest u to û that violates the constraint R(q'',u) ≥ 0 is at a distance α''=1.429 from û. Hence, q'' is more robust than q'.

Note, however, that

This example also illustrates why Info-Gap's robustness analysis cannot handle ordinary, plain, white Swans, let alone genuine (Australian) Black Swans.

See the discussion on this issue at Second Opinion on Info-Gap Decision Theory.

And how about this:

Thus, the method challenges us to question our belief in the nominal estimate, so that we evaluate whether differences within the horizon of uncertainty are 'plausible'. Our uncertainty should not be so severe that a reasonable nominal estimate cannot be selected.
Rout et al (2009, p. 785)

Since info-gap decision theory is non-probabilistic and likelihood-free, info-gap users are in no position to quantify levels, or degrees, or what have you, of "good". "reasonable", or "bad" that are applicable to the estimate. Nor are they in any position to determine what is more or less "plausible" within the horizon of uncertainty even when the plausible is only 'plausible'.

Add to this the fact that info-gap decision theory does not even begin to deal with the question of how the estimates are obtained. Namely, info-gap decision theory does not bother to give us so much as a clue on how to check/verify whether the estimate is bad, poor, good, excellent, perfect and it is clear that determining the quality of the estimate is an external issue. In other words, you come to info-gap decision theory with an estimate in hand. And in this case as well, there is nothing in info-gap decision theory itself that would enable it to distinguish between the quality of various estimates.

So what are the authors telling us?

The authors seem to be saying the obvious: unless you have good reasons to believe that the estimate you have is "pretty reasonably good" (whatever that means), it makes no sense to focus the robustness analysis on the immediate neighborhood of the estimate. In other words, it makes no sense to do what info-gap prescribes doing. In this case the authors agree with my criticism of info-gap decision theory.

But more than this, are the authors willing to stick their necks out and declare that an estimate subject to "considerable" uncertainty (be it a rumor or gut feeling or whatever) is so good that it makes sense to confine the robustness analysis to a given neighborhood of the estimate and call it a day?

In this case, the authors would have to do as follows:

  1. Stipulate the uncertainty space of the problem, call it U. That is, they would have to specify the smallest set that (the decision-maker is reasonably confident) contains the true value of the parameter of interest.

  2. Specify the value of the estimate.

  3. Conduct a robustness analysis that seeks decisions that are robust on the given uncertainty space U.

But this, one need hardly point out, is not what info-gap decision theory does!

Info-gap decision theory does not seek decision that are robust on U. It seeks decisions that are robust in the neighborhood of the given estimate, to wit: a decision that is not robust in the immediate neighborhood of the estimate is eliminated from any further consideration even if it performs exceptionally well in other neighborhoods of U.

So, ... how are the authors going to use info-gap decision theory to identify decisions that are robust on U rather than in the immediate neighborhood of the estimate?

In any case, suppose that the uncertainty is not so severe and we have in hand a reasonably good estimate. How then can the local analysis in the neighborhood of this estimate enable dealing with rare events, catastrophes etc? Hence, how about this:

For ecological management in the face of uncertainty, managers may use info-gap to gain some protection against catastrophic outcomes by answering the question: how wrong could this model be before outcomes are unacceptably bad?
Rout et al (2009, p. 785)

Note that info-gap decision theory does not -- indeed, is in principle unable to -- answer the question: how wrong could this model be before outcomes are unacceptably bad?

This is so because the true value of the parameter of interest is unknown and is subject to considerable uncertainty. There is therefore no way of knowing how wrong the model is.

Info-gap robustness model answers a completely different question, namely: what is the largest region of uncertainty around the estimate over which the performance constraint is satisfied?

Strictly speaking, this question has nothing to do with uncertainty as such. Moreover, the answer to this question does not depend on the "quality" of the nominal point, or estimate, used as the center point of the regions of uncertainty.

See discussion on this topic in my response to Burgamn's comments on my criticism of info-gap decision theory.

In short, info-gap decision theory does not -- much less is it able to -- deal with the question stated by the authors and it therefore cannot help managers gain protection against catastrophic outcomes.

As a matter of fact, info-gap decision theory constitutes the precise antithesis of what a theory for modeling, analyzing and managing severe uncertainty ought to be. Instead of exploring thoroughly the given complete uncertainty space, info-gap decision theory focuses its robustness analysis in the neighborhood of a point estimate.

Remarks:

This is definitely a move in the right direction, but ... it does not go far enough!

Stay tuned for more ...


The State of the Art Saga

The paper shows complete disrespect for the state of the art in decision-making under uncertainty. The discussion section basically amounts to an uncritical endorsement of info-gap decision theory. Well established methods that have become the "bread and butter" approaches to decision-making subject to uncertainty are not even mentioned. The paper thus provides an extrememly distorted picture of the area of decision-making under severe uncertainty.

It also ignores relevant publications that are critical of info-gap decision theory.

Stay tuned for more ...


Unsubstantiated and/or misleading statements

The paper is riddled with "problematic" statements. I shall mention just a few.

Stay tuned for more ...


Conclusions

The problem examined in the article is trivial, so much so that its essence can be solved by inspection.

The main results, namely (p. 782):

3. We find the number of absent surveys after which eradication should be declared to be relatively robust to uncertainty in the probability of presence. This solution depends on the nominal estimate of the probability of presence, the performance requirement and the cost of surveying, but not the cost of falsely declaring eradication.

4. More generally, to be robust to uncertainty in the probability of presence, managers should conduct at least as many surveys as the number that minimizes the total expected cost. This holds for any nominal model of the probability of presence.

are unsubstantiated and certain assertions made in the article are (technically) wrong.

On the whole, this article is a typical info-gap article. The only point of difference between this article and other info-gap publications is that the authors concede that (despite all the rhetoric in the info-gap literature) it is obvious that this theory is unsuitable for the treatment of severe uncertainty where the estimate is likely to be substantially wrong.

Otherwise, the paper follows the established info-gap line that info-gap decision theory is distinct and radically different from "common" theories for decision under uncertainty. Particularly jarring in this respect is its omission of the fact that info-gap robustness model is in fact a Maximin model.

Consequently the paper gives a thoroughly distorted account of the state of the art in robust decision-making under uncertainty.

Other Reviews

  1. Ben-Haim (2001, 2006): Info-Gap Decision Theory: decisions under severe uncertainty.

  2. Regan et al (2005): Robust decision-making under severe uncertainty for conservation management.

  3. Moilanen et al (2006): Planning for robust reserve networks using uncertainty analysis.

  4. Burgman (2008): Shakespeare, Wald and decision making under severe uncertainty.

  5. Ben-Haim and Demertzis (2008): Confidence in monetary policy.

  6. Hall and Harvey (2009): Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.

  7. Ben-Haim (2009): Info-gap forecasting and the advantage of sub-optimal models.

  8. Yokomizo et al (2009): Managing the impact of invasive species: the value of knowing the density-impact curve.

  9. Davidovitch et al (2009): Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island.

  10. Ben-Haim et al (2009): Do we know how to set decision thresholds for diabetes?

  11. Beresford and Thompson (2009): An info-gap approach to managing portfolios of assets with uncertain returns

  12. Ben-Haim, Dacso, Carrasco, and Rajan (2009): Heterogeneous uncertainties in cholesterol management

  13. Rout, Thompson, and McCarthy (2009): Robust decisions for declaring eradication of invasive species

  14. Ben-Haim (2010): Info-Gap Economics: An Operational Introduction

  15. Hine and Hall (2010): Information gap analysis of flood model uncertainties and regional frequency analysis

  16. Ben-Haim (2010): Interpreting Null Results from Measurements with Uncertain Correlations: An Info-Gap Approach

  17. Wintle et al. (2010): Allocating monitoring effort in the face of unknown unknowns

  18. Moffitt et al. (2010): Securing the Border from Invasives: Robust Inspections under Severe Uncertainty

  19. Yemshanov et al. (2010): Robustness of Risk Maps and Survey Networks to Knowledge Gaps About a New Invasive Pest

  20. Davidovitch and Ben-Haim (2010): Robust satisficing voting: why are uncertain voters biased towards sincerity?

  21. Schwartz et al. (2010): What Makes a Good Decision? Robust Satisficing as a Normative Standard of Rational Decision Making

  22. Arkadeb Ghosal et al. (2010): Computing Robustness of FlexRay Schedules to Uncertainties in Design Parameters

  23. Hemez et al. (2002): Info-gap robustness for the correlation of tests and simulations of a non-linear transient

  24. Hemez et al. (2003): Applying information-gap reasoning to the predictive accuracy assessment of transient dynamics simulations

  25. Hemez, F.M. and Ben-Haim, Y. (2004): Info-gap robustness for the correlation of tests and simulations of a non-linear transient

  26. Ben-Haim, Y. (2007): Frequently asked questions about info-gap decision theory

  27. Sprenger, J. (2011): The Precautionary Approach and the Role of Scientists in Environmental Decision-Making

  28. Sprenger, J. (2011): Precaution with the Precautionary Principle: How does it help in making decisions

  29. Hall et al. (2011): Robust climate policies under uncertainty: A comparison of Info-­-Gap and RDM methods

  30. Ben-Haim and Cogan (2011) : Linear bounds on an uncertain non-linear oscillator: an info-gap approach

  31. Van der Burg and Tyre (2011) : Integrating info-gap decision theory with robust population management: a case study using the Mountain Plover

  32. Hildebrandt and Knoke (2011) : Investment decisions under uncertainty --- A methodological review on forest science studies.

  33. Wintle et al. (2011) : Ecological-economic optimization of biodiversity conservation under climate change.

  34. Ranger et al. (2011) : Adaptation in the UK: a decision-making process.


Recent Articles, Working Papers, Notes

Also, see my complete list of articles
    Moshe's new book!
  • Sniedovich, M. (2012) Fooled by local robustness, Risk Analysis, in press.

  • Sniedovich, M. (2012) Black swans, new Nostradamuses, voodoo decision theories and the science of decision-making in the face of severe uncertainty, International Transactions in Operational Research, in press.

  • Sniedovich, M. (2011) A classic decision theoretic perspective on worst-case analysis, Applications of Mathematics, 56(5), 499-509.

  • Sniedovich, M. (2011) Dynamic programming: introductory concepts, in Wiley Encyclopedia of Operations Research and Management Science (EORMS), Wiley.

  • Caserta, M., Voss, S., Sniedovich, M. (2011) Applying the corridor method to a blocks relocation problem, OR Spectrum, 33(4), 815-929, 2011.

  • Sniedovich, M. (2011) Dynamic Programming: Foundations and Principles, Second Edition, Taylor & Francis.

  • Sniedovich, M. (2010) A bird's view of Info-Gap decision theory, Journal of Risk Finance, 11(3), 268-283.

  • Sniedovich M. (2009) Modeling of robustness against severe uncertainty, pp. 33- 42, Proceedings of the 10th International Symposium on Operational Research, SOR'09, Nova Gorica, Slovenia, September 23-25, 2009.

  • Sniedovich M. (2009) A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 2071-2079, Taylor and Francis Group, London.
  • .
  • Sniedovich M. (2009) A Classical Decision Theoretic Perspective on Worst-Case Analysis, Working Paper No. MS-03-09, Department of Mathematics and Statistics, The University of Melbourne.(PDF File)

  • Caserta, M., Voss, S., Sniedovich, M. (2008) The corridor method - A general solution concept with application to the blocks relocation problem. In: A. Bruzzone, F. Longo, Y. Merkuriev, G. Mirabelli and M.A. Piera (eds.), 11th International Workshop on Harbour, Maritime and Multimodal Logistics Modeling and Simulation, DIPTEM, Genova, 89-94.

  • Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)

  • Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)

  • Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08
    This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file )

  • Sniedovich, M. (2008) A call for the reassessment of Info-Gap decision theory, Decision Point, 24, 10.

  • Sniedovich, M. (2008) From Shakespeare to Wald: modeling wors-case analysis in the face of severe uncertainty, Decision Point, 22, 8-9.

  • Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!, Journal of Risk Finance, 9(3), 287-291.

  • Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)
    In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model.

  • Sniedovich. M. (2008) The Mighty Maximin! (PDF File)
    This paper is dedicated to the modeling aspects of Maximin and robust optimization.

  • Sniedovich, M. (2007) The art and science of modeling decision-making under severe uncertainty, Decision Making in Manufacturing and Services, 1-2, 111-136. (PDF File) .

  • Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)
    In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory.

  • My reply (PDF File) to Ben-Haim's response to one of my papers. (April 22, 2007)

    This is an exciting development!

    • Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.

    • Ben-Haim is not familiar with the fundamental concept point estimate. He does not realize that a function can be a point estimate of another function.

      So when you read my papers make sure that you do not misinterpret the notion point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.

      Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!


  • A formal proof that Info-Gap is Wald's Maximin Principle in disguise. (December 31, 2006)
    This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)
    It shows that Info-Gap is not a new theory but rather a simple instance of Wald's famous Maximin Principle dating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context of Game Theory (1928).

  • A proof that Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)
    This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File) .
    It shows that because Info-Gap deploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust.

  • A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)
    This is a very short article entitled The GAP in Info-Gap (PDF File) .
    It is a math-free version of the paper above. Read it if you are allergic to math.

  • A long essay entitled What's Wrong with Info-Gap? An Operations Research Perspective (PDF File) (December 31, 2006).
    This is a paper that I presented at the ASOR Recent Advances in Operations Research (PDF File) mini-conference (December 1, 2006, Melbourne, Australia).

Recent Lectures, Seminars, Presentations

If your organization is promoting Info-Gap, I suggest that you invite me for a seminar at your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use — let alone promote — Info-Gap as a decision-making tool.

Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.


Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.


Last modified: