Reviews of publications on Info-Gap decision theory

Review # 10(Posted: June 1, 2009; Last update: June 1, 2009)

Reference: Y. Ben-Haim, M. Zacksenhouse, C. Keren, C.C. Dacso

Do we know how to set decision thresholds for diabetes?

Medical Hypotheses,Volume 73, Issue 2, Pages 189-193, 2009Abstract The diagnosis of diabetes, based on measured fasting plasma glucose level, depends on choosing a threshold level for which the probability of failing to detect disease (missed diagnosis), as well as the probability of falsely diagnosing disease (false alarm), are both small. The Bayesian risk provides a tool for aggregating and evaluating the risks of missed diagnosis and false alarm. However, the underlying probability distributions are uncertain, which makes the choice of the decision threshold difficult. We discuss an hypothesis for choosing the threshold that can robustly achieve acceptable risk. Our analysis is based on info-gap decision theory, which is a non-probabilistic methodology for modelling and managing uncertainty. Our hypothesis is that the non-probabilistic method of info-gap robust decision making is able to select decision thresholds according to their probability of success. This hypothesis is motivated by the relationship between info-gap robustness and the probability of success, which has been observed in other disciplines (biology and economics). If true, it provides a valuable clinical tool, enabling the clinician to make reliable diagnostic decisions in the absence of extensive probabilistic information. Specifically, the hypothesis asserts that the physician is able to choose a diagnostic threshold that maximizes the probability of acceptably small Bayesian risk, without requiring accurate knowledge of the underlying probability distributions. The actual value of the Bayesian risk remains uncertain. Acknowledgement This work was supported in part by a Grant from the Abramson Center for the Future of Health, Houston, TX. Scores TUIGF:100%

SNHNSNDN:100%

GIGO:200%

Since the idea of regarding Info-Gap's robustness model as a "proxy" for a probabilistic model of "success" had already been raised in a number of Ben-Haim's publications, I shall first examine how it sits in the general context of Info-Gap decision theory. Then, I shall examine it in some detail to see how it fares with regard to the specific Info-Gap model discussed in the paper.

## Overview

Let us begin by noting that the journal Medical Hypotheses is devoted to the publication of ... medical hypotheses. According to its Guidelines for Authors:

The purpose of Medical Hypotheses is to publish interesting theoretical papers. The journal will consider radical, speculative and non-mainstream scientific ideas provided they are coherently expressed.So, the question is: does the hypothesis proposed in the article fit the bill? Is it indeed radical and non-mainstream? And if so, in what sense is it radical and non-mainstream?

To work out an answer to this question you need not bother to read the paper in its entirety. The answer stares you in the face ... in the abstract, where we read (emphasis is mine):

Our analysis is based on info-gap decision theory, which is anon-probabilisticmethodology for modelling and managing uncertainty.Our hypothesis is that the non-probabilistic method of info-gap robust decision making is able to select decision thresholds according to their probability of success.The main objective of this review is to show that the proposition made by this hypothesis is not just "radical and non-mainstream", it is in fact unscientific, or more accurately counter-scientific. And to be able to appreciate why this is so keep in mind that Info-Gap's robustness model is by definition

localand that the uncertainty that this model is designed to take on issevere.So the good news is that the future of probability theory and statistics is not in danger ... not yet!

In greater detail, consider the two circles, A and B on the left. The large circle represents the space that affects the choice and ranking of decisions according to their probability of success. The much smaller circle, B, represents the space that affects the choice and ranking of decisions as prescribed by Info-Gap's robustness analysis.

That B is typically a relatively small subset of A is a direct consequence of Info-Gap's local robustness analysis being conducted only on an infinitesimally small area of the (unbounded) uncertainty space (A) that Info-Gap decision theory presumably takes on.

So mathematically speaking we can look at the proposed hypothesis as being grounded in a comparison between the results yielded by two algorithms: one is defined on A, call it f, and one is defined on B, call it g. Algorithm f yields the ranking of decisions according to their probability of success, whereas Algorithm g yields the ranking of decisions according to their robustness (as prescribed by Info-Gap).

The authors' hypothesis is that these two algorithms yield the same results.

But the point to note here is that for this hypothesis to hold, Algorithm g must exhibit a significant level of redundancy and/or degeneracy. However, since redundancy and/or degeneracy is not a generic characteristic of the probabilistic models of "success" examined by the authors, the inference is that this must be imposed on the two models through certain assumptions that would link the performance of the two algorithms.

Since the authors do not postulate any such assumptions, there is no reason to believe that the proposed hypothesis is valid.

To the contrary, there is every reason to believe that in the absence of some inherent

redundancy/degeneracyin the specific probabilistic model of success examined in the article, this hypothesis cannot hold. It is simplytoo good to be true: how can a non-probabilistic analysis of a small subset of a much large set be a reliable proxy for a probabilistic analysis of "success" over the much larger set?Indeed, if we accept this "too good to be true" hypothesis, why not accept "too good to be true" offers such as this:

Date: Sun, 7 Jun 2009 01:47:13 +0200 (MEST) From: ??????? Reply-To: ??????? Subject: My Friend My Friend How are you today? Hope all is well with you and your family? I am using this opportunity to inform you that this multi-million-dollar business has been concluded with the assistance of another partner from Chile who financed the transaction to a logical conclusion. Due to your effort,sincerity courage and trust worthiness You showed during the course of the transaction. I have left a certified Bank Draft. for you worth of $1,300,000.00 cashable anywhere in the world. Mr. ??????The absurd in the proposed hypothesis is on a par with the absurd in the proposition that

what you see through a keyhole is a reliable proxy of what you see through an open door.

vs Hide/Show the BIG picture!

As they say: never believe what you see through a keyhole!

And if you like cheese, then perhaps the following analogy will be more meaningful to you:

vs Hide/Show the BIG picture!

Perhaps.

One of the most impressive illustrations of this point is no doubt this:

vs Hide/Show the BIG picture!

Commonly known as "the tip of the iceberg".

However, to my mind, the most edifying analogy is the distinction between a

localand aglobaloptimum.

vs Hide/Show the BIG picture!

As we well know, generally, a local optimum is not necessarily a global optimum. Indeed, we go to great lengths to explain to our students why it is important to avoid confusing between these two related concepts.

A similar distinction between local and global robustness of course applies in the case of Info-Gap. But we find no trace of it in the Info-Gap literature. To wit:

- Info-Gap's rhetoric
promisesaglobalrobustness againstsevereuncertainty,whereas

- Info-Gap's robustness analysis
deliversalocalrobustness in the neighborhood of awild guess.Worse, not only is there no awareness whatsoever, in the Info-Gap literature, of the difference between these two types of robustness, there is no awareness of the fact that the robustness yielded by Info-gap is local.

And to top it all off, Info-Gap scholars persistently fail (or avoid) to indicate that Info-Gap's robustness model is in fact a simple instance of Wald's famous Maximin model (circa 1940). Meaning that the Info-Gap analysis is in fact a (local) worst-case analysis.

In a word, only a failure to appreciate the distinction between global and local robustness could have inspired this hypothesis.

## Illustrative example

Suppose that robustness is sought with respect to the constraint f(x) ≥ 0 over the set X=[-7,7], where f is the function shown below:

If x is the realization of a

random variable, we can consider the probability that f(x) ≥ 0 as a measure of robustness. In this case this probability is the probability that the random variable takes value in the blue intervals shown below:If the probability distribution of this random variable is known, then it would be a simple task to compute the probability that f(x) ≥ 0.

But what should we do if we face severe uncertainty and we do not know the distribution of this random variable?

The hypothesis proposes that Info-Gap's non-probabilistic robustness model can be used as a proxy for obtaining the unknown probability distribution.

To apply this model we shall have to estimate the "true" value of x and conduct the robustness analysis in the neighborhood of this estimate, call it x*. For instance, if we assume that x*=3, we shall conduct the robustness analysis in the interval, say [1.5,4], as shown below:

So in this case the hypothesis is that Info-Gap's local robustness analysis in the interval X'=[1.5,4] will yield a reliable proxy for the probability of f(x) ≥ 0 over the interval X=[-7,7].

In short, if we accept that the hypothesis has any merit whatsoever, we must accept that Info-Gap's robustness model is endowed with substantial extraordinary powers. Namely, we must accept that it has the capability to generate

reliable probabilisticresults out of anon-probabilisticmodel of uncertainty, and all this by means of alocalanalysis in the neighborhood of apoor estimate!This is too good to be true!

## Reality Check

So, to set the record straight on the proposition that Info-Gap robustness is a reliable proxy for the probability of success, consider the following formal result:

Reality Check Theorem:The proposed hypothesis is

notvalid.That is, consider the generic Info-Gap robustness model

α(q,û):= {α≥0: R(q,u) ≤ R _{c}, ∀u∈U(α,û)} , q∈Qand assume that α(q'',û) > α(q',û) for some decisions q' and q'' in Q. Then this

does not implythat Pr[R(q'',u) ≤ R_{c}] > Pr[R(q',u) ≤ R_{c}], where Pr[event] denotes the probability of "event" as determined by some (unknown) probability distribution on the uncertainty spaceU.

Proof.It is sufficient to show that there are cases where for some decisions q' and q'' such that α(q'',û) > α(q',û) there is a probability distribution on

Usuch that Pr[R(q',u) ≤ R_{c}] > Pr[R(q'',u) ≤ R_{c}].Let

D(q',q''):= {u∈ U: R(q',u) ≤ R_{c}; R(q'',u)> R_{c}}By definition then, D(q',q'') denotes the subset of

Uon which decision q' satisfices the performance requirement and decision q'' violates this requirement.Now consider any case where D(q',q'') is not empty. Then clearly, there is a probability distribution on

Usuch that Pr[R(q',u) ≤ R_{c}] > Pr[R(q'',u) ≤ R_{c}]. For example, any distribution whose support is a subset of D(q',q'') will do the job.QEDClearly then, the proposed hypothesis does not hold in the context of Info-Gap's generic robustness model.

Our next task is to explain the fundamental flaws in the thinking behind the hypothesis.

## Garbage In — Garbage Out Axiom

The question that we need to ask ourselves is as follows:

How is it possible that a methodology employing anon-probabilisticmodel of uncertainty is capable of selecting decisions according to theirprobabilityof success?The short answer is clear:

Either this hypothesis is groundless, or ... there exist certain implicit powerful links between the probabilistic and non-probabilistic models that empower the latter to mimic the behavior of the former.Regarding the second option, see for example the well known concept deterministic equivalent that is used extensively in stochastic programming. All that needs to be pointed out here is that, the existence of deterministic equivalent models relies on the validity of very strong assumptions.

The longer answer is as follows:

Either this hypothesis is groundless, or ... there exist certain implicit powerful links between the probabilistic and non-probabilistic models that empower the latter to mimic the behavior of the former.It does not take much to explain why this hypothesis is groundless with regard to Info-Gap's robustness model. Because, to claim that this hypothesis is valid for Info-Gap's robustness model, without invoking the support of certain prerequisite assumptions, is as good as declaring in public that the universally accepted Garbage In -- Garbage Out (GIGO) Axiom is null and void.

But in this case Info-Gap decision theory puts itself at loggerheads with "Conventional Science" as follows:

Conventional Science Info-Gap Decision Theory

wild guess -----> Model -----> wild guess

wild guess -----> Model -----> reliable

robust decisionThat is, in science the ruling convention is that results generated by a model can be only as good as the estimate on which they are based. Yet, Info-Gap decision theory proclaims its robustness model capable of generating reliable robust decisions by an analysis of the neighborhood of a wild guess of the true value of the parameter of interest. Namely, without bothering to give the slightest justification, Info-gap attributes extraordinary powers to its robustness model, to reliably extrapolate from local results in a small neighborhood of a wild guess to the entire vast uncertainty space

U.In short, by contravening the Garbage In -- Garbage Out Axiom, Info-Gap decision theory is at odds with a fundamental precept of science thereby showing itself to be an unscientific theory. Therefore, the very idea of proposing such an hypothesis is of a piece with the voodoo (counter-scientific) approach taken by Info-Gap to decision- making under uncertainty.

It is apposite to add here that if we accept for a fact Info-Gap's extraordinary ability to generate reliable robust decisions out of a wild guess, then we may as well wind up the discipline of Decision Making Under Severe Uncertainty and declare it redundant. For, dealing with decision-making problems subject to severe uncertainty would now amount to child's play:

1-2-3 fool-proof recipe for decision-making under severe uncertainty

- Ignore the severity of the uncertainty.

- Focus instead on the neighborhood of your best estimate of the parameter of interest.

- Don't worry if you lack an estimate, a wild guess will do
^{*}.Wouldn't this be great?!

^{*}Should you need it, the recipe for obtaining a wild guess is simplicity itself:

- Wet your index finger and put it in the air.

- Think of a number and double it.
See it on-line at wiki.answers.com/Q/What_is_best_estimate_and_how_do_i_calculate_it.

## The Hypothesis

Given the state of the art in decision theory and robust optimization, what we have here is

Eithera majorOr,breakthroughin decision theory and robust optimization!a huge blunder attesting to afundamental misapprehension/misconceptionof the central difficulty confronting decision-making in the face of severe uncertainty.So, to repeat the obvious:

The basic difficulty facing decision-making under severe uncertainty is that the estimate we have of the true value of the parameter of interest is poor and likely to be substantially wrong. Hence, there is no reason to believe that a local analysis in the neighborhood of the estimate will yield reliable results.The question is then: what should be done given this state of affairs?!

In a nutshell, the severity of the uncertainty makes a

globaltreatment of robustness imperative. That is, it demands that the entire uncertainty space, or a suitable global approximation thereof, be explored.Here then are two simple reasons why the proposed hypothesis is groundless:

- Info-Gap's robustness analysis is not conducted over the entire specified uncertainty space.

- In the framework of Info-Gap decision theory, the estimate used is a wild guess and is likely to be substantially wrong.
So, the question is: how can the proposed hypothesis have any merit whatsoever when Info-Gap's robustness model itself (before you even begin to ponder the hypothesis) is fundamentally flawed? And the answer is plain: for the hypothesis to hold, not only is it imperative that extremely stringent conditions hold to establish the envisaged powerful nexus between Info-Gap's robustness model and the probabilistic model of "success". It is imperative that the model itself be sound. Since no such conditions are incorporated on the (already fundamentally flawed) generic Info-Gap robustness model, it is not surprising that the hypothesis is generally not valid.

By the same token, given that no such conditions are imposed on the models presented in the paper, it follows that there is no reason to believe that the hypothesis can hold for these models.

To examine this contention more formally, assume that the performance constraint is defined as follows:

R(q,u) ≤ R

_{c}where q represents the decision variable, u represent the parameter of interest, R represents the performance functions and R

_{c}denotes the required performance level.In this framework, the probability of success is

Pr

_{U}[R(q,u) ≤ R_{c}]where Pr

_{U}[event] denotes the probability of "event". The subscriptUis used as a reminder that the probability is computed over the uncertainty spaceU, namely the set of possible values of u.So far so good.

Now, according to Info-Gap decision theory, the robustness of decision q is defined as follows:

α(q,û):= max {α: R(q,u) ≤ R

_{c}, ∀u∈U(α,û)}where û denotes the estimate of the true value of u and U(α,û) denotes a region of uncertainty of size α around û.

So, the main thesis of this paper is as follows:

Hypothesis:

α(q'',û) > α(q',û) ————> Pr _{U}[R(q'',u) ≤ R_{c}] > Pr_{U}[R(q',u) ≤ R_{c}]In words: if Info-Gap's analysis deems decision q'' to be more robust than decision q', then q'' has also a greater probability of success (satisficing the performance requirement).

## Some Basic Facts

Before we subject the proposed hypothesis to a formal analysis, let us list the basic facts about Info-Gap's robustness analysis and its relationship to a probabilistic model of success:

Fact 1: The probability of success, namely Pr_{U}[R(q,u) ≤ R_{c}], is determined by a probability distribution over the entire region of uncertaintyUso that potentially it hangs on the performance of decision q relative to the entire regionU. The key point is that the probability of success is aglobalproperty insofar asUis concerned. The picture is this:

û

Fact 2: Info-Gap's robustness analysis is conducted in the neighborhood of a point estimate of the parameter of interest. So generally, Info-Gap's robustness analysis is completely oblivious to the performance of decision q in large regions ofU. This is particularly striking ifUisunbounded. The key point is that "Info-Gap robustness" is alocalproperty insofar asUis concerned. The picture is this:

No Man's Land û No Man's Land

Fact 3: Under severe uncertainty, the point estimate is a wild guess of the true value of u and is likely to be substantially wrong. The picture is this:

No Man's Land û No Man's Land u* where u* denotes the true value of u.

Fact 4: There are no grounds whatsoever to proceed from an assumption that it is highly probable that the true value of u is in the immediate value of û. In fact, given the severity of the uncertainty assumed by Info-Gap decision theory, there is no reason to believe that the probability of u overUhas high peaks.We are now ready for a formal examination of the proposed hypothesis.

## Anatomy of an Hypothesis

To examine the hypothesis more closely, consider the following three general obvious observations:

- It is straightforward to show — by means of a simple counter example — that this hypothesis is in general invalid, namely that situations exist where the hypothesis does not hold.

- It is straightforward to show — prove formally — that the hypothesis is valid for some trivial degenerate cases.

- It is extremely easy to conduct numerical experiments in the framework of the specific application considered in the paper (maximizing the probability of a correct diagnosis of diabetes), so as to put this hypothesis to the test.
So, let us now examine each observation in greater detail.

Counter ExampleConsider the case shown in the following picture:

Info-Gap decision theory deems q'' to be more robust, but it is clear that q' dominates q'' over most of the uncertainty space

U=ℜ. The dark areas at the bottom of the picture show the regions ofUwhere the constraint is satisfied by these two decisions.So, there are no too ways about it, q' is far more robust than q'' relative to the complete uncertainty space

U.It follows then that if the probability of u is not "concentrated" in the two very small intervals where q'' dominates q', the hypothesis is not valid. For example, if Pr

_{U}[u∈U(α(q'',û)] is sufficiently small, then clearly the hypothesis does not hold.It can be shown (see FAQ # 62) that in some trivial degenerate cases Info-Gap's robustness in the neighborhood of an estimate — even a very bad one — is a "proxy" for the true robustness over the entire uncertainty space

Degenerate Valid ExampleU. In such cases — which are of little interest, if any — the hypothesis is valid regardless of the specific structure of the probabilistic model, namely Pr_{U}[event]. However, as explained in FAQ # 62, in these cases there is no need for the Info-Gap model in the first place. The results can be obtained directly from the performance constraint. The estimate û is just a nuisance parameter.The specific Info-Gap robustness model studied in the paper is so simple that closed-form solutions are readily available. Meaning that, it is extremely easy to test this hypothesis by conducting numerical experiments via simulation. It is unclear, therefore, why the authors did not bother to do this in order to see whether the hypothesis can have any merit.

Numerical Experiments## The "Why is the hypothesized plausible?" section

For obvious reasons the Journal requires that contributors explain why the radical hypotheses that they propose are plausible. This is done under the heading "Why is the hypothesized plausible?".

In this section of their paper the authors argue that their hypothesis is plausible because their Info-Gap analysis is based on two assumptions (page 4):

The info-gap models of uncertainty which underlie our example, Eqs. (5) and (6), assume two things: (i) that the cumulative distribution functions (CDFs) are normal (Gaussian) and (ii) that the estimated moments deviate fractionally by an unknown amount.They then proceed to argue that these assumptions are reasonable.

But the hypothesis is not just about the Info-Gap models proposed in the article. The whole point of the hypothesis is that it makes a statement about a relationship between Info-Gap's robustness model

anda probabilistic model of "success". So what should have been discussed under this heading is the plausibility of the assumed kinship between these two models.So as the readers cannot possibly gather this from this article, it is important to take note that contrary to the local analysis that is prescribed by Info-Gap's local robustness model, the probability of success is detemined by a

globalexpectation operator over the entire uncertainty spaceU.The picture is this:

- The green island represents the complete uncertainty space, namey
U.

- The black dot represents the estimate û of the true (unknown) value of the parameter of interest.

- The white circle represents the subset of
Uon which Info-Gap decision theory conducts its robustness analysis, namely U(α*,û), where α*:= max {α(q,û): q∈Q}, recalling that U(α,û) denotes the region of uncertainty of size α around the estimate û.

Let A =

Uand B = U(α*,û). Since the authors assume thatUisunbounded, B is actually an infinitesimally small subset of A. For illustrative purposes it is enlarged in this picture.Things to remember:

- Whereas Info-Gap decision theory conducts it business on A, the probabilistic model of success conducts its business on B.

- The estimate û is a wild guess and is likely to be substantially wrong.
Given that the black dot is a wild guess of the white square, to argue that an analysis of the small white circle is a proxy of an analysis of the entire green island is tatamount to practicing voodoo science.

In a nutshell, this glaring incongruity between Info-Gap's robustness model and models determining the probability of success not only brings out the implausibility of this hypothesis. It brings out that the hypothesis in fact violates the universally accepted GIGO Axiom.

## Conclusions

- The idea behind the proposed hypothesis is misguided: it asserts — without invoking relevant assumptions — that a local deterministic model yields the same results as a global probabilistic model.

- Unless stringent conditions are postulated to hold for the problem in question, there is no reason to believe that a local analysis in the neighborhood of a wild guess will yield a reliable result. This is precisely what the universally accepted GIGO Axiom dictates.

- Since no such conditions are postulated by the authors to hold for the specific problem under consideration, the authors' hypothesis is without any foundation.

- The comments in Review 6 and Review 9 regarding the likelihood-free nature of Info-Gap's robustness model are also relevant here.
- Ben-Haim (2009) discusses in detail the type of strong conditions that must be imposed (jointly) on the probability model and on Info-Gap's robustness model to justify the validity of the hypothesis.

It is therefore puzzling that no such conditions are discussed by the authors in this paper. Even more puzzling is the fact that the authors fail to cite Ben-Haim's (2009) paper and the results presented there regarding the connection between Info-Gap's robustness model and the probability of "success".

In short, we conclude that the future of probability theory and statistics ... is not in jeopardy ... not yet!

## Recommendation

In view of the above, I strongly recommended that Medical Hypothesis require authors to indicate clearly in their papers whether the hypotheses that they propose withstand the test of the GIGO Axiom. If this cannot be done, then an explanation should be provided.

Stay tuned ... there is more in store!

And something to think about in the meantime:

The authors use anunboundeduncertainty space. This means, among other things, that they assume that the fasting plasma glucose (FPG) concentration can be arbitrarily low and high!According to the information available to me

- A FPG concentration below 2 (mmo/l) is extremely low and there is a danger of unconciousness.

- A FPG concentration above 30 (mmo/l) indicates a high danger of severe electrolyte imbalance.

- Apparently some meters/strips cannot read FPG concentrations above 35 (mmo/l).
So isn't it safe to assume that the only relevant FPG concentrations are in the range, say [0,50] (mmo/l) rather than the impressive unbounded range (-∞,∞) used by the authors? Do you know anyone — alive or otherwise — with negative FPG concentration?

Any comments/suggestions on this issue will be greatly appreciated!

Note: a mole is approximately 6.022141796*10

^{23}molecules, and mmol/l = millimoles/liter. For blood glucose, the conversion from molecular count to weight is: mmol/l = 18 mg/dl (milligrams/deciliter).

## Other Reviews

- Ben-Haim (2001, 2006): Info-Gap Decision Theory: decisions under severe uncertainty.

- Regan et al (2005): Robust decision-making under severe uncertainty for conservation management.

- Moilanen et al (2006): Planning for robust reserve networks using uncertainty analysis.

- Burgman (2008): Shakespeare, Wald and decision making under severe uncertainty.

- Ben-Haim and Demertzis (2008): Confidence in monetary policy.

- Hall and Harvey (2009): Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.

- Ben-Haim (2009): Info-gap forecasting and the advantage of sub-optimal models.

- Yokomizo et al (2009): Managing the impact of invasive species: the value of knowing the density-impact curve.

- Davidovitch et al (2009): Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island.

- Ben-Haim et al (2009): Do we know how to set decision thresholds for diabetes?

- Beresford and Thompson (2009): An info-gap approach to managing portfolios of assets with uncertain returns

- Ben-Haim, Dacso, Carrasco, and Rajan (2009): Heterogeneous uncertainties in cholesterol management

- Rout, Thompson, and McCarthy (2009): Robust decisions for declaring eradication of invasive species

- Ben-Haim (2010): Info-Gap Economics: An Operational Introduction

- Hine and Hall (2010): Information gap analysis of flood model uncertainties and regional frequency analysis

- Ben-Haim (2010): Interpreting Null Results from Measurements with Uncertain Correlations: An Info-Gap Approach

- Wintle et al. (2010): Allocating monitoring effort in the face of unknown unknowns

- Moffitt et al. (2010): Securing the Border from Invasives: Robust Inspections under Severe Uncertainty

- Yemshanov et al. (2010): Robustness of Risk Maps and Survey Networks to Knowledge Gaps About a New Invasive Pest

- Davidovitch and Ben-Haim (2010): Robust satisficing voting: why are uncertain voters biased towards sincerity?

- Schwartz et al. (2010): What Makes a Good Decision? Robust Satisficing as a Normative Standard of Rational Decision Making

- Arkadeb Ghosal et al. (2010): Computing Robustness of FlexRay Schedules to Uncertainties in Design Parameters

- Hemez et al. (2002): Info-gap robustness for the correlation of tests and simulations of a non-linear transient

- Hemez et al. (2003): Applying information-gap reasoning to the predictive accuracy assessment of transient dynamics simulations

- Hemez, F.M. and Ben-Haim, Y. (2004): Info-gap robustness for the correlation of tests and simulations of a non-linear transient

- Ben-Haim, Y. (2007): Frequently asked questions about info-gap decision theory

- Sprenger, J. (2011): The Precautionary Approach and the Role of Scientists in Environmental Decision-Making

- Sprenger, J. (2011): Precaution with the Precautionary Principle: How does it help in making decisions

- Hall et al. (2011): Robust climate policies under uncertainty: A comparison of Info--Gap and RDM methods

- Ben-Haim and Cogan (2011) : Linear bounds on an uncertain non-linear oscillator: an info-gap approach

- Van der Burg and Tyre (2011) : Integrating info-gap decision theory with robust population management: a case study using the Mountain Plover

- Hildebrandt and Knoke (2011) : Investment decisions under uncertainty --- A methodological review on forest science studies.

- Wintle et al. (2011) : Ecological-economic optimization of biodiversity conservation under climate change.

- Ranger et al. (2011) : Adaptation in the UK: a decision-making process.

## Recent Articles, Working Papers, Notes

Also, see my complete list of articles

Moshe's new book! - Sniedovich, M. (2012) Fooled by local robustness,
Risk Analysis, in press.

- Sniedovich, M. (2012) Black swans, new Nostradamuses, voodoo decision theories and the science of decision-making in the face of severe uncertainty,
International Transactions in Operational Research, in press.

- Sniedovich, M. (2011) A classic decision theoretic perspective on worst-case analysis,
Applications of Mathematics,56(5), 499-509.

- Sniedovich, M. (2011) Dynamic programming: introductory concepts, in
Wiley Encyclopedia of Operations Research and Management Science(EORMS), Wiley.

- Caserta, M., Voss, S., Sniedovich, M. (2011) Applying the corridor method to a blocks relocation problem,
OR Spectrum, 33(4), 815-929, 2011.

- Sniedovich, M. (2011)
Dynamic Programming: Foundations and Principles, Second Edition,Taylor & Francis.

- Sniedovich, M. (2010) A bird's view of Info-Gap decision theory,
Journal of Risk Finance, 11(3), 268-283.

- Sniedovich M. (2009) Modeling of robustness against severe uncertainty, pp. 33- 42, Proceedings of the 10th International Symposium on Operational Research, SOR'09, Nova Gorica, Slovenia, September 23-25, 2009.

- Sniedovich M. (2009) A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds),
.Safety, Reliability and Risk Analysis: Theory, Methods and Applications,pp. 2071-2079, Taylor and Francis Group, London.

- Sniedovich M. (2009) A Classical Decision Theoretic Perspective on Worst-Case Analysis, Working Paper No. MS-03-09, Department of Mathematics and Statistics, The University of Melbourne.(PDF File)

- Caserta, M., Voss, S., Sniedovich, M. (2008) The corridor method - A general solution concept with application to the blocks relocation problem. In: A. Bruzzone, F. Longo, Y. Merkuriev, G. Mirabelli and M.A. Piera (eds.),
11th International Workshop on Harbour, Maritime and Multimodal Logistics Modeling and Simulation,DIPTEM, Genova, 89-94.

- Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)

- Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)

- Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08

This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file )

- Sniedovich, M. (2008) A call for the reassessment of Info-Gap decision theory,
Decision Point,24, 10.

- Sniedovich, M. (2008) From Shakespeare to Wald: modeling wors-case analysis in the face of severe uncertainty,
Decision Point,22, 8-9.

- Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!,
Journal of Risk Finance,9(3), 287-291.

- Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)

In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model.

- Sniedovich. M. (2008) The Mighty Maximin! (PDF File)

This paper is dedicated to the modeling aspects of Maximin and robust optimization.

- Sniedovich, M. (2007) The art and science of modeling decision-making under severe uncertainty,
Decision Making in Manufacturing and Services,1-2, 111-136. (PDF File) .

- Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)

In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory.

- My reply (PDF File) to Ben-Haim's response to one of my papers. (April 22, 2007)
This is an exciting development!

- Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.

- Ben-Haim is not familiar with the fundamental concept
point estimate. He does not realize that a function can be a point estimate of another function.So when you read my papers make sure that you do not misinterpret the notion

point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!

- A formal proof that
Info-GapisWald's Maximin Principlein disguise. (December 31, 2006)

This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)

It shows thatInfo-Gapis not a new theory but rather a simple instance ofWald's famous Maximin Principledating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context ofGame Theory(1928).

- A proof that
Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)

This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File) .

It shows that becauseInfo-Gapdeploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust.

- A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)

This is a very short article entitled The GAP in Info-Gap (PDF File) .

It is a math-free version of the paper above. Read it if you are allergic to math.

- A long essay entitled
What's Wrong with Info-Gap? An Operations Research Perspective(PDF File) (December 31, 2006).

This is a paper that I presented at theASOR Recent Advances in Operations Research(PDF File) mini-conference (December 1, 2006, Melbourne, Australia).## Recent Lectures, Seminars, Presentations

If your organization is promoting Info-Gap, I suggest that you invite me for a seminar at your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use — let alone promote — Info-Gap as a decision-making tool.

Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.

- ASOR Recent Advances, 2011, Melbourne, Australia, November 16 2011. Presentation: The Power of the (peer-reviewed) Word. (PDF file).

- Alex Rubinov Memorial Lecture The Art, Science, and Joy of (mathematical) Decision-Making, November 7, 2011, The University of Ballarat. (PDF file).

- Black Swans, Modern Nostradamuses, Voodoo Decision Theories, and the Science of Decision-Making in the Face of Severe Uncertainty (PDF File) .

(Invited tutorial, ALIO/INFORMS Conference, Buenos Aires, Argentina, July 6-9, 2010).

- A Critique of Info-Gap Decision theory: From Voodoo Decision-Making to Voodoo Economics(PDF File) .

(Recent Advances in OR, RMIT, Melbourne, Australia, November 25, 2009)

- Robust decision-making in the face of severe uncertainty(PDF File) .

(GRIPS, Tokyo, Japan, October 16, 2009)

- Decision-making in the face of severe uncertainty(PDF File) .

(KORDS'09 Conference, Vilnius, Lithuania, September 30 -- OCtober 3, 2009)

- Modeling robustness against severe uncertainty (PDF File) .

(SOR'09 Conference, Nova Gorica, Slovenia, September 23-25, 2009)

- How do you recognize a Voodoo decision theory?(PDF File) .

(School of Mathematical and Geospatial Sciences, RMIT, June 26, 2009).

- Black Swans, Modern Nostradamuses, Voodoo Decision Theories, Info-Gaps, and the Science of Decision-Making in the Face of Severe Uncertainty (PDF File) .

(Department of Econometrics and Business Statistics, Monash University, May 8, 2009).

- The Rise and Rise of Voodoo Decision Theory.

ASOR Recent Advances, Deakin University, November 26, 2008. This presentation was based on the pages on my website (voodoo.moshe-online.com).

- Responsible Decision-Making in the face of Severe Uncertainty (PDF File) .

(Singapore Management University, Singapore, September 29, 2008)

- A Critique of Info-Gap's Robustness Model (PDF File) .

(ESREL/SRA 2008 Conference, Valencia, Spain, September 22-25, 2008)

- Robust Decision-Making in the Face of Severe Uncertainty (PDF File) .

(Technion, Haifa, Israel, September 15, 2008)

- The Art and Science of Robust Decision-Making (PDF File) .

(AIRO 2008 Conference, Ischia, Italy, September 8-11, 2008 )

- The Fundamental Flaws in Info-Gap Decision Theory (PDF File) .

(CSIRO, Canberra, July 9, 2008 )

- Responsible Decision-Making in the Face of Severe Uncertainty (PDF File) .

(OR Conference, ADFA, Canberra, July 7-8, 2008 )

- Responsible Decision-Making in the Face of Severe Uncertainty (PDF File) .

(University of Sydney Seminar, May 16, 2008 )

- Decision-Making Under Severe Uncertainty: An Australian, Operational Research Perspective (PDF File) .

(ASOR National Conference, Melbourne, December 3-5, 2007 )

- A Critique of Info-Gap (PDF File) .

(SRA 2007 Conference, Hobart, August 20, 2007)

- What exactly is wrong with Info-Gap? A Decision Theoretic Perspective (PDF File) .

(MS Colloquium, University of Melbourne, August 1, 2007)

- A Formal Look at Info-Gap Theory (PDF File) .

(ORSUM Seminar , University of Melbourne, May 21, 2007)

The Art and Science of Decision-Making Under Severe Uncertainty(PDF File) .

(ACERA seminar, University of Melbourne, May 4, 2007)

- What exactly is Info-Gap? An OR perspective. (PDF File)

ASOR Recent Advances in Operations Researchmini-conference (December 1, 2006, Melbourne, Australia).

Disclaimer:This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.

Last modified: [an error occurred while processing this directive]