The Spin Stops Here!
  Decision-Making Under Severe Uncertainty
Faqs | Help | @ | Contact | home  
voodoostan info-gap decision theory info-gap economics severe uncertainty mighty maximin robust decisions responsible decisions

Reviews of publications on Info-Gap decision theory

Last modified: Wednesday, 24-Feb-2010 20:17:09 MST

On this page I comment on publications dealing with Info-Gap decision theory. Although this commentary is intended primarily for the benefit of the authors and reviewers of these publications, it clearly lends support to and justifies the rationale behind my Info-Gap Campaign.

Requests for reviews of Info-Gap publications that are not currently on the list will be considered.

  Latest News  
       

Overview

Over the past two years I have been asked, on several occasions, the following intriguing question:

Given that the flaws in Info-Gap decision theory are so obvious and grave, how do you explain the large number of publications on Info-Gap, particularly the significant number of articles published by refereed journals?

My usual answer to this question is as follows:

I'll be happy to discuss this matter with you over a cup of coffee (skinny latte, please!)

I do not plan to deviate from this answer here.

However, I believe that the reviews that I provide here will give you a good idea of what I may tell you over a cup of coffee so, try to read between the lines. And, for a fuller appreciation of one of the central points that I am making in this collection of reviews, read the discussion on fog, spin and rhetoric. Because, as you will see, whatever Info-Gap decision theory lacks in substance and rigor, is made up for, and covered up, by heavy fog, spin, and rhetoric.

I use the acronym TUIGF as shorthand for "The usual Info-Gap flaws". These flaws are discussed in detail in my compilation of FAQs about Info-Gap so there is no need to elaborate on them in these reviews. On the other hand, they will be used as benchmarks.

I also use the acronym SNHNSNDN as shorthand for "see nothing, hear nothing, say nothing, do nothing". This refers to authors who, for various reasons, choose to ignore my criticism of Info-Gap decision theory, particularly my theorems proving that Info-Gap's robustness model is a Maximin model (circa 1940) and Info-Gap's opportuneness model is a Minimin model (circa 1952). For your convenience and enjoyment these beauties are displayed at the bottom of the page.

So, recall that Info-Gap's two main fundamental flaws are:

The question therefore arises: how is it that so many Info-Gap papers have been published in refereed journals? Surely, the flaws are so obvious that any referee with a basic knowledge of decision-making under severe uncertainty should have been able to identify them.

Indeed, even more basic is the point that Info-Gap scholars who should know better continue to publish Info-Gap papers, repeating the same unsubstantiated claims, again and again and again.

The objective of my reviews is to show these publications for what they are.

Selection criteria

I do not have strict criteria for the selection of papers for review. For obvious reasons I prefer the more recent papers that were published in refereed journals. But I shall consider other papers, including working papers -- if they shed new light on on the failings of Info-Gap decision theory and its applications.

So the plan is to start with recently (2009) published papers, especially papers that attempt to deal with my criticism.

If you have come across a publication that you think ought to be reviewed, please send me a copy of it and I shall have a look at it to see if it is suitable for this project.

How to read the reviews

I realize that the potential readership of my reviews may be diverse with various interests and degrees of knowledge of decision theory in general and Info-Gap decision theory in particular.

My aim is to address my reviews, in the first instance, to the authors of the articles, the referees who reviewed them, the editors of the journals who accepted the papers for publication, and the bodies that provided the funds to support the work on which the articles are based. So, please read the reviews with this point in mind.

Any assessment of Info-Gap's contribution to the state of the art in decision theory must be made against these claims by Ben-Haim (2001, 2006) — the Father of Info-Gap:

Info-gap decision theory is radically different from all current theories of decision under uncertainty. The difference originates in the modeling of uncertainty as an information gap rather than as a probability. The need for info-gap modeling and management of uncertainty arises in dealing with severe lack of information and highly unstructured uncertainty.

Ben-Haim (2006, p. xii)

and
In this book we concentrate on the fairly new concept of information-gap uncertainty, whose differences from more classical approaches to uncertainty are real and deep. Despite the power of classical decision theories, in many areas such as engineering, economics, management, medicine and public policy, a need has arisen for a different format for decisions based on severely uncertain evidence.

Ben-Haim (2006, p. 11)

So my reviews address primarily the following two questions:

As indicated above, the fact of the matter is that Info-Gap decision theory fails on both counts. So, it should come as no surprise that my criticism of Info-Gap decision theory is very harsh indeed.

The following mini-guide may be useful to some readers.

Reader type Tips
Author Please, read the relevant FAQs cited in my reviews before you send me an angry letter. I'd be most happy to discuss your paper with you.
Referee If you were not vigilant enough, or not sufficiently au fait with the literature on decision-making under severe uncertainty, make sure that next time you are better prepared before you proceed to referee an Info-Gap paper. In any event, take note that Info-Gap's fundamental flaws cannot be smoothed over by fancy rhetoric.
Editor Make sure that you send Info-Gap papers to qualified referees. Please, do not send me Info-Gap papers for review.
Info-Gap aficionado Relax! You can ignore my criticism. This is in fact what Info-Gap scholars -- with some exceptions -- have done so far.
Inactive Info-Gap Critic Wake up! Join my campaign.
PhD Student Indeed, not everything that gets published in refereed journals is "correct". Also, PhD supervisors are not always well-versed in the subject matter and can fall victim to the adverse effects of spin and rhetoric. Therefore, study carefully the claims that are made in the Info-Gap literature.
Grantor of research funds Read carefully the research proposals. If a proposal is too good to be true, then rest assured that it is. In the case of decision-making under severe uncertainty, watch out for voodoo decision theories that are in violations of the GIGO Axiom, but are dressed up by spin.
Accidental tourist It would be a good idea to read the first 20 items in FAQs about Info-Gap decision theory before reading the reviews.

I shall expand the list as required as we continue with this project. If you can't wait, send me a note about your very special Reader-type.

Reviews

The current list is short, but I plan to expand it. There are a number of new refereed Info-Gap articles that are now (April 5, 2009) in press, so I'll consider them once they are published.

  1. Ben-Haim (2001, 2006): Info-Gap Decision Theory: decisions under severe uncertainty.

  2. Regan et al (2005): Robust decision-making under severe uncertainty for conservation management.

  3. Moilanen et al (2006): Planning for robust reserve networks using uncertainty analysis.

  4. Burgman (2008): Shakespeare, Wald and decision making under severe uncertainty.

  5. Ben-Haim and Demertzis (2008): Confidence in monetary policy.

  6. Hall and Harvey (2009): Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.

  7. Ben-Haim (2009): Info-gap forecasting and the advantage of sub-optimal models.

  8. Yokomizo et al (2009): Managing the impact of invasive species: the value of knowing the density-impact curve.

  9. Davidovitch et al (2009): Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island.

  10. Ben-Haim et al (2009): Do we know how to set decision thresholds for diabetes?

  11. Beresford and Thompson (2009): An info-gap approach to managing portfolios of assets with uncertain returns

  12. Ben-Haim, Dacso, Carrasco, and Rajan (2009): Heterogeneous uncertainties in cholesterol management

  13. Rout, Thompson, and McCarthy (2009): Robust decisions for declaring eradication of invasive species

Let me know of any articles/books that you think should appear on this list.

In addition to a brief commentary on the Info-Gap content of the publications, I also provide TUIGF, SNHNSNDN and GIGO scores (%).

Review # 1 (Posted: April 5, 2009; Last update: April 10, 2009)

Reference: Ben-Haim, Y.
Info-Gap Decision Theory: decisions under severe uncertainty
Academic Press, 2006
Information-Gap Decision Theory: decisions under severe uncertainty
Academic Press, 2001
Scores TUIGF:100%
SNHNSNDN:200%
GIGO:100%

These are the two editions of the Info-Gap book. I shall refer to the second edition.

A number of my colleagues who are also critical of Info-Gap decision theory have long suggested that I review this book. However, at present I do not plan to do this. The reason is simple: the book abounds with so much empty rhetoric and spin that an "official" review will require concentrating on the rhetoric to the detriment of a detailed treatment of the more urgent failings of this theory. At this stage, therefore, I prefer to focus only on the theoretical, methodological and technical aspects of Info-Gap decision theory.

Readers who are interested to see how spin and hollow rhetoric play out in this literature can visit other pages on my site where some references are made to this aspect of the Info-Gap enterprise:

The comprehensive compilation FAQs about Info-Gap can be viewed as a review of the scientific aspects of Info-Gap decision theory. Please note that a PDF file of this compilation is available. This compilation will serve as core material for my planned book "A Critique of Info-Gap Decision Theory". The spin and rhetoric aspects of Info-Gap will be discussed in the planned manuscript "The Rise and Rise of Voodoo Decision-Making".

So, stay tuned ...

Note:
The super-high SNHNSNDN scored by the book is due to the erroneous analysis of the relationship between Info-Gap and Maximin, and the claim that Info-Gap's robustness analysis is not a Maximin analysis (page 93).

Review # 2 (Posted: April 11, 2009; Last update: April 14, 2009)

Reference: HELEN M. REGAN, YAKOV BEN-HAIM, BILL LANGFORD, WILLIAM G. WILSON, PER LUNDBERG, SANDY J. ANDELMAN, AND MARK A. BURGMAN
Robust decision-making under severe uncertainty for conservation management.
Ecological Applications, 15(4), 1471-1477, 2005.
Scores TUIGF:100%
SNHNSNDN:200%
GIGO:100%
Acknowledgement: We thank Dan Berleant, Mark Colyvan, Martin Drechsler, John Harwood, Tom Hobbs, and an anonymous reviewer for useful discussions and comments that assisted in the preparation of this paper. This work was conducted as part of the Setting priorities and making decisions for conservation risk management Working Group supported by the National Center for Ecological Analysis and Synthesis, a Center funded by NSF (Grant #DEB-0072909), the University of California, and the Santa Barbara campus.

This is a relatively old paper. Still, it is on my list because, to the best of my knowledge, it is the first publication discussing the application of Info-Gap in applied ecology. So, it may shed some light on how the uncritical use of Info-Gap decision theory evolved in this discipline.

The rather high SNHNSNDN score reflects the authors' distorted portrayal of the tools provided by classical decision theory for the treatment of severe uncertainty. Consider this (page 1471, emphasis is mine):

For decision-making under uncertainty, the usual procedure is to assign probabilities to each of the relevant states and utilities to each of the outcomes. The approach usually taken is to maximize expected utility.

Clearly, the implication is that classical decision theory has no other means to offer for the treatment of "unusual" non-probabilistic cases. Or, that using other procedures for this purpose is not common practice in classical decision theory, or something to this effect. The inference, therefore, is that given classical decision theory's manifest shortcomings, one turns to Info-Gap decision theory to fill in the gap!

But the point is that other than making this sweeping statement the authors do not bother to show this to be the case. This is unfortunate because a quick look at one of the main references cited by the authors, namely Resnik (1987), reveals that Chapter 2 is entitled Decisions Under Ignorance. It is divided into the following sections:

Chapter 2: Decisions Under Ignorance

The probabilistic methods of classical decision theory are discussed in Chapter 3: Decisions Under Risk: Probability.

Apparently, the authors did not realize that relevant to their discussion in Resnik (1987) is not Chapter 3: Decisions Under Risk: Probability, but rather Chapter 2: Decisions Under Ignorance!?

For, had they studied this chapter, namely Chapter 2, they would have realized that Info-Gap's robustness model is simply a Maximin model (see proof on-line at the bottom of the page and in WIKIPEDIA).

Then, on page 1473 we read this (emphasis is mine):

The power and novelty of the info-gap approach is in the ability to explore the sensitivity of the decision to a wide range of different types of parameter, functional, and structural errors and uncertainties simultaneously, given that we do not know the extent of uncertainty in the system at the outset.

Of course, the precise opposite is the case. The purported "power and novelty" attributed to Info-Gap decision theory are in fact its two major failings.

Here is a schematic representation of the situation:

û
<-------- U --- Complete Region of Uncertainty --- U -------->

The black area represents the complete region of uncertainty U and the white dot represents the estimate û of the parameter of interest.

Now, with this representation in front of us, I should call attention to the following: No assumption whatsoever must be made as to whether any particular value of u is more/less likely than any other value of this parameter. This is totally in line with the fact that Info-Gap decision theory not only makes no assumptions about "likelihood", it bans any talk of likelihood.

Furthermore, given that, as emphatically pointed out in this paper, the uncertainty is severe, we have not the slightest clue which value of u is the true value.

This means that to determine the robustness of a decision it is essential to establish how well the decision performs relative to the entire region of uncertainty U, or a proper approximation thereof.

To see then what Info-Gap's purported "powerful" approach accomplishes, consider the following schematic representation of the results generated by Info-Gap's robustness analysis for decision d. The red area around the estimate, denoted by U(α(d,û),û), represents the largest safe region around the estimate as determined by Info-Gap's robustness model.

No Man's LandûNo Man's Land
U(α(d,û),û)
Safe

Take special note of Info-Gap's No Man's Land. This is the subset of the complete region of uncertainty that Info-Gap's robustness model does not explore. Roughly, this is the collection of points in U that are not in the safe region U(α(d,û),û) around the estimate û.

This schematic representation vividly illustrates why I regard Info-Gap decision theory as a classic example of voodoo decision-making: rather than tackling the severity of the uncertainty under consideration, one is instructed to ignore it altogether.

In my lectures/seminars I often describe this fundamental flaw in Info-Gap decision theory as a "treasure hunt":

Treasure Hunt

  • The island represents the complete region of uncertainty under consideration (the region where the treasure is located).

  • The tiny black dot represents the estimate of the parameter of interest (estimate of the location of the treasure).

  • The large white circle represents the region of uncertainty pertaining to info-gap's robustness analysis.

  • The small white square represents the true (unknown) value of the parameter of interest (true location of the treasure).

So, basing our search plan on Info-Gap Decision Theory, we may zero in on the neighborhood of downtown Melbourne, while for all we know, the true location of the treasure may well be in the Middle of the Simpson desert, or perhaps just north of Brisbane.

Perhaps.

Note that in this picture the estimate is placed at a considerable distance from the true value of the parameter of interest. This, of course has a point and purpose. It depicts the contention (Ben-Haim 2006, pp. 280-281 ) that under severe uncertainty the estimate is a poor indication of the true value of the parameter and is likely to be substantially wrong.

Remark:

I have been accused by Info-Gap scholars for exaggerating the size of the No Man's Land relative to the size of the "safe" area around the estimate.

So let me explain, yet again, why — from a methodological point of view — the No Man's Land must be depicted as much larger than the "safe" area, in the schematic representations of the working of Info-Gap's robustness model.

According to Ben-Haim (2006, p. 210, color is mine):

Most of the commonly encountered info-gap models are unbounded.

The inference to be drawn then from this statement is that in most of the commonly encountered info-gap models, the "safe" area around the estimate is infinitely minuscule — in comparison to the complete region of uncertainty. Consequently, not only is it the case that my sketches do not exaggerate the flaw in Info-Gap decision theory. This flaw is so severe that it cannot possibly be exaggerated: the complete region of uncertainty is unbounded yet the robustness analysis is a priori confined to the neighborhood of a poor estimate that is likely to be substantially wrong.

More on this can be found in FAQ # 26 and FAQ # 71.

And last but not least, no reference whatsoever is made in the paper to the thriving area of optimization theory that deals specifically with robust decisions, namely Robust Optimization. For those who are unfamiliar with this field of optimization theory I should point out that Robust Optimization concerns itself with -- among other things -- non-probabilistic methods for robust decision-making under severe uncertainty. Not surprisingly, Wald's Maximin paradigm is used extensively in this field.

Review # 3 (Posted: April 5, 2009; Last update: April 5, 2009)

Reference: Atte Moilanen, Michael C. Runge, Jane Elith, Andrew Tyre, Yohay Carmel, Eric Fegraus, Brendan A. Wintle, Mark Burgman, Yakov Ben-Haim.
Planning for robust reserve networks using uncertainty analysis
Ecological modelling, 199, 115-124, 2006.
Abstract Planning land-use for biodiversity conservation frequently involves computer-assisted reserve selection algorithms. Typically such algorithms operate on matrices of species presence-absence in sites, or on species-specific distributions of model predicted probabilities of occurrence in grid cells. There are practically always errors in input data-erroneous species presence-absence data, structural and parametric uncertainty in predictive habitat models, and lack of correspondence between temporal presence and long-run persistence. Despite these uncertainties, typical reserve selection methods proceed as if there is no uncertainty in the data or models. Having two conservation options of apparently equal biological value, one would prefer the option whose value is relatively insensitive to errors in planning inputs. In this work we show how uncertainty analysis for reserve planning can be implemented within a framework of information-gap decision theory, generating reserve designs that are robust to uncertainty. Consideration of uncertainty involves modifications to the typical objective functions used in reserve selection. Search for robust-optimal reserve structures can still be implemented via typical reserve selection optimization techniques, including stepwise heuristics, integer-programming and stochastic global search.
Acknowledgement This work was conducted as a part of the Working Group on "Setting priorities and making decisions for conservation risk management", supported by the National Center for Ecological Analysis and Synthesis, a Center funded by NSF (Grant # DEB-94-21535), the University of California at Santa Barbara, and the State of California. A.M. acknowledges support from the Academy of Finland project #1206883 and The Finnish Center of Excellence Programme 2000-2005, grant #44887.
ScoresTUIGF:110%
SNHNSNDN:100%
GIGO:100%

This is a typical TUIGF article. The reason for it being on my list is the following quote (last paragraph of the discussion section, emphasis is mine):

In summary, we recommend info-gap uncertainty analysis as a standard practice in computational reserve planning. The need for robust reserve plans may change the way biological data are interpreted. It also may change the way reserve selection results are evaluated, interpreted and communicated. Information-gap decision theory provides a standardized methodological framework in which implementing reserve selection uncertainty analyses is relatively straightforward. We believe that alternative planning methods that consider robustness to model and data error should be preferred whenever models are based on uncertain data, which is probably the case with nearly all data sets used in reserve planning.

My favorite quote from this paper, however, is this (page 118)

Under severe uncertainty (Ben-Haim, 2006), we do not know the horizon of uncertainty, that is, we do not have a meaningful estimate of the error, but info-gap theory still allows us to specify an uncertainty model.

In other words, in spite of the fact that you do not have a meaningful estimate of the error, you somehow construct a meaningful quantitative uncertainty model.

Furthermore, Info-Gap decision theory prescribes using this very model to determine the robustness of decisions against severe uncertainty, so as to ultimately select the best -- most robust -- decision. So, following Info-Gaps prescription, the proposition made by this article is that an analysis in the neighborhood of the poor estimate will enable you to identify decisions that are robust against severe uncertainty! It is as if the authors have never heard of the GIGO Axiom.

What the authors do not tell us is this: Info-Gap's uncertainty model does not quantify the uncertainty under consideration. It merely imposes on the complete region of uncertainty an ad hoc distance function that stipulates the distance of points in the regions of uncertainty from the given poor estimate. This distance tell us nothing about the uncertainty. The fact that u' is closer to the estimate than u'' says nothing about the uncertainty associated with u' and u''. For instance, it does not tell us that u' is more/less likely than u''.

So what is the rationale for a priori restricting the robustness analysis to the neighborhood of the poor estimate?

Interestingly, this fixation with the neighborhood of the estimate means that not only are Black Swans totally outside the purview of Info-Gap decision theory, which as you will recall, is purportedly designed to tackle the severest (Knightian) uncertainty imaginable. It can't even deal with ordinary, plain, snow-white swans.

The extra 10% for the TUIGF score is awarded for the recommendation to adopt ".. info-gap uncertainty analysis as a standard practice in computational reserve planning ..."

Review # 4 (Posted: April 5, 2009; Last update: April 14, 2009)

Reference: Burgman, M.
Shakespeare, Wald and decision making under severe uncertainty.
Decision Point (23), p. 10
October, 2008
ScoresTUIGF:100%
SNHNSNDN:0%
GIGO:120%

This is Burgman's response to my article "From Shakespeare to Wald: modelling worst-case analysis in the face of severe uncertainty" (Decision Point 22, 2008, pp. 8-9)

I had already responded to Burgman's (2008) paper elsewhere on my web site, see

Info-Gap decision theory and the small applied world of environmental decision-making
PDF version                   HTML version

But for your convenience I mention it briefly here. You can regard this as a "review" of Burgman's (2008) article.

For an idea as to the kind of objections that Burgman (2008) raises to my criticism of Info-Gap decision theory, consider this.

Burgman's (2008) seeks to portray my criticism of Info-Gap decision theory as being no more than semantic. For example, he criticizes me for misrepresenting the decision problems in the conservation biology and applied ecology literature. He points out that in my paper I formulate this questions as follows:

How wrong can I be, yet get an acceptable level of performance?

He then goes on to claim

But this question wasn't asked in the first place. Rather, the question asked in various applications of Info-gap has been the following: How wrong can this model be, without jeopardizing an acceptable level of performance?

So, I had to remind him that my formulation is based on that in Halpern et al (2006, p. 2), a formulation of the question that reads as follows:

How wrong can one be and still get an acceptable result?

I also had to remind him that this is not the question that is in fact addressed by Info-Gap's robustness model. Rather, Info-Gap's robustness model addresses the following question

How much can I deviate from a given estimate so that the performance requirement is satisfied throughout the region of uncertainty (around the estimate) which is contained within this deviation?

The point is that my criticism of Info-Gap raises the ire of its scholars/users because it shows this theory for what it is: a voodoo decision theory par excellence. Specifically, my unrelenting hammering of the obvious that, measured as it is in the neighborhood of the point estimate û, Info-Gap's robustness is local. Thus, in the context of severe uncertainty, where the estimate is assumed to be a wild guess, or at best a mildly wild guess, Info-Gap's analysis amounts to a voodoo analysis.

It is important to read my response to Burgman's (2008) comments because these give a good indication of how deep-rooted are the misconception about what Info-Gap decision theory promises on one hand and what it actually delivers on the other.

The high GIGO score is for the assertion that my definition of "severe uncertainty" is " ... too narrow to be useful ...". As indicated in my response,

The low SNHNSNDN score is in recognition of Burgman's (2008) reference to my claim that Info-Gap's robustness model is a Maximin model.

Review # 5 (Posted: April 20, 2009; Last update: May 17, 2009)

Reference: Ben-Haim, Y. and Demertzis, M.
Confidence in monetary policy.
DNB Working Paper No. 192 (PDF file)
December, 2008
Abstract In situations of relative calm and certainty, policy makers have confidence in the mechanisms at work and feel capable of attaining precise and ambitious results. As the environment becomes less and less certain, policy makers are confronted with the fact that there is a trade-off between the quality of a certain outcome and the confidence (robustness) with which it can be attained. Added to that, in the presence of knightian uncertainty, confidence itself can no longer be represented in probabilistic terms (because probabilities are unknown). We adopt the technique of Info-Gap Robust Satisficing to first define confidence under Knightian uncertainty, and second quantify the trade-off between quality and robustness explicitly. We apply this to a standard monetary policy example and provide Central Banks with a framework to rank policies in a way that will allow them to pick the one that either maximizes confidence given an acceptable level of performance, or alternatively, optimize performance for a given level of confidence.
ScoresTUIGF:100%
SNHNSNDN:5000%
GIGO:100%

This is a working paper of De Netherlandshe Bank. It is on my list due to its extremely high SNHNSNDN score. I shall therefore concentrate only on this aspect of the paper.

The objective of this review is twofold. Firstly, my aim is to make it clear that the authors' distinction between what they term "minimax strateries" and the "info-gap robust satisficing" strategy, that culminates in their proposition to adopt the "info-gap robust satisficing" strategy as a framework for formulating monetary policy, stems from a grossly erroneous, hence thoroughly misleading conception of the Maximin/Minimax paradigm.

Secondly, to show that the explanation that purportedly identifies "fundamental distinctions" between the so called "minimax strateries" and the "info-gap robust satisficing" strategy amounts to no more than hollow rhetoric and pure spin.

For the benefit of readers who are new to my Web-site, I ought to point out that this grossly erroneous distinction is of a piece with the broader thesis that right from the start proclaimed Info-Gap as new and distinctly different as follows (emphais is mine):

Info-gap decision theory is radically different from all current theories of decision under uncertainty. The difference originates in the modeling of uncertainty as an information gap rather than as a probability. The need for info-gap modeling and management of uncertainty arises in dealing with severe lack of information and highly unstructured uncertainty.

Ben-Haim (2006, p. xii)

and
In this book we concentrate on the fairly new concept of information-gap uncertainty, whose differences from more classical approaches to uncertainty are real and deep. Despite the power of classical decision theories, in many areas such as engineering, economics, management, medicine and public policy, a need has arisen for a different format for decisions based on severely uncertain evidence.

Ben-Haim (2006, p. 11)

So, as you can see, Info-Gap decision theory is being promoted as nothing short of a "revolutionary" theory indeed, a breakthrough in decision theory.

It is important therefore to be clear that not only is it the case that Info-Gap's robustness model is neither new nor radically different from classical models, but that it is in fact a special case of the most famous paradigm in classical decision theory for the treatment of severe uncertainty.

In other words, to pull the rug out from under the proposition that the justification for using Info-gap's robustness model is due to the "fundamental distictions" between this model and Minimax/Maximin, it is important to expose the profound misrepresentation, in this paper, of the relation between Info-Gap decision theory and Wald's Maximin model.

To begin with, I ought to point out that Ben-Haim — the creator of Info-Gap decision theory — is fully aware of my criticism of Info-Gap decision theory, notably the short theorem proving that Info-Gap's robustness model is in fact a simple instance of the Maximin model (see the Maximin Theorem at the bottom of the page). Ben-Haim does not dispute the validity of this theorem.

And yet, Ben-Haim and Demertzis do not hesitate to make the following claims (2008, p. 17, emphasis is mine):

Info-gap robust-satisficing is motivated by the same perception of uncertainty which motivates the min-max class of strategies: lack of reliable probability distributions and the potential for severe and extreme events. We will see that the robust-satisficing decision will sometimes coincide with a min-max decision. On the other hand we will identify some fundamental distinctions between the min-max and the robust-satisficing strategies and we will see that they do not always lead to the same decision.

In other words, without taking issue with, or rebutting, or explicitly commenting on the theorem which decisively proves that Info-Gap's robustness model is a run-of-the mill Maximin model, the authors philosophize at length on the distinctions between Info-Gap's robustness model and the so called 'min-max' strategies. Worse, in what seems like a maneovour aimed at ducking their scholarly duty to take on the theorem directly, the authors refer to 'min-max' strategies rather than to Maximin strategies. Worse yet, without bothering to give a formal definition to the 'min-max' model that the authors presumably compare to Info-Gap's robustness model, they go on about the purportedly different results yielded by the purportedly different strategies.

I shall not address these points here, as a detailed analysis of the misconceptions pervading Ben-Haim's understanding of the relationship between Info-Gap decision theory and the Maximin paradigm can be found in a number of my articles (eg. Sniedovich (2007), Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model) as well as in Faqs about Info-Gap Decision Theory and in WIKIPEDIA.

But to give you an idea of how profound these misconceptions are, consider the first purported fundamental distinction that the authors identify between Info-Gap and 'min-max' (2008, p. 17):

First of all, if a worst case or maximal uncertainty is unknown, then the min-max strategy cannot be implemented. That is, the min-max approach requires a specific piece of knowledge about the real world: "What is the greatest possible error of the analyst's model?". This is an ontological question: relating to the state of the real world. In contrast, the robust-satisficing strategy does not require knowledge of the greatest possible error of the analyst's model. The robust-satisficing strategy centers on the vulnerability of the analyst's knowledge by asking: "How wrong can the analyst be, and the decision still yields acceptable outcomes?" The answer to this question reveals nothing about how wrong the analyst in fact is. The answer to this question is the info-gap robustness function, while the true maximal error may or may not exceed the info-gap robust satisficing. This is an epistemic question, relating to the analyst's knowledge, positing nothing about how good that knowledge actually is. The epistemic question relates to the analyst's knowledge, while the ontological question relates to the relation between that knowledge and the state of the world. In summary, knowledge of a worst case is necessary for the min-max approach, but not necessary for the robust-satisficing approach.

The authors are badly in error regarding the relation of these two facts:

It is important to note that, the second fact in no way implies that the domains on which Maximin/Minimax models operate are required to be bounded. For example, consider the following classic Minimax model, exhibiting one of the most famous saddle points on Planet Earth:

 p := min  max  x2 - y2
x∈ℜy∈ℜ

ℜ = real line.

The optimal solution to this simple Minimax problem is the saddle point (x,y) = (0,0), yielding p=0. Note that the objective function here, namely the function f=f(x,y) defined by f(x,y)=x2 - y2, is unbounded on ℜ2.

But more than this, the unbounded vs. bounded horizons issue which presumably accounts for the difference between the two models is totally spurious. Indeed, insofar as Info-Gap's model is concerned, this point is utterly irrelevant. This is so because, insofar as Info-gap's model is concerned, it matters not in the slightest whether the horizon is bounded or unbounded. For, the robustness sought by Info-Gap's model is dictated by one consideration alone: the performance requirement, namely a constraint. Therefore, only two 'cases' need to be considered here:

In other words, Info-Gap's robustness model cares not one iota about the degree, the level, the point, or what have you, at which the constraint is violated. Insofar as Info-gap's robustness model is concerned, all violations of the constraints are 'worst-cases'. So, irrespective of whether the complete region of uncertainty is unbounded, there always is a worst-case.

In short, the authors are wrong — very wrong — in their assessment of what constitutes a 'worst-case' in a Maximin/Minimax formulation of Info-Gap's robustness model.

And to impress on the reader the heights to which rhetoric/spin is taken in this paper, here is an extended quote from the discussion on the relationship between Info-Gap robustness and Maximin — in fact 'min-max'. Clearly implicit in this dissertation is the fact that at least one of the authors (ie. Ben-Haim) is fully aware of the theorem that decisively shows that Info-Gap's robustness model can indeed be formulated as a simple Maximin model. But, in what seems a clear effort to avoid "direct contact" with the Maximin theorm the authors talk about "min-max" (one suspects rather than Maximin) as though "min-max" has got nothing to do with Maximin. What is the point of discoursing at length about differences between the two models without first of all proving this theorem to be false. But no attempt whatsoever is made to dispute the validity of this theorem. Instead, this is what you read in the paper:

4.2 Min-Max, Robust Control, and Robust-Satisficing

We take a brief intermezzo to compare the robust satisficing strategy with a class of alternatives. The term 'min-max', 'robust control' and 'worst-case' refer to a collection of decision strategies which attempt to ameliorate a maximally adverse outcome. This can of course be formulated in a variety of ways. In one way or another, whether explicitly or implicitly, a greatest level of uncertainty or a worst possible outcome is posited. Then a strategy is sought which maximally diminishes the impact of this outcome.

Info-gap robust-satisficing is motivated by the same perception of uncertainty which motivates the min-max class of strategies: lack of reliable probability distributions and the potential for severe and extreme events. We will see that the robust-satisficing decision will sometimes coincide with a min-max decision. On the other hand we will identify some fundamental distinctions between the min-max and the robust-satisficing strategies and we will see that they do not always lead to the same decision.

First of all, if a worst case or maximal uncertainty is unknown, then the min-max strategy cannot be implemented. That is, the min-max approach requires a specific piece of knowledge about the real world: "What is the greatest possible error of the analyst's model?". This is an ontological question: relating to the state of the real world. In contrast, the robust-satisficing strategy does not require knowledge of the greatest possible error of the analyst's model. The robust-satisficing strategy centers on the vulnerability of the analyst's knowledge by asking: "How wrong can the analyst be, and the decision still yields acceptable outcomes?" The answer to this question reveals nothing about how wrong the analyst in fact is. The answer to this question is the info-gap robustness function, while the true maximal error may or may not exceed the info-gap robust satisficing. This is an epistemic question, relating to the analyst's knowledge, positing nothing about how good that knowledge actually is. The epistemic question relates to the analyst's knowledge, while the ontological question relates to the relation between that knowledge and the state of the world. In summary, knowledge of a worst case is necessary for the min-max approach, but not necessary for the robust-satisficing approach.

The second consideration is that the min-max approaches depend on what tends to be the least reliable part of our knowledge about the uncertainty. Under Knightian uncertainty we do not know the probability distribution of the uncertain entities. We may be unsure what are typical occurrences, and the systematics of extreme events are even less clear. Nonetheless the min-max decision hinges on ameliorating what is supposed to be a worst case. This supposition may be substantially wrong, so the min-max strategy may be mis-directed.

A third point of comparison is that min-max aims to ameliorate a worst case, without worrying about whether an adequate or required outcome is achieved. This strategy is motivated by severe uncertainty which suggests that catastrophic outcomes are possible, in conjunction with a precautionary attitude which stresses preventing disaster. The robust-satisficing strategy acknowledges unbounded uncertainty, but also incorporates the outcome requirements of the analyst. The choice between the two strategies — min-max and robust-satisficing — hinges on the priorities and preferences of the analyst.

The fourth distinction between the min-max and robust-satisficing approaches is that they need not lead to the same decision, even starting with the same information.

Ben-Haim and Demertzis (2008, p. 17)

What this misguided thesis proves is not that Info-Gap's robustness model is not a Maximin model. What it does prove, though, is the authors' obvious misconceptions about the modeling aspects of the Maximin/Minimax paradigm, their misapprehension as to how the Maximin/Minimax paradigm is modeled, and so on. All this bars them from grasping the full extent of the affinity between Info-Gap's robustness model and Wald's Maximin model:

Maximin Theorem:

Info-Gap's Robustness model Corresponding instance of Wald's Maximin model
max {α ≥ 0: r(d,u) ≤ r* , ∀u∈U(α,û)}     ≡    
max min f(d,u,α,û)
  α ≥ 0     u∈U(α,û)  

where f(d,u,α,û) = α if r(d,u) ≤ r*; and f(d,u,α,û) = -∞, otherwise.

Proof of the Maximin Theorem:

Instance of Wald's Maximin Model Equivalent Math Programming formulation
max min f(d,u,α,û)
  α ≥ 0     u∈U(α,û)  
    ≡    
max { v: v ≤ f(d,u,α,û), ∀ u∈U(α,û) }
  α ≥ 0  
v ∈ ℜ
 
    ≡    
max { α: α ≤ f(d,u,α,û), ∀ u∈U(α,û) }
  α ≥ 0  
 
     ≡    
max   { α: α ≤ f(d,u,α,û), ∀ u∈U(α,û) }
 
     ≡    
max   { α: r(d,u) ≤ r*, ∀ u∈U(α,û) }    
Info-Gap's Robustness Model

So, the bottom line is this: Info-Gap's robustness model is a simple instance of Wald's famous Maximin model. And no amount of rhetoric/spin can change it. This simple instance of Wald's famous Maximin model always yields the same decision(s) that are yielded by Info-Gap's robustness model.

Note that The Maximin Theorem is constructive: it sets out a simple recipe for constructing the instance of the generic Maximin model that represents Info-Gap's robustness model.

No amount of rhetoric/spin can change this bottom line.

The conceptual and technical mistakes that led Ben-Haim astray in this matter are discussed in detail in the article Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model and in FAQ # 20.

Remark:

For the record, I should point out that Ben-Haim is fully aware of the existence of this theorem, what is more, that he does not dispute its validity. The situation is similar with respect to the Invariance Theorem.

It is therefore most regrettable, indeed inexcusable, that Ben-Haim has chosen to waltz around these theorems by means of spurious explanations to thereby extend already existing errors rather than admit to mistakes.

It will be interesting to see how long will Ben-Haim pursue this deliberate strategy of avoiding to deal with theorems that invalidate his repeated claims and pronouncements regarding Info-Gap's unique role and place in decision theory.

Review # 6 (Posted: April 5, 2009; Last update:April 14, 2009)

Reference: Hall, J. and Harvey, H.
Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis.
Eighth International Conference on Hydroinformatics
January 12-16, 2009, Concepcion, Chile.
(PDF file)
Abstract Flood risk analysis is subject to uncertainties, often severe, which have the potential to undermine engineering decisions. This is particularly true in strategic planning, which requires appraisal over long periods of time. Traditional economic appraisal techniques largely ignore this uncertainty, preferring to use a precise measure of performance, which affords the possibility of unambiguously ranking options in order of preference. In this paper we describe an experimental application of information-gap theory, or info-gap for short to a flood risk management decision. Info-gap is a quantified non-probabilistic theory of robustness. It provides a means of examining the sensitivity of a decision to uncertainty. Rather than simply presenting a range of possible values of performance, info-gap explores how this range grows as uncertainty increases. This allows considerably greater opportunity for insight into the behaviour of our model of option performance. The information generated may be of use in improving the model, refining the options, or justifying the selection of one option over the others in the absence of an unambiguous rank order. Secondly, we demonstrate the possibility of exploring the value of waiting until improved knowledge becomes available by constructing options that explicitly model this possibility.
Acknowledgement The work described in this publication was supported by the European Community's Sixth Framework Programme through the grant to the budget of the Integrated Project FLOODsite, Contract GOCE-CT-2004-505420. This paper reflects the authors' views and not those of the European Community. Neither the European Community nor any member of the FLOODsite Consortium is liable for any use of the information in this paper.
ScoresTUIGF:90%
SNHNSNDN:100%
GIGO:50%

I do not know for a fact that this is a refereed paper. Still, I consider it important for this project because, to the best of my knowledge, this is the first paper by Info-Gap proponents in which some sort of an attempt is made to deal with the fact that Info-Gap's robustness model engages in a ... voodoo analysis. I suspect that this attempt at rectifying the model is due to my criticism of Info-Gap — which, I am confident, is known to the authors.

So what exactly is it that I find so interesting in this paper?

Although this paper is afflicted with TUIGF, of particular interest is the following remarkable statement made immediately after the description of Info-Gap's regions of uncertainty, the horizon of uncertainty α and the estimate û of the parameter of interest (page 2, emphasis is mine):

An assumption remains that values of u become increasingly unlikely as they diverge from û.

In other words, this assumption indicates that the estimate û is the most likely value of the parameter u and that the likelihood of u decreases as u deviates from û. The inference clearly seem to be that this assumption is introduced to justify Info-Gap's fixation with the estimate as the focus of the robustness analysis.

Particularly noteworthy in the phrasing of this assumption is the word remains. What exactly are we to make of remain? Does it mean that in the context of Info-Gap, which boasts of being a non-probabilistic likelihood-free theory, this assumption was all along the case and it thus remains? If so how does it square with the "official" Info-Gap decision theory? Or, is this a "new" assumption? One that the authors decided to append to the "official" theory? If it is a newly added assumption then surely this must be made clear. Whatever the case, the authors must explain how this assumption tallies with the claim that the uncertainty under consideration is severe?

Three comments on this remarkable assumption:

In any event, despite its failure, this attempt at dealing with the fundamental flaw in Info-Gap's uncertainty model attests to the fact that some Info-Gap scholars begin to realize that -- in the absence of such assumptions -- Info-Gap decision theory cannot escape being a manifestly voodoo decision theory.

Remark: Attempts of this kind are doomed to fail simply because it is untenable to justify a local robustness analysis in the neighborhood of the estimate û and at the same time assume that the estimate is a poor indication of the true value of u and is likely to be substantially wrong. Info-Gap scholars would thus be well advised to consult the literature on Robust Optimization to see how severe uncertainty is modeled and managed without a priori localizing the analysis around a poor estimate. I also suggest that they remind themselves of the GIGO Axiom, and of course, the second challenge cited on the list of top challenges for making robust decision (emphasis is mine):

2. Managing uncertainty
Decisions depend on the best estimates of past performance, assessments of the current situation and visions into the future. Where the past performance may be known, the current is clouded by its immediacy and the future is a best guess. The robustness of any decision and the risk incurred in making that decision is only as good as the estimates on which it is based. Making estimation even more challenging, virtually all estimates that affect decisions are uncertain. Uncertainty can not be eliminated, but it can be managed.

Top Ten Challenges for Making Robust Decisions
The Decision Expert Newsletter™ -- Volume 1; Issue 2

But it is important to take note that one cannot postulate that the estimate û is so good that it is sufficient to conduct the robustness analysis in its neighborhood, and at the same time contend that the uncertainty in the true value of u is severe.

For the record, I should also point out that the article maintains the SNHNSNDN attitude towards the Maximin/Info-Gap connection. It continues to propagate the myth that Info-Gap decision theory is unique in its non-probabilistic, likelihood-free approach to the quantification and managing of uncertainty. Not only does this article fail to make the slightest mention of the famous Maximin paradigm, it takes what seems to be a rather evasive stance in this regard as born out by the assertion (page 2):

" ... All of these approaches rely upon some normalised measure being applied over the space of possibilities. Info-gap theory, but [sic] contrast does not employ normalised measure at all, so does not fit anywhere within the information-theoretic hierarchy of theories of uncertainty developed by Klir [9]. Rather, uncertainty is represented by a family of nested sets bounding the variation of system behaviour about some nominal value û. ..."

Why do the authors make do with the reference to Klir's article in the journal Reliability Engineering and Systems Safety? Why don't they refer to the theories developed by the founders of modern decision theory in the 1950s? In particular, why don't the authors refer to standard textbooks in decision theory, operations research, and robust optimization, where Wald's Maximin paradigm (circa 1940) is portrayed as the primary paradigm for decision-making under severe uncertainty? And why don't the authors refer to Sniedovich's (2007) proof that Info-Gap's robustness model is in fact a simple Maximin model?

How long can senior Info-Gap scholars keep their heads buried in the sand with regard to the Maximin/Info-Gap connection?

My view on this is rather cynical.

I believe that there is considerable pressure these days on academics to portray the methods that they use/develop as new and revolutionary. This is especially the case in applications for research grants. Indeed, what is the likelihood of your being awarded a grant if you state clearly in your grant application that your research will be based on an old mainstream theory that is described in undergraduate textbooks?

Relevant FAQs: 1-81, especially 13-19, 37, 78.

Review # 7 (Posted: April 5, 2009; Last update: May 2, 2009)

Reference: Ben-Haim, Y.
Info-gap forecasting and the advantage of sub-optimal models
European Journal of Operational Research, 197, 203-213, 2009.
Abstract We consider forecasting in systems whose underlying laws are uncertain, while contextual information suggests that future system properties will differ from the past. We consider linear discrete-time systems, and use a non-probabilistic info-gap model to represent uncertainty in the future transition matrix. The forecaster desires the average forecast of a specific state variable to be within a specified interval around the correct value. Traditionally, forecasting uses a model with optimal fidelity to historical data. However, since structural changes are anticipated, this is a poor strategy. Our first theorem asserts the existence, and indicates the construction, of forecasting models with sub-optimal-fidelity to historical data which are more robust to model error than the historically optimal model. Our second theorem identifies conditions in which the probability of forecast success increases with increasing robustness to model error. The proposed methodology identifies reliable forecasting models for systems whose trajectories evolve with Knightian uncertainty for structural change over time. We consider various examples, including forecasting European Central Bank interest rates following 9/11.
Scores TUIGF:100%
SNHNSNDN:200%
GIGO:100%

This is a typical Info-Gap article that repeats the standard errors, misconceptions and misleading information, associated with Info-Gap decision theory. It makes no reference whatsoever to the Maximin connection thus giving the reader the false impression that the model offered here is "different". It makes no reference whatsoever to the thriving literature on Robust Optimization thus depriving the reader of the knowledge about the wider context in which it belongs. But worse than all is the absence of any discussion on the localness of Info-Gap's robustness analysis. This gives the reader a thoroughly wrong idea of the results yielded by this analysis. I should therefore point out in this regard that it is this local treatment of severe uncertainty — especially "true" Knightian uncertainty — that makes Info-Gap decision theory a classic example of a voodoo decision theory.

To enable you to see through this paper, it will be useful to simplify its notation and consider a special case of the linear model formulated in it. So consider the dynamic system

      x(t+1) = Ax(t) , t=0,1,2,3, ... , k

where t denotes time, x(t) represents the state of the process (a vector) at time t and A is the transition matrix. The value of the initial state x(0) is given. We want to find the value of the terminal state x(k+1), in fact, the m-th component of this vector, xm(k+1).

The difficulty confronting us is that the transition matrix A is unknown: its true value is subject to severe uncertainty, in fact KNIGHTIAN uncertainty. All we have is an estimate of the true value of A, call it Ã. Note that we do not index matrix A by t because Ben-Haim (2009) assumes that although the value of A is unknown, it remains constant over time.

So, given this state-of-affairs we proceed to determine the value of a matrix of the same dimensions as A, call it B, to predict the final state of the process. Our prediction will thus be given by the value of y(k+1) generated by the system

      y(t+1) = By(t) , t=0,1,2,3, ... , k

with y(0)=x(0).

The error in the prediction is then

(*)       e = y(k+1) − x(k+1) = B k+1x(0) − Ak+1x(0) = (Bk+1 − Ak+1)x(0)

observing that this is a vector and that of interest to us is the m-th component of this vector, namely em.

The question is then: what is the best choice of B, given the initial vector x(0), the estimate à of A, and the severity of the uncertainty in the true value of A?

The argument made in the paper is that even if — according to some criterion — à is the best estimate of the true value of A, it is not always the case that à is the best choice for B. But this is a hollow argument as the paper tells us nothing about how the best estimate is determined. Specifically, the paper does not proceed from the assumption that the estimate à is "best" with respect to the error of interest, namely em. Therefore, to begin with, we would have had no reason to assume that à is the best choice for B. So, what is the point of arguing that à is not always the best choice for B?

In short, the first theorem in the paper makes a trivial argument. It "shows" that subject to some technical conditions, à is not the best choice for B. And to appreciate how pointless this theorem really is, it is sufficient to point out that according to the model presented in the paper the best choice for B is defined as a value of B that maximizes the size (α) of the region of uncertainty around à subject the following performance requirement:

      |Em(B,A)| ≤ ε , ∀ A ∈ U(α,Ã)

where Em(B,A) denotes the m-th component of the error vector e generated by B and A according to (*) and U(α,Ã) denotes the region of uncertainty of size α centered at Ã.

So, what we have in Ben-Haim's (2009) theorem is a classic example of what Jan Odhnoff (1965) argues in the last paragraph of his paper (my emphasis):

It seems meaningless to draw more general conclusions from this study than those presented in section 2.2. Hence, that section maybe the conclusion of this paper. In my opinion there is room for both 'optimizing' and 'satisficing' models in business economics. Unfortunately, the difference between 'optimizing' and 'satisficing' is often referred to as a difference in the quality of a certain choice. It is a triviality that an optimal result in an optimization can be an unsatisfactory result in a satisficing model. The best things would therefore be to avoid a general use of these two words.
Jan Odhnoff
On the Techniques of Optimizing and Satisficing
The Swedish Journal of Economics
Vol. 67, No. 1 (Mar., 1965)
pp. 24-39

I fully sympathize with Odhnoff's frustration.

Who on planet Earth expects an optimal solution to Problem P to be a feasible solution — let alone an optimal solution — to Problem Q where these two problems are substantially different from each other?!?!?!?!?!

Yet, this is precisely what the paper's title — "Info-gap forecasting and the advantage of sub-optimal models" — claims to show. The paper purportedly shows, in Theorem 1, that the "best" estimate à is not the best choice — namely is sub-optimal — for B when the objective is to maximize the robustness of B according to the Info-Gap prescription.

But, given that the paper does not give us even the slightest clue about the sense in which the estimate à is "best", what exactly is the merit of Theorem 1?

This is really incredible!

Next, let us take a quick look at the robustness model set out in the paper, that is the model according to which B values are ranked. It reads as follows (+) :

      α(B,Ã) = {α ≥ 0: |Em(B,A)| ≤ ε , ∀A∈U(α,Ã)}

The larger α(B,Ã) the better. Thus, the optimal value of B is the one that maximizes the value of α(B,Ã) with respect to B.

(+) Remark: There are serious "typographical" (?) errors in the paper in the expressions defining the regions of uncertainty (eq. (5), p. 204) and the robustness model (eq. (9), p. 205).

Two observations with respect to this robustness model:

These observations apply to all Info-Gap's robustness models.

I shall not bother you with the paper's other flaws, except to call attention to one that speaks volumes about this enterprise: the paper's short, uninformative, and unrepresentative list of references.

The list of references in Ben-Haim (2009) cites no more than 7 publications — a fact that hardly enables making a case for the claim that the paper offers a new forecasting methodology. Thus, although the topic under consideration is essentially about "robust decision making in the face of severe uncertainty", there is no reference in the paper to the very important and popular area of Operations Research called Robust Optimization. Worse, no references whatsoever is made in the paper to the state of the art in decision-making under severe uncertainty. No mention whatsoever is made of how classical decision theory, operations research, and robust optimization deal with robust decision-making under severe uncertainty. Consequently, on top of there being no indication that the proposed robustness model is a Maximin model, there is no discussion whatsoever on how, why, and in what in what sense, is the robustness model proposed in the paper new or different relative to robustness models used in classical decision theory and operations research.

It is important to remember, therefore, that over the past fifty years Maximin/Minimax models have become almost synonymous with robust decision-making not only in classical decision theory but in other areas as well. For instance, here is the abstract of the entry Robust Control by Noah Williams in the New Palgrave Dictionary of Economics, Second Edition, 2008:

Robust control is an approach for confronting model uncertainty in decision making, aiming at finding decision rules which perform well across a range of alternative models. This typically leads to a minimax approach, where the robust decision rule minimizes the worst-case outcome from the possible set. This article discusses the rationale for robust decisions, the background literature in control theory, and different approaches which have been used in economics, including the most prominent approach due to Hansen and Sargent.

The following quote is from the book Robust Statistics by Huber (1981, pp. 16-17):

But as we defined robustness to mean insensitivity with regard to small deviations from assumptions, any quantitative measure of robustness must somehow be concerned with the maximum degradation of performance possible for an e-deviation from the assumptions. The optimally robust procedure minimizes this degradation and hence will be a minimax procedure of some kind.

And, of course, it is important to call attention to the refereed papers (eg. Sniedovich (2007, 2008)) outlining formal proofs that Info-Gap's robustness model is a simple Maximin model. This proof is also available at WIKIPEDIA.

Remarks:

Review # 8 (Posted: April 5, 2009; Last update: April 14, 2009)

Reference: HIROYUKI YOKOMIZO, HUGH P. POSSINGHAM, MATTHEW B. THOMAS, AND YVONNE M. BUCKLEY
Managing the impact of invasive species: the value of knowing the density-impact curve
Ecological Applications, 19(2), 376-386, 2009.
Abstract Economic impacts of invasive species worldwide are substantial. Management strategies have been incorporated in population models to assess the effectiveness of management for reducing density, with the implicit assumption that economic impact of the invasive species will also decline. The optimal management effort, however, is that which minimizes the sum of both the management and impact costs. The relationship between population density and economic impact (what we call the "density-impact curve") is rarely examined in a management context and could take several nonlinear forms. Here we determine the effects of population dynamics and density-impact curves of different shapes on optimal management effort and discover cases where management is either highly effective or a waste of resources. When an inaccurate density-impact curve is used, the increase in total costs due to over -- or under investment in management can be large. We calculate the increase in total costs incurred if the density-impact curve is incorrect and find that the greater the maximum impact caused by an invasive species, the more important it is not only to reduce its density, but also to know the shape of the density-impact relationship accurately. Lack of information regarding the relationship between density and economic impact causes the most acute problems for invaders that cause high impact at low density, where management typically will be too little, too late. For species that are only problematic at high density, ignorance of the density-impact curve can lead to over investment in management with little reduction in impact.
Acknowledgement This work was supported by a Grant-in-Aid for Scientific Research of JSPS to H. Yokomiizo and by the Australian Research Council's "Discovery Projects" funding scheme (project number DP0771387) grant to Y. M. Buckley and additional ARC grants to H. P. Possingham. We thank Peter Baxter, Peter Brown, Céline Clech-Goods, John Dwyer, Jennifer Firn, Niels Hintzen, Andy Sheppard, and two anonymous referees for helpful comments.
Scores TUIGF:100%
SNHNSNDN:100%
GIGO:100%

This article does not deal with Info-Gap decision theory as such. Nevertheless, it is on my list to illustrate how the misconceptions about Info-Gap decision theory are disseminated in various disciplines, in this case applied ecology.

The complete reference to Info-Gap decision theory in this article is as follows (page 384, emphasis is mine):

An extension of the current study would be to determine the optimal management effort under uncertainty of the density-impact curve by, for example, assuming a probability distribution for the parameters of the density-impact relationship or information-gap decision theory (Ben-Haim 2001). Information-gap decision theory derives the most robust management option to meet a minimum performance requirement under severe uncertainty (Ben-Haim 2001, Regan et al. 2005).

Sounds great, doesn't it?

So, if this is your introduction to Info-Gap decision theory, you may interpret this reference to Info-Gap decision theory as follows:

An article in a refereed journal in the area of applied ecology describes Info-Gap decision theory as a theory that "... derives the most robust management option to meet a minimum performance requirement under severe uncertainty ...". This allows me to refer to Info-Gap in my research work and articles as the theory that actually delivers these goods, citing the above references as "proof".

No wonder then that Info-Gap decision theory is used uncritically in the areas of applied ecology and conservation biology. More on this can be found in my Decision Point articles (2008, issue 22, issue 24), in my online Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia, and in my article

Info-Gap decision theory and the small applied world of environmental decision-making
PDF version                   HTML version

Review # 9 (Posted: April 27, 2009; Last update: May 10, 2009)

Reference*: Lior Davidovitch, Richard Stoklosa, Jonathan Majer, Alex Nietrzeba, Peter Whittle, Kerrie Mengersen, Yakov Ben-Haim.
Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island
Journal of Environmental Management (in press, available online 21 April 2009)
Abstract Surveillance for invasive non-indigenous species (NIS) is an integral part of a quarantine system. Estimating the efficiency of a surveillance strategy relies on many uncertain parameters estimated by experts, such as the efficiency of its components in face of the specific NIS, the ability of the NIS to inhabit different environments, and so on. Due to the importance of detecting an invasive NIS within a critical period of time, it is crucial that these uncertainties be accounted for in the design of the surveillance system. We formulate a detection model that takes into account, in addition to structured sampling for incursive NIS, incidental detection by untrained workers. We use info-gap theory for satisficing (not minimizing) the probability of detection, while at the same time maximizing the robustness to uncertainty. We demonstrate the trade-off between robustness to uncertainty, and an increase in the required probability of detection. An empirical example based on the detection of Pheidole megacephala on Barrow Island demonstrates the use of info-gap analysis to select a surveillance strategy.
Acknowledgement This work was initiated during the "Workshop on Surveillance and Uncertainty", in Hobart, Tasmania, which was sponsored by the Australian Centre for Excellence in Risk Analysis. Support from the CRC for National Plant Biosecurity is also acknowledged.
Scores TUIGF:100%
SNHNSNDN:100%
GIGO:120%
* The publisher requests that this article in press be cited as follows:

Davidovitch, L., et al., Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island, Journal of Environmental Management (2009), doi:10.1016/j.jenvman.2009.03.011

This is a typical Info-Gap article that repeats the standard errors, misconceptions and misleading information, associated with Info-Gap. It makes no reference whatsoever to the Maximin connection thus giving the reader the false impression that the model offered here is "different". It makes no reference whatsoever to the thriving literature on Robust Optimization thus depriving the reader of the knowledge about the wider context in which it belongs. But worse than all is the absence of any discussion on the localness of Info-Gap's robustness analysis. This gives the reader a thoroughly wrong idea of the results yielded by this analysis. I should therefore point out in this regard that it is this local treatment of severe uncertainty — especially "true" Knightian uncertainty — that makes Info-Gap decision theory a classic example of a voodoo decision theory.

These aspects of Info-Gap decision theory are discussed in detail in Faqs about Info-Gap decision theory. So here I shall comment only on three issues.

1. Satisficing vs Optimizing

This article continues to perpetuate the myth of the superiority of "satisficing" vs "optimizing" — by now a fixture in the Info-Gap literature — apparently aimed to justify Info-Gap's so called "robust satisficing" strategy. "Optimal decisions" — so the argument goes — are inferior to "satisficing decisions" because optimal decisions are not robust, namely they have zero robustness (page 4, emphasis is mine):

Another immediate result is that the robustness of the optimal result — the maximal reward under our best estimate û — is not robust. In fact, it has zero robustness, meaning that a slight deviation from our estimation û may prevent us from meeting the requirement rc. Note that the optimal result also has zero opportuneness, since it is achieved without deviating from the estimate.

To see how erroneous hence badly misleading this contention is, let us first get clear on what this thesis actually maintains. Consider then the following two optimization problems:

Problem A     Problem B
Optimal Reward ProblemRobust Satisficing Problem
  z*(û):= max   R(q,û)  
q∈Q
  α(rc,û):= max   max {α≥0: R(q,u) ≥ rc , ∀u∈U(α,û)}  
q∈Q

Let q' denote the optimal decision in the framework of Problem A and let q'' denote the optimal decision in the framework of Problem B.

Thus, according to Info-Gap decision theory, the robustness of these decisions is, by definition, as follows:

α(q',rc,û):= max {α ≥ 0: R(q',u) ≥ rc , ∀u∈U(α,û)}

α(q'',rc,û):= max {α ≥ 0: R(q'',u) ≥ rc , ∀u∈U(α,û)}

Clearly then, since q'' is optimal with respect to Problem B, its robustness cannot be smaller than the robustness of q', namely α(q'',rc,û) = α(rc,û) ≥ α(q',rc,û).

But this does not imply that the robustness of q' is zero. Indeed, the following example shows that the authors' claim that q' has zero robustness meaning that a slight deviation from our estimation û may prevent us from meeting the requirement rc is false.

Counter Example # 1

Consider the case where u is a scalar, û=0, Q={q',q''}, U(α,û)={u: |u-û| ≤ α, |u|≤4}, rc = 0.5 and R(q,u) is as follows:

So q' is optimal with respect to Problem A and q'' is optimal with respect to Problem B. The robustness of q' is equal to 3.5 and the robustness of q'' is equal to 3.8. In short, not only that the robustness of q' is not equal to zero, it is actually on a par with the robustness of q''.

But to bring out more forcefully how absurd the authors' claim is, consider the case where the optimal decision yielded by Problem A is also optimal for Problem B. In such cases not only is the robustness of q' not equal to zero, q' is in fact the most robust decision!!!!!!!!!!!!!

The following example illustrates this obvious point.

Counter Example # 2

Consider the case where u is a scalar, û=0, Q={q(1),q(2)}, U(α,û)={u: |u-û| ≤ α, |u|≤4}, rc = 1 and R(q,u) is as follows:

Clearly, q(1) dominates q(2) for all u in U=[-4,4], hence q(1) is optimal for both Problem A and Problem B. Therefore, we set q'=q''=q(1).

So not only is it the case that the robustness of q' is not zero, q' is the most robust decision!

The following example is designed to illustrate how confusion reigns, in this article, over the difference between what the authors call "optimal reward" and the violation of the performance requirement R(q,u) ≥ rc.

Counter Example # 3

Consider the case where u is a scalar, û=0, Q={q(1),q(2)}, U(α,û)={u: |u-û| ≤ α, |u|≤4}, rc = 0.5 and R(q,u) is as follows:

Note that in the region [û,4], a slight decrease in the value of u decreases slightly the value of the reward R(q',u). But this does not imply that a slight change in u in the neighborhood of the estimate û will violate the performance requirement R(q',u) ≥ rc=0.5.

Also note that q'' has the same feature: in the region [-4,û], a slight increase in the value of u decreases slightly the reward R(q'',u). But this does not imply that a slight change in u in the neighborhood of the estimate û will violate the performance requirement R(q'',u) ≥ rc=0.5.

Regarding robustness, q' is almost as robust as q'', noting that α(q',rc,û)=1 and α(q'',rc,û)=1.108.

Clearly, the authors' claim is false in this case.

I take this opportunity to illustrate the real consequences of Info-Gap's robustness analysis. As you will recall, one of Info-Gap's main failings is that its robustness definition, hence analysis, hence verdicts, are local. This failing renders it thoroughly unsuitable for the treatment of severe uncertainty. The following example illustrates this point.

Counter Example # 4

Consider the case where u is a scalar, û=0, Q={q',q''}, U(α,û)={u∈ℜ: |u-û| ≤ α}, rc = 3 and R(q,u) is defined as follows:
R(q',u) := 4.2 + 6u2 -6|u|

R(q'',u) := 3.4 - 4u2

The picture is this:

Clearly, the authors' claim is false here.

Moreover, Info-Gap's robustness model selects q'' as the optimal (most robust) decision and assesses it to be more robust than q' despite the fact that R(q',u) > R(q'',u) almost everywhere in the unbounded region of uncertainty U=ℜ, the exception being the tiny intervals [-0.4,-0.2] and [0.2,0.4].

In short, Info-Gap decision theory ranks decisions according to their robustness (in the neighborhood of the estimate û) where robustness is determined with respect to the performance requirement R(q,u) ≥ rc. Therefore, in comparing the robustness of decisions, the same value must be assigned to rc in the comparison. Otherwise the comparison is rendered meaningless.

The only conclusion that can be drawn from all this is that Problem A typically yields decisions that generate larger rewards in the immediate neighborhood of the estimate, whereas Problem B typically yields decisions that are more robust in the neighborhood of the estimate. But this in no way implies that q' is inferior to q''. All that is implied here is that q' is different from q''. The only way to decide between q' and q'' would be by conducting a Pareto trade-off between reward and robustness that are representative of the complete region of uncertainty.

So the bottom line is this: if the problem under consideration is such that the stated objective is to maximize robustness subject to a performance requirement, then clearly an optimal decision that is obtained for a different problem altogether — for instance the maximization of the performance function — should not be expected to be optimal with respect to the problem under consideration. This triviality does not demonstrate that satisficing is "better" than optimizing (see discussion on this triviality in FAQ # 68).

The following quote is very relevant here. It is taken from the last paragraph of Jan Odhnoff's (1965) paper (emphasis is mine):

It seems meaningless to draw more general conclusions from this study than those presented in section 2.2. Hence, that section maybe the conclusion of this paper. In my opinion there is room for both 'optimizing' and 'satisficing' models in business economics. Unfortunately, the difference between 'optimizing' and 'satisficing' is often referred to as a difference in the quality of a certain choice. It is a triviality that an optimal result in an optimization can be an unsatisfactory result in a satisficing model. The best things would therefore be to avoid a general use of these two words.
Jan Odhnoff
On the Techniques of Optimizing and Satisficing
The Swedish Journal of Economics
Vol. 67, No. 1 (Mar., 1965)
pp. 24-39

I fully sympathize with Odhnoff's frustration.

And to round out the discussion on the muddled arguments that are often advanced in the Info-gap literature regarding the distinct differences between Info-Gap's so-called "robust satisficing" strategy and some alleged "direct optimization strategy"; and the presumed great merits attributed to Info-Gap "robust satisficing" strategy, consider again the formulation of the two problems in question:

Problem A     Problem B
Optimization of reward Optimization of robustness
z*(û):= max   R(q,û)
q∈Q
  α(rc,û):= max   max {α≥0: R(q,u) ≥ rc , ∀u∈U(α,û)}  
q∈Q

For one thing, as brought out by this juxtaposition, the depiction of Info-Gap's robustness model as a "satisficing model" — as opposed to an "optimization model" — is grossly misleading.

Info-gap's robustness model is defintely an optimization model, one that seeks to maximize robustness. For another, the great fuss about Info-Gap's "satisficing robustness" is equally misleading. There is nothing unusual in constraints being incorporated in optimization models. In fact, this is common practice in optimization. So if robustness is a consideration, it would be duly incorporated in the formulation of the optimization problem considered, either as an objective to be maximized or as a constraint to be satisfied.

Indeed, the Pareto-tradeoff conducted by Info-Gap decision theory can be carried out in two different ways:

Observe that in both cases the problem is a (constrained) optimization problem and not merely a "satisficing" problem. In particular, the problem represented by Info-Gap's robustness model — namely Problem B — is a constrained maximization problem: it involves maximizing the deviation (α) from the estimate subject to a performance constraint.

The irony is that often it is actually easier to conduct the Pareto-tradeoff analysis by treating performance as an objective and robustness as a constraint!

Sadly, the triviality regarding "satisficing vs optimizing" discussed by Jan Odhnoff (1965) more than forty years ago continues to be propagated in the Info-Gap literature, including the article under review here. This contributes nothing to good environmental management.

It is regrettable that this muddle continues to be propagated in the Info-Gap literature.

Remark: Since any "satisficing problem" can easily be re-formulated as an equivalent "optimizing problem", the issue here is not whether it is better to optimize or satisfice. Rather, the issue is what should be optimized and what should be satisficed.

2. Incorporating "beliefs" in Info-Gap's uncertainty model

Info-Gap decision theory prides itself on being non-probabilistic and likelihood-free (see Review 6). So on what grounds do the authors claim that (page 4, emphasis is mine):

As the horizon of uncertainty α gets larger, the sets U(α,û) become more inclusive. The info-gap model expresses the decision maker's beliefs about uncertain variation of u around û.

As indicated in Review 6, Ben-Haim bans any imputation of "likelihood" to Info-Gap's uncertainty model. So, how does this statement sit with Ben-Haim's position on likelihood?

But more than this, any imputation of "belief" to Info-Gap's uncertainty model is in stark contradiction to Ben-Haim's own exhortations explicitly banning "belief" from the model. For example, in his book, Ben-Haim (2006, p. 22) notes the following (emphasis is mine):

Uncertainty is the potential for deviation of an actual realization from its normative form. Neither norm nor any specific potential realization is uncertain; it is the potential for deviation of one from the other which is info-gap uncertainty.

The spatial analogy for info-gap uncertainty demonstrates that we need no concept of chance, frequency of recurrence, likelihood, plausibility or belief in order to speak of uncertainty.

Ben Haim (2006, p. 22)

And how about this?

Since the horizon of uncertainty is unknown and unbounded, there is no worst case. Since no measure functions of probability (or plausibility, or belief, etc.) are specified by an info-gap model, the analyst cannot calculate statistical expectations and cannot probabilistically insure against the unknown contingencies identified in the info-gap model.
Ben-Haim Y. and Jeske, K. (2003, p. 12)
Bias in Financial Markets: Robust Satisficing with Info Gaps
FRB of Atlanta Working Paper No. 2003-35.
Available at SSRN: http://ssrn.com/abstract=487585

And this:

Info-gap models are axiomatically utterly different from both probability and fuzzy logic, since info-gap models focus on the set-structure of uncertainty rather than on measure-theoretical representations. Info-gap models are particularly suited to representing sparse information since they make no assertions about frequencies of, or beliefs about, rare events.
Ben-Haim Y (2002, November 5)
Quote from the abstract of a seminar at MIT
entitled: Info-Gap Decision Theory For Design And Planning Or: Why 'Good' Is Preferable To 'Best'

And this:

Info-gap models are axiomatically utterly different from both probability and fuzzy logic, since info-gap models focus on the set-structure of uncertainty rather than on measure-theoretical representations. Info-gap models are particularly suited to representing sparse information since they make no assertions about frequencies of, or beliefs about, rare events.
Ben-Haim Y (2003, June 5)
Quote from the abstract of a seminar at Los Alamos National Lab
entitled: Info-gap decision theory for design and planning

Suppose, however, that we ignore this blatant contradiction.

What should be made explicit in the Info-Gap literature is the following: what "beliefs" are expressed by Info-Gap's robustness model? For instance, what "beliefs" are represented by say

U(α,û)={u∈ℜ: |u - û| ≤ α} , α ≥ 0

and where exactly are "beliefs" of this type described in the two Info-Gap books, or elsewhere in the Info-Gap literature?

Suppose, for example, that in the context of this model û=0 so that

U(α,û)={u∈ℜ: |u| ≤ α} , α ≥ 0

and that this model is based on our initial "beliefs" regarding uncertain variation of u around û=0. How would we modify this model to account for the fact that our "beliefs" regarding uncertain variation of u around û=0 have changed?

More importantly, where do we find, in the Info-Gap literature, guidelines instructing how to incorporate "beliefs" in the construction of Info-Gap's model of uncertainty?

As for the problem under investigation in the article, what exactly are the "beliefs" that are expressed by the regions of uncertainty defined in the article by eq. (9)? Where are these "beliefs" described and quantified in the article?

Does this statement indicate that Info-Gap's uncertainty model expresses the "belief" that values of u near the estimate û are more likely or more "believable" than values of u that are further from û? If this is indeed so, then what is all this talk about severe uncertainty and on what grounds do the authors claim the following (page 4, emphasis is mine)?

Info-gap models are used to quantify non-probabilistic "true" (Knightian) uncertainty (Ben-Haim, 2006).

I urge the authors to examine Hall and Harvey's (2009) paper regarding this issue. The bottom line is this: Info-Gap's uncertainty model as such does not represent any "beliefs" whatsoever. Hence, any attribution of "belief" to the model must be carefully quantified and introduced formally as an additional assumption. For example, Hall and Harvey (2009) incorporate "likelihood" in Info-Gap's uncertainty model by explicitly adding the assumption (emphasis is mine):

An assumption remains that values of u become increasingly unlikely as they diverge from û.

Is this the kind of "beliefs" the authors have in mind? If so, they must state this in so many words.

Of course, as indicated in Review 6, the trouble with positing any such additional assumption is that it embroils the Info-Gap decision theory in yet greater difficulties. For, any assumption of this kind is utterly incompatible with the basic thesis that Info-Gap decision theory is designed specifically for the severest uncertainty imaginable, namely "true" Knightian uncertainty.

In summary.

The basic working assumption postulated by Info-Gap decision theory is that the estimate û is a wild guess, a poor indication of the true value of u, and is likely to be substantially wrong. This assumption is thoroughly correct, for it reflects the simple fact that Info-Gap decision theory is designed to manage severe uncertainty.

So whatever "belief" structure is imputed to Info-Gap's uncertainty model, the fact remains that the overwhelming "belief" is that the true value of u is "distant" from the estimate û. This is particularly the case — indeed, commonly the case — when the region of uncertainty, is unbounded. Indeed, if the "belief" is that the true value of u is in the neighborhood of the estimate, why should we care a straw about the region of uncertainty being unbounded?

In short, if the intention is indeed to modify Info-Gap's uncertainty model and let it express " ... the decision maker's "belief" about uncertain variation of u around û ..." then the authors must give an exact quantitative specification — as done for instance in Hall and Harvey (2009) — of this "belief" and then explain how this "belief" dwells under the same roof with the "true knightian" uncertainty that Info-Gap decision theory is supposed to deal with.

More importantly yet, perhaps, the authors would care to explain how such a quantification of "beliefs" is radically different (after normalization, if necessary) from say, the "beliefs" quantified by subjective probabilities, or the "beliefs" expressed by membership functions of fuzzy sets.

Be that as it may, the belief-based theory resulting from incorporating "beliefs" in Info-Gap's uncertainty model is clearly incongruous with Ben-Haim's Info-Gap theory.

3. How wrong can the estimate be?

On page 6 the authors state the following (emphasis is mine):

We will use info-gap analysis to estimate "how wrong can we be?", or how wrong can the estimates be, and still allow us to obtain an acceptable probability of detection.

As explained in FAQ # 28 and in Info-gap decision theory and the small applied world of environmental decision-making (Point 2), Info-Gap decision theory does not — much less can it — address such questions for the simple reason that the true value of u is unknown and is subject to severe uncertainty.

Rather, the question addressed by Info-Gap's robustness model is defined as follows:

How much can we deviate from the given estimate û so that the performance requirement is satisfied everywhere within the region of uncertainty stipulated by the deviation?

The following pictures illustrate the distinction between "being wrong" and "determining a safe deviation from the estimate".

Recall that Info-Gap's robustness model is defined as follows:

α(rc,û):= max   max {α≥0: R(q,u) ≥ rc , ∀u∈U(α,û)}
q∈Q

The picture is then as follow

No Man's LandûNo Man's Land
U(α(rc,û),û)
Safe

where the red area represents the "safe" region of uncertainty around the estimate and the "No Man's Land" is the region that is left unexplored by Info-Gap's robustness model.

The "size" of this safe area is α(rc,û) which means that the maximum safe deviation from the estimate is α(rc,û). In the picture this safe deviation is relatively small compared to the size of the complete region of uncertainty (shown in black).

That much is clear.

However, the whole point here is that the value of α(rc,û) gives us not the merest clue as to how "wrong" û is, and how wrong the points in the safe region are. The only way to establish how "wrong" these points are is to determine their "distance" from the true value of u. But this value is unknown and is subject to severe uncertainty.

For example, consider the case where the true value of u, denoted by u*, is at the far right hand-side of the region of uncertainty, as shown below:

No Man's LandûNo Man's Landu*
U(α(rc,û),û)
Safe  

Here the estimate is clearly "very wrong" as illustrated by its great distance from the true value of u.

The whole difficulty posed by severe uncertainty is that we have no inkling as to where u* is. This means that we have not the foggiest idea how wrong û is, and consequently how wrong can we be and still satisfy the performance requirement.

All we know is that the maximum "safe" deviation from the estimate is equal to α(rc,û).

In short: The authors confuse two completely different concepts:

It should be stressed that if the estimate û is poor, then the value of the largest safe deviation from the estimate is a poor indication of robustness against the uncertainty in the true value of u.

Remark: There are a number of distracting typos in the article. For instance, the term "minimizing" should be replaced by "maximizing" in the Abstract and Conclusions sections in relation to the probability of detection. Clearly, the objective is to maximize the probability of detection, not to minimize it.

Similarly, the term "decreases" should be replaced by "increases" at the end of Section 4 in the discussion on what constitutes a more demanding critical level of performance rc. That is, insofar as the performance constraint R(q,u) ≥ rc is concerned, a more demanding rc is a larger rc. Hence, rc is more demanding when it increases rather than when it decreases. Indeed, when rc is sufficiently small, the performance constraint is superfluous and can be ignored.

Review # 10 (Posted: June 1, 2009; Last update: June 1, 2009)

Reference*: Y. Ben-Haim, M. Zacksenhouse, C. Keren, C.C. Dacso
Do we know how to set decision thresholds for diabetes?
Medical Hypotheses (in press, available online 5 April 2009)
Abstract The diagnosis of diabetes, based on measured fasting plasma glucose level, depends on choosing a threshold level for which the probability of failing to detect disease (missed diagnosis), as well as the probability of falsely diagnosing disease (false alarm), are both small. The Bayesian risk provides a tool for aggregating and evaluating the risks of missed diagnosis and false alarm. However, the underlying probability distributions are uncertain, which makes the choice of the decision threshold difficult. We discuss an hypothesis for choosing the threshold that can robustly achieve acceptable risk. Our analysis is based on info-gap decision theory, which is a non-probabilistic methodology for modelling and managing uncertainty. Our hypothesis is that the non-probabilistic method of info-gap robust decision making is able to select decision thresholds according to their probability of success. This hypothesis is motivated by the relationship between info-gap robustness and the probability of success, which has been observed in other disciplines (biology and economics). If true, it provides a valuable clinical tool, enabling the clinician to make reliable diagnostic decisions in the absence of extensive probabilistic information. Specifically, the hypothesis asserts that the physician is able to choose a diagnostic threshold that maximizes the probability of acceptably small Bayesian risk, without requiring accurate knowledge of the underlying probability distributions. The actual value of the Bayesian risk remains uncertain.
Acknowledgement This work was supported in part by a Grant from the Abramson Center for the Future of Health, Houston, TX.
Scores TUIGF:100%
SNHNSNDN:100%
GIGO:200%
* The publisher requests that this article in press be cited as follows:

Ben-Haim Y et al. Do we know how to set decision thresholds for diabetes?. Medical Hypotheses (2009), doi:10.1016/ j.mehy.2008.12.053

Since the idea of regarding Info-Gap's robustness model as a "proxy" for a probabilistic model of "success" had already been raised in a number of Ben-Haim's publications, I shall first examine how it sits in the general context of Info-Gap decision theory. Then, I shall examine it in some detail to see how it fares with regard to the specific Info-Gap model discussed in the paper.

Overview

Let us begin by noting that the journal Medical Hypotheses is devoted to the publication of ... medical hypotheses. According to its Guidelines for Authors:

The purpose of Medical Hypotheses is to publish interesting theoretical papers. The journal will consider radical, speculative and non-mainstream scientific ideas provided they are coherently expressed.

So, the question is: does the hypothesis proposed in the article fit the bill? Is it indeed radical and non-mainstream? And if so, in what sense is it radical and non-mainstream?

To work out an answer to this question you need not bother to read the paper in its entirety. The answer stares you in the face ... in the abstract, where we read (emphasis is mine):

Our analysis is based on info-gap decision theory, which is a non-probabilistic methodology for modelling and managing uncertainty. Our hypothesis is that the non-probabilistic method of info-gap robust decision making is able to select decision thresholds according to their probability of success.

The main objective of this review is to show that the proposition made by this hypothesis is not just "radical and non-mainstream", it is in fact unscientific, or more accurately counter-scientific. And to be able to appreciate why this is so keep in mind that Info-Gap's robustness model is by definition local and that the uncertainty that this model is designed to take on is severe.

So the good news is that the future of probability theory and statistics is not in danger ... not yet!

In greater detail, consider the two circles, A and B on the left. The large circle represents the space that affects the choice and ranking of decisions according to their probability of success. The much smaller circle, B, represents the space that affects the choice and ranking of decisions as prescribed by Info-Gap's robustness analysis.

That B is typically a relatively small subset of A is a direct consequence of Info-Gap's local robustness analysis being conducted only on an infinitesimally small area of the (unbounded) uncertainty space (A) that Info-Gap decision theory presumably takes on.

So mathematically speaking we can look at the proposed hypothesis as being grounded in a comparison between the results yielded by two algorithms: one is defined on A, call it f, and one is defined on B, call it g. Algorithm f yields the ranking of decisions according to their probability of success, whereas Algorithm g yields the ranking of decisions according to their robustness (as prescribed by Info-Gap).

The authors' hypothesis is that these two algorithms yield the same results.

But the point to note here is that for this hypothesis to hold, Algorithm g must exhibit a significant level of redundancy and/or degeneracy. However, since redundancy and/or degeneracy is not a generic characteristic of the probabilistic models of "success" examined by the authors, the inference is that this must be imposed on the two models through certain assumptions that would link the performance of the two algorithms.

Since the authors do not postulate any such assumptions, there is no reason to believe that the proposed hypothesis is valid.

To the contrary, there is every reason to believe that in the absence of some inherent redundancy/degeneracy in the specific probabilistic model of success examined in the article, this hypothesis cannot hold. It is simply too good to be true: how can a non-probabilistic analysis of a small subset of a much large set be a reliable proxy for a probabilistic analysis of "success" over the much larger set?

Indeed, if we accept this "too good to be true" hypothesis, why not accept "too good to be true" offers such as this:

Date: Sun, 7 Jun 2009 01:47:13 +0200 (MEST) 
From: ??????? 
Reply-To: ??????? 
Subject: My Friend 
  
My Friend  

How are you today? Hope all is well with you and your family?

I am using this opportunity to inform you that this multi-million-dollar business has
been concluded with the assistance of another partner from Chile who financed the
transaction to a logical conclusion. Due to your effort,sincerity courage and trust
worthiness You showed during the course of the transaction.
 
I have left a certified Bank Draft. for you worth of $1,300,000.00 cashable anywhere in the world.  
  
Mr. ??????

The absurd in the proposed hypothesis is on a par with the absurd in the proposition that what you see through a keyhole is a reliable proxy of what you see through an open door.

vs
Hide/Show the BIG picture!


As they say: never believe what you see through a keyhole!

And if you like cheese, then perhaps the following analogy will be more meaningful to you:

vs
Hide/Show the BIG picture!


Perhaps.

One of the most impressive illustrations of this point is no doubt this:

vs
Hide/Show the BIG picture!


Commonly known as "the tip of the iceberg".

However, to my mind, the most edifying analogy is the distinction between a local and a global optimum.

vs
Hide/Show the BIG picture!


As we well know, generally, a local optimum is not necessarily a global optimum. Indeed, we go to great lengths to explain to our students why it is important to avoid confusing between these two related concepts.

A similar distinction between local and global robustness of course applies in the case of Info-Gap. But we find no trace of it in the Info-Gap literature. To wit:

whereas

Worse, not only is there no awareness whatsoever, in the Info-Gap literature, of the difference between these two types of robustness, there is no awareness of the fact that the robustness yielded by Info-gap is local.

And to top it all off, Info-Gap scholars persistently fail (or avoid) to indicate that Info-Gap's robustness model is in fact a simple instance of Wald's famous Maximin model (circa 1940). Meaning that the Info-Gap analysis is in fact a (local) worst-case analysis.

In a word, only a failure to appreciate the distinction between global and local robustness could have inspired this hypothesis.

Illustrative example

Suppose that robustness is sought with respect to the constraint f(x) ≥ 0 over the set X=[-7,7], where f is the function shown below:

If x is the realization of a random variable, we can consider the probability that f(x) ≥ 0 as a measure of robustness. In this case this probability is the probability that the random variable takes value in the blue intervals shown below:

If the probability distribution of this random variable is known, then it would be a simple task to compute the probability that f(x) ≥ 0.

But what should we do if we face severe uncertainty and we do not know the distribution of this random variable?

The hypothesis proposes that Info-Gap's non-probabilistic robustness model can be used as a proxy for obtaining the unknown probability distribution.

To apply this model we shall have to estimate the "true" value of x and conduct the robustness analysis in the neighborhood of this estimate, call it x*. For instance, if we assume that x*=3, we shall conduct the robustness analysis in the interval, say [1.5,4], as shown below:

So in this case the hypothesis is that Info-Gap's local robustness analysis in the interval X'=[1.5,4] will yield a reliable proxy for the probability of f(x) ≥ 0 over the interval X=[-7,7].

In short, if we accept that the hypothesis has any merit whatsoever, we must accept that Info-Gap's robustness model is endowed with substantial extraordinary powers. Namely, we must accept that it has the capability to generate reliable probabilistic results out of a non-probabilistic model of uncertainty, and all this by means of a local analysis in the neighborhood of a poor estimate!

This is too good to be true!

Reality Check

So, to set the record straight on the proposition that Info-Gap robustness is a reliable proxy for the probability of success, consider the following formal result:

Reality Check Theorem:

The proposed hypothesis is not valid.

That is, consider the generic Info-Gap robustness model

α(q,û):= {α≥0: R(q,u) ≤ Rc , ∀u∈U(α,û)} , q∈Q

and assume that α(q'',û) > α(q',û) for some decisions q' and q'' in Q. Then this does not imply that Pr[R(q'',u) ≤ Rc] > Pr[R(q',u) ≤ Rc], where Pr[event] denotes the probability of "event" as determined by some (unknown) probability distribution on the uncertainty space U.

Proof.

It is sufficient to show that there are cases where for some decisions q' and q'' such that α(q'',û) > α(q',û) there is a probability distribution on U such that Pr[R(q',u) ≤ Rc] > Pr[R(q'',u) ≤ Rc].

Let

D(q',q''):= {u∈U: R(q',u) ≤ Rc; R(q'',u)> Rc}

By definition then, D(q',q'') denotes the subset of U on which decision q' satisfices the performance requirement and decision q'' violates this requirement.

Now consider any case where D(q',q'') is not empty. Then clearly, there is a probability distribution on U such that Pr[R(q',u) ≤ Rc] > Pr[R(q'',u) ≤ Rc]. For example, any distribution whose support is a subset of D(q',q'') will do the job. QED

Clearly then, the proposed hypothesis does not hold in the context of Info-Gap's generic robustness model.

Our next task is to explain the fundamental flaws in the thinking behind the hypothesis.

Garbage In — Garbage Out Axiom

The question that we need to ask ourselves is as follows:

How is it possible that a methodology employing a non-probabilistic model of uncertainty is capable of selecting decisions according to their probability of success?

The short answer is clear:

Either this hypothesis is groundless, or ... there exist certain implicit powerful links between the probabilistic and non-probabilistic models that empower the latter to mimic the behavior of the former.

Regarding the second option, see for example the well known concept deterministic equivalent that is used extensively in stochastic programming. All that needs to be pointed out here is that, the existence of deterministic equivalent models relies on the validity of very strong assumptions.

The longer answer is as follows:

Either this hypothesis is groundless, or ... there exist certain implicit powerful links between the probabilistic and non-probabilistic models that empower the latter to mimic the behavior of the former.

It does not take much to explain why this hypothesis is groundless with regard to Info-Gap's robustness model. Because, to claim that this hypothesis is valid for Info-Gap's robustness model, without invoking the support of certain prerequisite assumptions, is as good as declaring in public that the universally accepted Garbage In -- Garbage Out (GIGO) Axiom is null and void.

But in this case Info-Gap decision theory puts itself at loggerheads with "Conventional Science" as follows:

Conventional ScienceInfo-Gap Decision Theory    
wild guess   -----> Model ----->  wild guess
wild guess   -----> Model ----->reliable
robust decision

That is, in science the ruling convention is that results generated by a model can be only as good as the estimate on which they are based. Yet, Info-Gap decision theory proclaims its robustness model capable of generating reliable robust decisions by an analysis of the neighborhood of a wild guess of the true value of the parameter of interest. Namely, without bothering to give the slightest justification, Info-gap attributes extraordinary powers to its robustness model, to reliably extrapolate from local results in a small neighborhood of a wild guess to the entire vast uncertainty space U.

In short, by contravening the Garbage In -- Garbage Out Axiom, Info-Gap decision theory is at odds with a fundamental precept of science thereby showing itself to be an unscientific theory. Therefore, the very idea of proposing such an hypothesis is of a piece with the voodoo (counter-scientific) approach taken by Info-Gap to decision- making under uncertainty.

It is apposite to add here that if we accept for a fact Info-Gap's extraordinary ability to generate reliable robust decisions out of a wild guess, then we may as well wind up the discipline of Decision Making Under Severe Uncertainty and declare it redundant. For, dealing with decision-making problems subject to severe uncertainty would now amount to child's play:

1-2-3 fool-proof recipe for decision-making under severe uncertainty
  1. Ignore the severity of the uncertainty.

  2. Focus instead on the neighborhood of your best estimate of the parameter of interest.

  3. Don't worry if you lack an estimate, a wild guess will do*.

Wouldn't this be great?!

*Should you need it, the recipe for obtaining a wild guess is simplicity itself:

See it on-line at wiki.answers.com/Q/What_is_best_estimate_and_how_do_i_calculate_it.

The Hypothesis

Given the state of the art in decision theory and robust optimization, what we have here is

Either

a major breakthrough in decision theory and robust optimization!
Or,
a huge blunder attesting to a fundamental misapprehension/misconception of the central difficulty confronting decision-making in the face of severe uncertainty.

So, to repeat the obvious:

The basic difficulty facing decision-making under severe uncertainty is that the estimate we have of the true value of the parameter of interest is poor and likely to be substantially wrong. Hence, there is no reason to believe that a local analysis in the neighborhood of the estimate will yield reliable results.

The question is then: what should be done given this state of affairs?!

In a nutshell, the severity of the uncertainty makes a global treatment of robustness imperative. That is, it demands that the entire uncertainty space, or a suitable global approximation thereof, be explored.

Here then are two simple reasons why the proposed hypothesis is groundless:

  1. Info-Gap's robustness analysis is not conducted over the entire specified uncertainty space.

  2. In the framework of Info-Gap decision theory, the estimate used is a wild guess and is likely to be substantially wrong.

So, the question is: how can the proposed hypothesis have any merit whatsoever when Info-Gap's robustness model itself (before you even begin to ponder the hypothesis) is fundamentally flawed? And the answer is plain: for the hypothesis to hold, not only is it imperative that extremely stringent conditions hold to establish the envisaged powerful nexus between Info-Gap's robustness model and the probabilistic model of "success". It is imperative that the model itself be sound. Since no such conditions are incorporated on the (already fundamentally flawed) generic Info-Gap robustness model, it is not surprising that the hypothesis is generally not valid.

By the same token, given that no such conditions are imposed on the models presented in the paper, it follows that there is no reason to believe that the hypothesis can hold for these models.

To examine this contention more formally, assume that the performance constraint is defined as follows:

R(q,u) ≤ Rc

where q represents the decision variable, u represent the parameter of interest, R represents the performance functions and Rc denotes the required performance level.

In this framework, the probability of success is

PrU[R(q,u) ≤ Rc]

where PrU[event] denotes the probability of "event". The subscript U is used as a reminder that the probability is computed over the uncertainty space U, namely the set of possible values of u.

So far so good.

Now, according to Info-Gap decision theory, the robustness of decision q is defined as follows:

α(q,û):= max {α: R(q,u) ≤ Rc , ∀u∈U(α,û)}

where û denotes the estimate of the true value of u and U(α,û) denotes a region of uncertainty of size α around û.

So, the main thesis of this paper is as follows:

Hypothesis:
α(q'',û) > α(q',û)     ————>     PrU[R(q'',u) ≤ Rc] > PrU[R(q',u) ≤ Rc]

In words: if Info-Gap's analysis deems decision q'' to be more robust than decision q', then q'' has also a greater probability of success (satisficing the performance requirement).

Some Basic Facts

Before we subject the proposed hypothesis to a formal analysis, let us list the basic facts about Info-Gap's robustness analysis and its relationship to a probabilistic model of success:

We are now ready for a formal examination of the proposed hypothesis.

Anatomy of an Hypothesis

To examine the hypothesis more closely, consider the following three general obvious observations:

  1. It is straightforward to show — by means of a simple counter example — that this hypothesis is in general invalid, namely that situations exist where the hypothesis does not hold.

  2. It is straightforward to show — prove formally — that the hypothesis is valid for some trivial degenerate cases.

  3. It is extremely easy to conduct numerical experiments in the framework of the specific application considered in the paper (maximizing the probability of a correct diagnosis of diabetes), so as to put this hypothesis to the test.

So, let us now examine each observation in greater detail.

Counter Example

Consider the case shown in the following picture:

Info-Gap decision theory deems q'' to be more robust, but it is clear that q' dominates q'' over most of the uncertainty space U=ℜ. The dark areas at the bottom of the picture show the regions of U where the constraint is satisfied by these two decisions.

So, there are no too ways about it, q' is far more robust than q'' relative to the complete uncertainty space U.

It follows then that if the probability of u is not "concentrated" in the two very small intervals where q'' dominates q', the hypothesis is not valid. For example, if PrU[u∈U(α(q'',û)] is sufficiently small, then clearly the hypothesis does not hold.

Degenerate Valid Example

It can be shown (see FAQ # 62) that in some trivial degenerate cases Info-Gap's robustness in the neighborhood of an estimate — even a very bad one — is a "proxy" for the true robustness over the entire uncertainty space U. In such cases — which are of little interest, if any — the hypothesis is valid regardless of the specific structure of the probabilistic model, namely PrU[event]. However, as explained in FAQ # 62, in these cases there is no need for the Info-Gap model in the first place. The results can be obtained directly from the performance constraint. The estimate û is just a nuisance parameter.

Numerical Experiments

The specific Info-Gap robustness model studied in the paper is so simple that closed-form solutions are readily available. Meaning that, it is extremely easy to test this hypothesis by conducting numerical experiments via simulation. It is unclear, therefore, why the authors did not bother to do this in order to see whether the hypothesis can have any merit.

The "Why is the hypothesized plausible?" section

For obvious reasons the Journal requires that contributors explain why the radical hypotheses that they propose are plausible. This is done under the heading "Why is the hypothesized plausible?".

In this section of their paper the authors argue that their hypothesis is plausible because their Info-Gap analysis is based on two assumptions (page 4):

The info-gap models of uncertainty which underlie our example, Eqs. (5) and (6), assume two things: (i) that the cumulative distribution functions (CDFs) are normal (Gaussian) and (ii) that the estimated moments deviate fractionally by an unknown amount.

They then proceed to argue that these assumptions are reasonable.

But the hypothesis is not just about the Info-Gap models proposed in the article. The whole point of the hypothesis is that it makes a statement about a relationship between Info-Gap's robustness model and a probabilistic model of "success". So what should have been discussed under this heading is the plausibility of the assumed kinship between these two models.

So as the readers cannot possibly gather this from this article, it is important to take note that contrary to the local analysis that is prescribed by Info-Gap's local robustness model, the probability of success is detemined by a global expectation operator over the entire uncertainty space U.

The picture is this:

  • The green island represents the complete uncertainty space, namey U.

  • The black dot represents the estimate û of the true (unknown) value of the parameter of interest.

  • The white circle represents the subset of U on which Info-Gap decision theory conducts its robustness analysis, namely U(α*,û), where α*:= max {α(q,û): q∈Q}, recalling that U(α,û) denotes the region of uncertainty of size α around the estimate û.

Let A = U and B = U(α*,û). Since the authors assume that U is unbounded, B is actually an infinitesimally small subset of A. For illustrative purposes it is enlarged in this picture.

Things to remember:

  • Whereas Info-Gap decision theory conducts it business on A, the probabilistic model of success conducts its business on B.

  • The estimate û is a wild guess and is likely to be substantially wrong.

Given that the black dot is a wild guess of the white square, to argue that an analysis of the small white circle is a proxy of an analysis of the entire green island is tatamount to practicing voodoo science.

In a nutshell, this glaring incongruity between Info-Gap's robustness model and models determining the probability of success not only brings out the implausibility of this hypothesis. It brings out that the hypothesis in fact violates the universally accepted GIGO Axiom.

Conclusions

  1. The idea behind the proposed hypothesis is misguided: it asserts — without invoking relevant assumptions — that a local deterministic model yields the same results as a global probabilistic model.

  2. Unless stringent conditions are postulated to hold for the problem in question, there is no reason to believe that a local analysis in the neighborhood of a wild guess will yield a reliable result. This is precisely what the universally accepted GIGO Axiom dictates.

  3. Since no such conditions are postulated by the authors to hold for the specific problem under consideration, the authors' hypothesis is without any foundation.

  4. The comments in Review 6 and Review 9 regarding the likelihood-free nature of Info-Gap's robustness model are also relevant here.

  5. Ben-Haim (2009) discusses in detail the type of strong conditions that must be imposed (jointly) on the probability model and on Info-Gap's robustness model to justify the validity of the hypothesis.

    It is therefore puzzling that no such conditions are discussed by the authors in this paper. Even more puzzling is the fact that the authors fail to cite Ben-Haim's (2009) paper and the results presented there regarding the connection between Info-Gap's robustness model and the probability of "success".

In short, we conclude that the future of probability theory and statistics ... is not in jeopardy ... not yet!

Recommendation

In view of the above, I strongly recommended that Medical Hypothesis require authors to indicate clearly in their papers whether the hypotheses that they propose withstand the test of the GIGO Axiom. If this cannot be done, then an explanation should be provided.

Stay tuned ... there is more in store!

And something to think about in the meantime:

The authors use an unbounded uncertainty space. This means, among other things, that they assume that the fasting plasma glucose (FPG) concentration can be arbitrarily low and high!

According to the information available to me

So isn't it safe to assume that the only relevant FPG concentrations are in the range, say [0,50] (mmo/l) rather than the impressive unbounded range (-∞,∞) used by the authors? Do you know anyone — alive or otherwise — with negative FPG concentration?

Any comments/suggestions on this issue will be greatly appreciated!

Note: a mole is approximately 6.022141796*1023 molecules, and mmol/l = millimoles/liter. For blood glucose, the conversion from molecular count to weight is: mmol/l = 18 mg/dl (milligrams/deciliter).

Review # 11 (Posted: June 15, 2009; Last update: August 13, 2009)

Reference: An info-gap approach to managing portfolios of assets with uncertain returns
Bryan Beresford-Smith and Colin J. Thompson
Journal of Risk Finance
10(3), 277-287, 2009.
Abstract Purpose
The purpose of this paper is to provide a quantitative methodology based on information-gap decision theory for dealing with (true) Knightian uncertainty in the management of portfolios of assets with uncertain returns.

Design/methodology/approach
Portfolio managers aim to maximize returns for given levels of risk. Since future returns on assets are uncertain the expected return on a portfolio of assets can be subject to significant uncertainty. Information-gap decision theory is used to construct portfolios that are robust against uncertainty.

Findings
Using the added dimensions of aspirational parameters and performance requirements in information-gap theory, the paper shows that one cannot simultaneously have two robust-optimal portfolios that outperform a specified return and a benchmark portfolio unless one of the portfolios has arbitrarily large long and short positions.

Research limitations/implications
The paper has considered only one uncertainty model and two performance requirements in an information-gap analysis over a particular time frame. Alternative uncertainty models could be introduced and benchmarking against proxy portfolios and competitors are examples of additional performance requirements that could be incorporated in an information-gap analysis.

Practical implications
An additional methodology for applying information-gap modeling to portfolio management has been provided.

Originality/value This paper provides a new and novel approach for managing portfolios in the face of uncertainties in future asset returns.

Keywords: Portfolio investment, Financial modelling, Uncertainty management, Information management

Paper type: Research paper.

Acknowledgement The authors are grateful to Yakov Ben-Haim for many illuminating discussions on info-gap theory and for a critical reading of an early version of this paper.
Scores TUIGF:100%
SNHNSNDN:75%
GIGO:100%

The main reason for my taking up this paper for review is to address an explicit statement made by the authors about results reported on in my 2008 article, which was published in the same journal.

On page 278 of the article we read (emphasis is mine):

The main problem with CAPM and related models is that they are based on expected future returns on assets that in principle are unknown and subject to considerable uncertainty. In such situations we are dealing with "true uncertainty" in the sense of Knight (1921) who was the first to distinguish between "risk" based on known probability distributions and true uncertainty when the underlying statistical distributions are unknown. Knight's ideas have been further developed by several authors over the years and in particular by Ben-Haim (2006) who has developed a quantitative formulation known as information-gap decision theory. This theory has recently been shown by Sniedovich (2008) to be formally equivalent to Wald's maximin model in classical decision theory (French, 1988).

This text addresses three prevailing myths about Info-Gap decision theory, namely:

In the quoted text the authors dispel Myth 1 and Myth 2 and continue to propagate Myth 3.

Some comments:

  1. To begin with, it cannot be emphasized enough that Info-Gap decision theory is, as a matter of principle, unable to deal with Knightian uncertainty. Indeed, adopting as it does a local approach to the treatment of uncertainty, hence to robustness against severe uncertainty, Info-Gap's robustness model constitutes the exact antithesis of what a proper/correct/sound treatment of Knightian uncertainty calls for.

    For, how can a theory even be expected to meet the challenges posed by severe uncertainty, especially "true" Knightian uncertainty, if it a priori fixes only on a wild guess and limits its entire robustness analysis to the immediate neighborhood of this wild guess?

    The answer is: it can't!

    The picture is this:

    No Man's LandûNo Man's Land
    <-------------- Complete region of uncertainty under consideration -------------->

    where û denotes the estimate of the parameter of interest, the black area represents the complete region of uncertainty under consideration, the red area around û represents the region of uncertainty that actually affects the results generated by Info-Gap's robustness analysis, and the vast No Man's Land represents that part of the complete region of uncertainty that has no impact whatsoever on the results generated by Info-Gap's robustness model.

    Clearly, how can one possibly expect Info-Gap decision theory — that as attested by this statement is designed to take on the severest uncertainty imaginable "True Knightian Uncertainty — to deal with Black Swans namely, extreme events that would fall under "True Knightian Uncertainty", if it cannot even handle plain, ordinary, "white swans" that occur in the vast No Man's Land.

    And what is the wonder that such a theory cannot possibly be expected to deal with this type of uncertainty?!

    After all, Info-Gap's uncertainty and robustness models were originally (Ben-Haim, 1996 ) put forward to deal with a relatively mild uncertainty and variability around given "nominal values". Yet, without the slightest modification having been made in these models to meet the challenges of severe uncertainty, precisely the same models, prescribing precisely the same treatment of uncertainty, were imported lock stock and barrel into the two more recent Info-Gap books (Ben-Haim, 2001, 2006) to offer an approach to decision under the severest uncertainty imaginable: "knightian" uncertainty.

    To repeat, how can one possibly expect that models designed for the treatment of a mild uncertainty will be suitable for the treatment of "true" Knightian uncertainty!

    Indeed, the obove picture speaks for itself. It gives a graphic depiction of the fact that Info-Gap decision theory does not even come close to tackling the severity of the uncertainty that it claims to manage. Info-Gap's formula for dealing with the ("True Knightian") uncertainty that it professes to take on is simply ... to ignore it. This is the effect of Info-Gap prescribing a robustness analysis that fixes only on a given wild guess of the true value of the parameter of interest and its immediate neighborhood, to the exclusion of practically the entire region of uncertainty!

    The robustness that one obtains then cannot possibly be declared robustness against severe uncertainty.

    All one can say is that the results yielded by the Info-Gap analysis provide a local robustness in the vicinity of a wild guess!

  2. And so while it is of course a fact that in his 2001 and 2006 books and in many of his articles, Ben-Haim makes constant reference to Knightian uncertainty, thus giving the impression that Info-Gap is practically tailored to handle this type of forbidding uncertainty; the truth is that Info-Gap constitutes the precise antithesis of what a theory for the treatment of severe uncertainty ought to be.

    Because, to repeat, the uncertainty and robustness models deployed by Info-Gap decision theory are the same models deployed in Ben-Haim's 1996 book where the uncertainty under consideration is not assumed to be severe, and where there is no mention of Knightian uncertainty.

  3. Regrettably, the statement describing the relationship between Ben-Haim's Info-Gap decision theory and Wald's Maximin model which is attributed to my 2008 paper is mistaken and therefore requires a major correction.

    In my 2008 paper I set out a detailed proof of a theorem showing that Info-Gap's robustness model is an instance of Wald's Maximin model. That is, I show that the generic Info-Gap robustness model, namely

    is a simple instance of Wald's generic Maximin model, namely of

    The proof is constructive: it shows that Info-Gap's robustness model is the simple instance of Wald's generic Maximin model specified by

    where

    So, the point of this theorem is not that Info-Gap's robustness model is equivalent to Wald's Maximin model. Rather, the point of this theorem is that by virtue of its specific objective function and sets of admissible states, Info-Gap's robustness model constitutes a simple instance of the classical Maximin model. This means, of course, that like countless other instances of Wald's generic Maximin model, the Info-Gap model is subsumed by the Maximin model. The immediate implication is, of course, that Maximin is incomparably more general and powerful than Info-Gap's robustness model so that there can be no talk whatsoever of equivalence between the two models.

    Indeed, by analogy, the assertion that Info-Gap's robustness model is equivalent to Wald's Maximin model is as mistaken as the assertion that the family of polynomials specified by

    p(x) = 1 + αx2+βx4

    is equivalent to the family of polynomials specified by

    q(x) = a + bx + cx2+dx3+ex4+fx5+gx6+hx7

    or that the class of exponential distributions

      ,   λ > 0

    is equivalent to the class of Weibull distributions

      ,   λ > 0, k > 0

    All three assertions share a common feature: they are in error!

  4. On page 279 the authors formulate the Info-Gap robustness model for decision x, namely

    They then go on to argue that this expression shows that their " ... info-gap model is formally at least, equivalent to conventional max-min decision theoretic models (French, 2006) ..."

    But, this argument

    • Is mistaken. At best it indicates that this particular Info-Gap model is an instance of Wald's generic Maximin model.

    • Gives a distorted picture of how the Maximin/info-Gap connection is in fact treated in the Info-Gap decision literature.

    • Is far too simplistic to actually capture the kinship between the two models.

    To wit:

    • Wald's Maximin model is far more general than this Info-Gap model. Indeed, this model is but an instance of Info-Gap's generic robustness model, which in turn is just a simple instance of Wald's generic Maximin model. So how can this model possibly be equivalent to Wald's generic Maximin model??!?!?! It cannot be, and is definitely not, equivalent to Wald's Maximin model.

    • There is no reference whatsoever to the fact that Ben-Haim — the Father of Info-Gap — continues to stick to his guns, maintaining his denials that Info-Gap's robustness model is a Maximin model. These denials are made not only in a paper published in the same journal — cited in my 2008 paper — but in other writings and presentations to date, where this question is raised. This includes, among other publications, Ben-Haim's two books on Info-Gap, and his compilation of FAQs about Info-Gap. Also see Review 5 and Review 12.

      So, if — as claimed by the authors — the above conclusion follows so directly from the expression describing the above Info-Gap model, shouldn't they have addressed Ben-Haim's persistent claims that Info-Gap's robustness model is not a Maximin model?


    • The fact that a "max" and a "min" occur in the formulation of a model does not automatically render this model a Wald's Maximin model. Indeed, to formally show that the above model is a Maximin model it is necessary to reformulate the elements of the model, eg. incorporate the constraint in the objective function of the Maximin model, as done in my 2008 paper.

      To illustrate,

      is not a Maximin model, and

      is a Maximin model.

      In fact, it is not essential for a model to have an "inner" max or min to be a Maximin model. For instance,

      is a perfectly kosher Maximin model.

      These subtle modeling issues are discussed and explained in Wikipedia. Also, see my discussion on math formulations of the Maximin model.

    5. But, more than anything else, in view of their contention that Info-Gap "is equivalent" to Wald's maximin model, shouldn't the authors have given us at least some indication as to the rationale behind their proposition to use Info-Gap in the first place?!

    Shouldn't they have made it clear why, in their view, is there any point, merit, or advantage to turn to Info-Gap rather than stick with the old warhorse Wald's Maximin model, which, one need hardly point out, is the most prevalent model used in the field of "Robust Optimization" and in "Robust portfolio optimization"?!

  5. It is no less regrettable that the authors do not address the other fundamental flaws in Info-Gap decision theory — flaws that are described and criticized in detail in my 2007 paper — particularly because these flaws bear directly on the validity of the analysis and results presented in the article.

    Specifically, because my arguments (proof and analysis) show that Info-Gap's robustness model fails to tackle the severity posed by the "true" Knightian uncertainty considered in the article — the implications for their analysis and results are obvious.

On the brighter side, though, the authors' statement is a major breakthrough for my Info-Gap campaign. For, after more than three years of hammering this fact, I finally succeeded to convince two dedicated Info-Gap adherents that Ben-Haim's repeated assertions that Info-Gap's robustness model is not a Maximin model are erroneous! (see Review 5).

It is interesting that Ben-Haim still claims that Info-Gap's robustness model is not a Maximin model (see Review 12).

To learn more about the Maximin/Info-Gap connection and Info-Gap's flawed implementation of its Maximin robustness model go to the compilation of FAQs about Info-Gap.

Review # 12 (Posted: July 10, 2009; Last update:August 13, 2009)

Reference: Heterogeneous uncertainties in cholesterol management
Yakov Ben-Haim, Clifford C. Dacso, Jonathon Carrasco, Nithin Rajan
International Journal of Approximate Reasoning
50, 1046–1065, 2009
Abstract Physicians use clinical guidelines to inform judgment about therapy. Clinical guidelines do not address three important uncertainties: (1) uncertain relevance of tested populations to the individual patient, (2) the patient’s uncertain preferences among possible outcomes, and (3) uncertain subjective and financial costs of intervention. Unreliable probabilistic information is available for some of these uncertainties; no probabilities are available for others. The uncertainties are in the values of parameters and in the shapes of functions. We explore the usefulness of info-gap decision theory in patient-physician decision making in managing cholesterol level using clinical guidelines. Info-gap models of uncertainty provide versatile tools for quantifying diverse uncertainties. Info-gap theory provides two decision functions for evaluating alternative therapies. The robustness function assesses the confidence—in light of uncertainties—in attaining acceptable outcomes. The opportuneness function assesses the potential for better-than-anticipated outcomes. Both functions assist in forming preferences among alternatives. Hypothetical case studies demonstrate that decisions using the guidelines and based on best estimates of the expected utility are sometimes, but not always, consistent with robustness and opportuneness analyses. The info-gap analysis provides guidance when judgment suggests that a deviation from the guidelines would be productive. Finally, analysis of uncertainty can help resolve ambiguous situations.
Acknowledgement The authors gratefully acknowledge the support of the Abramson Center for the Future of Health. The authors are indebted to comments by Scott Ferson, Malka Gorfine, Matthias Troffaes and Miriam Zacksenhouse.
Scores TUIGF:100%
SNHNSNDN:6000%
GIGO:100%

Overview

This is a typical Info-Gap paper: it is full of the usual errors, misconceptions, and obfuscations that by now have become part and parcel of the Info-Gap enterprise.

Since 2006, I have repeatedly pointed out these two embarrassing facts about Info-Gap decision theory:

The reason that the truth about Fact 1 is embarrassing is simple. Since its introduction in the late 1990s, Info-Gap decision theory has been acclaimed as distinct, novel, revolutionary, and radically different from all current theories for decision under severe uncertainty.

Yet, my Maximin Theorem shows this claim for what it is: a pure myth!

The reason that the truth here is embarrassing is not merely that Info-Gap's robustness model is a reinvention of the wheel. Indeed, the truth here is hugely embarrassing because Info-Gap's robustness model is in fact a simple instance of none other than the most famous model in classical decision theory for the treatment of severe uncertainty: Wald's Maximin model. In other words, Info-Gap's robustness model is a simple instance of a model that, since the 1950s, has become the bread and butter approach to the management of severe uncertainty in classical decision theory and robust optimization.

The truth about Fact 2 is embarrassing because Info-Gap decision theory is hailed as a theory that is particularly suitable for the treatment of severe uncertainty

Yet, my Invariance Theorem shows this claim for what it is: a pure myth!

It shows that Info-Gap decision theory in fact constitutes the precise antithesis of what a theory for the treatment of severe uncertainty ought to be. In other words, it shows that Info-Gap decision theory "deals" with severe uncertainty by simply ... ignoring the severity of the uncertainty altogether. This fact renders Info-Gap decision theory a voodoo decision theory par excellence.

Details concerning these and other myths about Info-Gap decision theory can be found in my discussion on Myths and Facts about Info-Gap and in FAQs about Info-Gap.

So, you may well wonder: given that Info-Gap decision theory is so gravely flawed, and that its flaws are so detrimental to it, how is it that the paper under review here was accepted for publication in a refereed journal?

The answer is very simple: whatever Info-Gap decision theory lacks in substance and rigor, is made up for, and covered up, by heavy fog, spin, and rhetoric.

Indeed, to be able to pin down the truth about this theory, I had to cut through some mighty fog, spin, and rhetoric.

Some Basic Facts

The good news is that now that I have managed to accomplish this, the fundamental flaws in Info-Gap decision theory are clearly there for all to see. For example, to see that Info-Gap decision theory does not tackle the severity of the uncertainty at all, but simply ignores it, you need not even examine the Invariance Theorem. A quick look at this simple picture reveals most of the story:

No Man's LandûNo Man's Land
<-------------- Complete region of uncertainty under consideration -------------->

where û denotes the estimate of the parameter of interest, the black area represents the complete region of uncertainty under consideration, the red area around û represents the region of uncertainty that actually affects the results generated by Info-Gap's robustness analysis, and the vast No Man's Land represents that part of the complete region of uncertainty that has no impact whatsoever on the results generated by Info-Gap's robustness model.

Recalling that under conditions of severe uncertainty the estimate û is a wild guess, a poor indication of the true value of the parameter of interest and is likely to be substantially wrong, it is immediately clear that Info-Gap decision theory violates the universally accepted Garbage In Garbage Out (GIGO) Axiom. This crucial fact renders this theory a voodoo decision theory par excellence.

Example 1: On q' and q''.

Consider the case shown in the following picture:

where q' and q'' are two decisions, Rc=6 is the critical reward level associated with the performance constraint R(q,u) ≤ Rc, and u denotes the parameter of interest whose true value is subject to severe uncertainty. Note that the estimate of the true value of u is û = 0, and that the complete region of uncertainty is U=(-∞,∞).

The performance requirement R(q,u) ≤ Rc means that we prefer decisions whose R(q,u) values are small — ideally below the critical level Rc = 6 — over U.

It is assumed that R(q'',u) continues its (linear) ascent in both directions and that R(q',u) continues its (quadratic) descent in both directions. The colored (red and blue) areas at the bottom of the picture show the regions of U where the performance requirement is satisfied by these two decisions.

It is clear that q' dominates q'' over most of the uncertainty space U=(-∞,∞). In fact, q'' violates the performance requirement on most of U, whereas q' satisfices the performance requirement on most of U.

But this does not prevent Info-Gap decision theory to deem q'' to be more robust than q'.

So, there are no too ways about it: q' is far more robust than q'' with respect to the given performance requirement R(q,u) ≤ Rc relative to the complete uncertainty space U. Yet, Info-Gap's peculiar (local) definition of robustness deems q'' to be more robust.

You are cordially invited to a guided tour of Info-Gap robustness analysis including an interactive animation and an explanation of the reasons why Info-Gap decision theory regards q'' to be more robust than q'.

In so doing, Info-Gap decision theory puts itself at loggerheads with the universally accepted Garbage In — Garbage Out (GIGO) Axiom and the well known dictum that the results of an analysis can be only as good as the estimate on which they are based. So, the point is that in the case of Info-Gap decision theory the results yielded by its analysis can be no better than ... wild guesses.

To suggest otherwise would amount to suggesting that Info-Gap decision theory possesses mysterious (magical) powers that enable it to translate an analysis around a wild guess into a reliable methodology for the treatment of severe uncertainty.

Conventional ScienceInfo-Gap Decision Theory    
wild guess   -----> Model ----->  wild guess
wild guess   -----> Model -----> reliable
robust decision

If this is not a classic example of a voodoo decision theory what is? And isn't it also modern alchemy?

The trouble of course is that this simple fact is, as a rule, enveloped in the Info-Gap publications by fog, spin, and rhetoric. The article under review here is no exception. If anything, the fog, spin, and rhetoric have reached here unprecedented heights. Therefore, to give a full account of this phenomenon will take volumes.

But, to give you an idea, I shall illustrate it in action in connection with the Maximin issue. That is, I shall show how the simple fact that Info-Gap's robustness model is actually a simple instance of Wald's famous Maximin model is covered up by vintage Info-Gap fog, spin, and rhetoric.

The story goes like this.

In 2007 I advised users/promoters of Info-Gap decision theory that — contrary to repeated claims in the Info-Gap literature — not only is Info-Gap's robustness model neither new nor radically "different" from classical decision theory models, it is in fact a simple instance of Wald's famous Maximin model (circa 1940).

To substantiate this claim, I provided a detailed proof.

For our limited purposes here, it will be convenient to summarize this result as follows:

Maximin Theorem:

Info-Gap's Robustness model Corresponding instance of Wald's Maximin model
max {α ≥ 0: r(d,u) ≤ r* , ∀u∈U(α,û)}     ≡    
max min f(d,α,u)
  α ≥ 0     u∈U(α,û)  

where f(d,α,u) = α if r(d,u) ≤ r*; and f(d,α,u) = -∞, otherwise.

Note that this implies that

α(û)  := max   max {α≥0: r(d,u) ≤ r*, ∀u∈U(α,û)}
d∈D
 
   ≡  
  α(û):= maxmin f(d,α,u)
d∈D  u∈U(α,û)  
α≥0

Proof of the Maximin Theorem:

Instance of Wald's Maximin Model Equivalent Math Programming formulation
max min f(d,α,u)
  α ≥ 0     u∈U(α,û)  
    ≡    
max { v: v ≤ f(d,α,u), ∀ u∈U(α,û) }
  α ≥ 0  
v ∈ ℜ
 
    ≡    
max { α: α ≤ f(d,α,u), ∀ u∈U(α,û) }
  α ≥ 0  
 
     ≡    
max   { α: α ≤ f(d,α,u) , ∀ u∈U(α,û) }
 
     ≡    
max   { α: r(d,u) ≤ r*, ∀ u∈U(α,û) }    
Info-Gap's Robustness Model

The bottom line is there for all to see: Info-Gap's robustness model is a simple instance of Wald's famous Maximin model, namely

Wald's Maximin Model
z*:=   max min g(x,s)
x∈X   s∈S(x)  

More on this model can be found in the discussion on the Maximin model.

This simple instance of Wald's famous Maximin model always yields the same results as those yielded by Info-Gap's robustness model. No amount of empty rhetoric and spin can change this fact.

Note that the Maximin Theorem is constructive: it sets out a simple recipe for constructing that instance of the generic Maximin model that represents Info-Gap's robustness model.

No amount of fancy rhetoric or spin or fog can change this bottom line.

So far so good.

Given this proof, the question of course is: how can anyone possibly argue that Info-Gap's robustness model is not a Maximin model?

In other words, the proof of the Maximin Theorem is so simple, indeed straightforward, so how can one possibly prove the opposite?

Answer: faulty reasoning supported by plenty of fog, spin and rhetoric.

Faulty Reasoning

For the benefit of readers who are not familiar with Info-Gap decision theory, let me begin the illustration with a simple example that requires no knowledge of Info-Gap.

Example 2: On Jack and Jill

How would you prove formally that a perfectly healthy, seven year old kangaroo from Western Australia, call her Jill, is not a marsupial?

Of course, if your reasoning is sound and you are using conventional scientific methods of biological identification/classification it is impossible to do this.

Because, as it is a well-established fact that kangaroos are marsupials, and Jill being a healthy kangaroo, it follows that Jill is a marsupial.

But employing the type of faulty reasoning that underlies Info-Gap decision theory, showing that Jill is not a marsupial is a trifle.

Because, all one needs to do to this end is to find a certified ... koala. Yes, trust me, a ... certified koala, not a kangaroo.


So, to simplify the discussion assume that we found a certified koala, call him Jack.

We now have a certified koala called Jack, and a kangaroo called Jill.

The rest, according to Info-Gap's reasoning, is a trivial task. To wit:

A proof that Jill the kangaroo is not a marsupial
  • Clearly, Jack is a marsupial.

  • Clearly, Jill is different from Jack.

  • Therefore, clearly Jill is not a marsupial.

Indeed, employing this kind of reasoning almost anything can be proved. For instance, consider this gravity-defying trick:

Example 3: On p and q.

A proof that q(x) = 1 + x + x2 is not a polynomial
  • Clearly, p(x):= x4 is a polynomial.

  • Clearly, q is different from p.

  • Therefore, clearly q(x)=1 + x + x2 is not a polynomial.

Let me now show you how — based on this kind of reasoning — Ben-Haim argues that Info-Gap's robustness models, namely

Info-Gap's robustness model
α(û)  = max   max {α≥0: r(d,u) ≤ r*, ∀u∈U(α,û)}
d∈D

is not a Maximin/Minimax model. Namely that it is not an instance of

Wald's generic Maximin Model
z*:=   max min g(x,s)
x∈X   s∈S(x)  

In line with the reasoning illustrated above, the first thing to do is to come up with a Maximin/Minimax model that is not equivalent to Info-Gap's robustness model.

This is an easy task because there are infinitely many such models. The following is the Minimax model selected by Ben-Haim:

Ben-Haim's Minimax Model
α(û;α'):=   minmax r(d,u)
d∈D  u∈U(α',û)  

where α'>0 is some given real number.

Note that this is a Minimax, rather than a Maximin, model!

Obviously, there is a value of α' such that the optimal decision generated by Ben-Haim's model for this value of α' is optimal with respect to the Info-Gap model. The difficulty is that this specific value of α is not known in advance.

Thus, it is very easy to show that, in general, this specific Minimax model is not equivalent to Info-Gap's robustness model.

Ben-Haim uses this trivial argument as "proof" that Info-Gap's robustness model is not a Minimax/Maximin model.

This, however, is not approximate reasoning: this is faulty reasoning.

All that Ben-Haim's "proof" shows is that the ad hoc, inappropriate, Minimax model that he created for this exercise is not equivalent to Info-Gap's robustness model. But, this does not show that Info-Gap's robustness model is not an instance of Wald's Maximin model. Indeed, as shown by the Maximin Theorem, there is definitely an instance of Wald's Maximin model that is equivalent to Info-Gap's robustness model.

In short, the picture is this:

Info-Gap's robustness model
α(û)  = max   max {α≥0: r(d,u) ≤ r*, ∀u∈U(α,û)}
d∈D
Ben-Haim's Incorrect Minimax modelCorrect Maximin model
ρ(û;α'):= minmax r(d,u)
d∈D  u∈U(α',û)  
  α(û):= maxmin α·(r(u,d) ◊ r*)
d∈D  u∈U(α,û)  
α≥0
Note: α' is a pre-specified positive number. Note: a◊b =1 iff a≤b; a◊b =0 otherwise.

Note that in contrast to Ben-Haim's incorrect Minimax model, the correct Maximin model generates, as an integral part of the maximization operation, the specific value of α that is required by Ben-Haim's incorrect Minimax model, namely α'=α(û).

Of course, had Ben-Haim's "proof" been shown for what it is, namely had it been formulated as shown above, then even the referees of journals specializing in approximate reasoning would no doubt have identified the flaw in the reasoning underlying the "proof".

But the trouble is that Ben-Haim — the Founder of Info-Gap decision theory — and his co-authors, set out to "make the case" for the indefensible claim that Info-Gap's robustness model is not a Maximin and/or Minimax model with the aid of an avalanche of empty rhetoric and spin that are unprecedented even by Info-Gap standards.

Fog, Spin and Rhetoric

Let us see then how thick the fog, how extensive the spin, and how high the rhetoric can be, in a situation where the question under consideration is the relationship between two simple mathematical models:

Wald's generic Maximin Model
z*:=   max min g(x,s)
x∈X   s∈S(x)  
Info-Gap's robustness model
α(û)  = max   max {α≥0: r(d,u) ≤ r*, ∀u∈U(α,û)}
d∈D

Consider the following dissertation, quoted from the paper under review, while all the while keeping in mind that the Maximin Theorem is staring at us from the pages of refereed journals, from WIKIPEDIA and from many articles posted on the web:

Min–max decisions. The min–max approach identifies a set of possible contingencies or models or relevant functions and seeks the decision which minimizes the worst (maximal) loss on this set. (The max-min approach maximizes the lowest (minimal) gain when considering benefits rather than losses.) This concept is implicit in many robust Bayesian realizations, and many of our comments there apply here as well. Epistemically, the info-gap and min–max approaches are similar in representing uncertainty without measure functions, though the min-max approach requires the choice of a specific set. The hybridization of a min-max with an info-gap approach is often attractive, as discussed in connection with Bayesian methods. Indeed, Wald’s work in the early 1940s on min-max considers sets of uncertain probability distributions [51].

Behavior. We will discuss two concepts: the observational equivalence of min-max with info-gap robust-satisficing, and the behavioral difference of these methods [52].

Observational equivalence: Suppose a robust-satisficing decision maker must choose between two options, D1 and D2, and requires an outcome no worse than Lc in Fig. 14. This leads the robust-satisficer to choose decision D1, which is more robust than D2 at this requirement. An observer can describe this behavior by supposing the decision maker to be an min-maxer who believes that the horizon of uncertainty is α1, because at this level of uncertainty the maximum potential loss from D1 is less than from D2. Conversely, a min-maxing decision maker who believes that α1 is the true horizon of uncertainty would likewise choose D1 over D2. An observer could describe this by supposing the decision maker is a robust-satisficer whose requirement is Lc. In short, either strategy can be used to describe observed behavior by ascribing particular beliefs to the decision maker. In other words, the modelling of decision-behavior under uncertainty is under-determined in choosing between robust-satisficing and min-maxing.


Fig. 14. Crossing robustness curves, illustrating the observational equivalence and behavioral difference between min-maxing and robust-satisficing.

Behavioral difference: Suppose a min-maxing decision maker believes that the horizon of uncertainty can be as large as α2 in Fig. 14, but no larger. The min-maxer will prefer D2, whose loss can be as large as Lm, but less than the maximum potential loss of D1. Suppose a robust-satisficing decision maker can accept a loss as large as Lc, but no larger. This robust-satisficing decision maker will prefer D1 over D2 since D1 can tolerate greater uncertainty for achieving this requirement. The robust-satisficer will choose D1 over D2 even if the min-maxer has convinced the robust-satisficer that α2 is the true horizon of uncertainty. In short, when the robustness curves for two decisions cross one another, a min-maxer and robust-satisficer may disagree about the decision, depending on their beliefs and requirements.

In conclusion, the observational equivalence between min-maxing and robust-satisficing means that modellers can use either strategy to describe observed behavior of decision makers. In contrast, the behavioral difference means that actual decision makers will not necessarily be indifferent between these strategies, and will choose a strategy according to their beliefs and aspirations.

The concepts of observational equivalence and behavioral difference have been noted before, in different terms. Walley writes [53, p. 10]:

Every [Dempster-Shafer] belief function can be represented as a lower envelope of a set of probability measures. This is merely a mathematical representation, however; it is misleading and unnecessary to regard a belief function as a lower bound for some unknown probability measure. In the same way, every coherent lower prevision can be represented as a lower envelope of a set of linear previsions, but this is no reason to regard the lower prevision as a model for partial information about an unknown linear prevision.

The observational equivalence of min-maxing and robust-satisficing asserts that either can be used as a mathematical representation of the other. The behavioral difference between these methods asserts that real decision makers with specific beliefs and requirements need not be indifferent between these methods.

Yakov Ben-Haim, Clifford C. Dacso, Jonathon Carrasco, Nithin Rajan (2009, pp. 1061-1062)

Note: Fig. 14 is mine. It is very similar to the original.

As clearly indicated by the Maximin Theorem, the authors' conclusion regarding the relationship between Info-Gap's robustness model and Wald's Maximin model is out and out false!

More specifically, the authors' statement

In conclusion, the observational equivalence between min-maxing and robust-satisficing means that modellers can use either strategy to describe observed behavior of decision makers. In contrast, the behavioral difference means that actual decision makers will not necessarily be indifferent between these strategies, and will choose a strategy according to their beliefs and aspirations.

is grossly misleading.

Keep in mind that the min-max model that this assertion refers to is the ill-considered, hence inappropriate, min-max model which, as I pointed out already, Ben-Haim always deploys to "make the case" for the alleged dissimilarity between Info-Gap and Maximin:

Ben-Haim's ill-considered Minimax Model
α(û;α'):=   minmax r(d,u)
d∈D  u∈U(α',û)  

This ill-conceived min-max model is discussed in detail in the article Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model and in FAQ # 20.

The authors' statement does not apply to

Wald's generic Maximin Model
z*:=   max min g(x,s)
x∈X   s∈S(x)  

Indeed, the Maximin Theorem assures us that Info-Gap's robustness model is a simple instance of this model.

For this reason the authors' statement is as misleading and as hair-raising as the statement

q(x) = 1 + x + x2     is not a polynomial

The point is this.

To so much as attempt a comparison between Info-Gap’s robustness model and Wald's Maximin model, in a manner suggesting that the two models are equals, is already grossly misleading, because such a comparison puts a general prototype model on the same footing with one of its countless instances.

Wald's Maximin model, as anyone in the business of decision theory would no doubt know, is an all-embracing, extremely flexible, hence powerful paradigm. This means of course that it has the inherent ability to subsume infinitely many instances. And as indicated by the Maximin Theorem, Info-Gap’s robustness model is one of these instances. So, as one of the countless instances of Wald's generic Maximin model, Info-Gap's robustness model will (surprise, surprise!) be different from other instances. Hence, as one of the instances of Wald's generic Maximin model, Info-Gap's robustness model is different from the ad hoc model formulated by Ben-Haim, which is equally an instance of Wald's generic Maximin model.

This fact merely indicates the obvious, namely that Wald's generic Maximin model is incomparably more general and powerful than Info-Gap's robustness model.

All one needs to do to prove formally that Info-Gap's robustness model is an instance of Wald's Maximin model is to show that at least one of the infinitely many instances of Wald's generic Maximin model is equivalent to Info-Gap's robustness model.

Instead, Ben-Haim, prefers to engage in empty rhetoric. That is, rather than go for the legitimate, perfectly suitable instance that is staring at him from the Maximin Theorem, Ben-Haim always picks his ill-considered instance of Wald's Maximin that is not equivalent to Info-Gap's robustness model, to "make a case" for an alleged difference” between Info-Gap's robustness model and Maximin.

All this means then is that the authors' conclusion applies only to the ill-considered Minimax model that they formulate in this paper. It does not apply to the simple instance of Wald's generic Maximin model that is specified by the Maximin Theorem. Indeed, this instance of Wald's Maximin model always yields the same results as those yielded by Info-Gap's robustness model. Therefore, the difference between these two models, namely Info-Gap's robustness model and the Maximin model specified by the Maximin Theorem, has to do with style not with substance. And as a consequence, the decision makers' choice between the two models has absolutely nothing to do with their beliefs and aspirations. Both are Maximin models and both yield the same results.

And so, the authors erroneous thesis does not prove that Info-Gap's robustness model is not a Maximin model. All it proves is the authors' obvious misconceptions about the modeling aspects of the Maximin/Minimax paradigm, their misapprehension as to how the Maximin/Minimax paradigm handles constraints, and so on. All this bars them from grasping the full extent of the affinity between Info-Gap's robustness model and Wald's Maximin model.

Note that the Maximin Theorem is constructive: it sets out a simple recipe for constructing the instance of the generic Maximin model that represents Info-Gap's robustness model.

No amount of fog/rhetoric/spin can change this bottom line.

As indicated above, the conceptual and technical mistakes that led Ben-Haim astray on this matter are discussed in detail in the article Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model and in FAQ # 20.

The trouble of course is that the avalanche of empty rhetoric and spin hides all this from view. What is more, it hides from view some fundamental technical errors.

Elementary Technical Matters

Unfortunately, even in this department the authors are on extremely shaky grounds.

For instance, consider the statement

Epistemically, the info-gap and min–max approaches are similar in representing uncertainty without measure functions, though the min-max approach requires the choice of a specific set.

How did the authors come up with this manifestly erroneous idea? Where did they get it from?

Consider the generic Maximin model

Wald's Maximin Model
z*:=   max min g(x,s)
x∈X   s∈S(x)  

Here the uncertainty space is associated with the inner (min) player and is denoted by S(x), where x denotes the decision selected by the outer (max) player. So, as it is there for all to see, the uncertainty space of the inner (min) player is not fixed at all: it is allowed to depend on the choice made by the outer (max) player. But what is more, nothing here "requires the choice of a specific set".

As a matter of fact, the uncertainty sets (S(x): x∈ X) are extremely flexible, certainly far more so than the uncertainty sets (U(α,û): α>0) deployed by Info-Gap decision theory. In particular, the sets (S(x): x∈ X) are not required to have any specific structure at all thus giving the modeller great freedom in their formulation. In contrast, the sets (U(α,û): α>0) deployed by Info-Gap decision theory are required to be nested (increasing with α).

Of course, this is precisely what makes Info-Gap's robustness and opportunessness models local, hence utterly unsuitable for decision-making under severe uncertainty. The nesting property of the uncertainty regions (U(α,û): α>0) prevents Info-Gap’s robustness and opportunessess models from exploring globally the complete uncertainty space.

More importantly, perhaps, Info-Gap decision theory assumes that the uncertainty regions [U(α,û): α>0] are not dependent on the decisions made by the decision-maker. In contrast, the Maximin model allows the sets [S(x): x∈ X] to depend on the decisions made by the decision maker.

To see how these constricting requirements of Info-Gap's uncertainty model greatly limit its capabilities, observe that the Maximin formulation of Info-Gap's uncertainty model is based on the following setup

x = (d,α)

S(d,α) = U(α,û)

recalling that the sets [U(α,û): α≥0] are nested (increasing with α).

The point to note here is that in this setting, the flexibility allowed by the Maximin model cannot be (indeed, is not) fully utilized in that S(d,α) does not depend on d, and furthermore, the sets (S(d,α): α≥0) are nested (increasing with α).

By the same token, the objective function of the Maximin model is much more general than the objective function of Info-Gap's robustness model. That is, the Maximin model allows its objective function g to be any real-valued function of x and s, whereas in the framework of Info-Gap's robustness model the objective is to maximize the value of α. Thus, as indicated above, the Maximin setup for Info-Gap's robustness model is

x = (d,α); s=u

g(d,α,u) = α·(r(u,d) ◊ r*)

observing that g(d,α,u) is equal to either α or 0.

In sum, the Maximin paradigm provides a modeling medium that is incomparably more general and powerful than the one offered by Info-Gap's robustness model. Is it surprising then that Info-Gap's robustness model is a simple instance of Wald's generic Maximin model?

Conclusions

For the benefit of readers who are not familiar with Info-Gap decision theory I need to point out that the dissertation in this paper, on the alleged differences and similarities between Info-Gap’s robustness model and the Maximin model, must be placed in its proper context. Observe then that the discussion in this paper comes in the wake of a long list of grossly misleading and unsubstantiated assertions on Info-Gap's role and place in decision theory and its capabilities to deal with severe uncertainty.

For instance, consider this (empahsis is mine):

Info-gap decision theory is radically different from all current theories of decision under uncertainty. The difference originates in the modeling of uncertainty as an information gap rather than as a probability. The need for info-gap modeling and management of uncertainty arises in dealing with severe lack of information and highly unstructured uncertainty.

Ben-Haim (2006, p. xii)

and
In this book we concentrate on the fairly new concept of information-gap uncertainty, whose differences from more classical approaches to uncertainty are real and deep. Despite the power of classical decision theories, in many areas such as engineering, economics, management, medicine and public policy, a need has arisen for a different format for decisions based on severely uncertain evidence.

Ben-Haim (2006, p. 11)

and
Optimization of the robustness in eq. (3.172) is emphatically not a worst-case analysis. In classical worst-case min-max analysis the decision maker minimizes the impact of the maximally damaging case. But an info-gap model of uncertainty is an unbounded family of nested sets: U(α,û), for all α≥0. Consequently, there usually is no worst case: any adverse occurrence is less damaging than some other more extreme event occurring at a larger value of α. What esq. (169) and (3.172) express is the greatest level of uncertainty consistent with satisficing to level rc. When the decision maker chooses the action q to maximize α(q,rc), what is maximized is the immunity to an unbounded ambient uncertainty. The closest this comes to "min-maxing" is that the action is chosen so that "bad" events (causing reward R* less than rc) occur as "far away" as possible (beyond a maximized value of α).
Ben-Haim (2006, p. 101 )

Or consider the opening paragraph of a paper posted on the web site FloodRiskNet in the UK since November 2007:

Making Responsible Decisions (When it Seems that You Can't)
Engineering Design and Strategic Planning Under Severe Uncertainty

What happens when the uncertainties facing a decision maker are so severe that the assumptions in conventional methods based on probabilistic decision analysis are untenable? Jim Hall and Yakov Ben-Haim describe how the challenges of really severe uncertainties in domains as diverse as climate change, protection against terrorism and financial markets are stimulating the development of quantified theories of robust decision making.

Hall and Ben-Haim, 2007, p. 1

These assertions can easily give the impression that Jim Hall and Yakov Ben-Haim have made a colossal breakthrough in decision theory. Indeed that they have managed to devise a methodology for responsible decision-making in the face of severe uncertainty that is capable of the most incredible feats.

The truth of course is summed up by the following facts:

The paper under review here is further testimony to the fact that Ben-Haim and his followers have no qualms to continue promulgating the same old myths about Info-Gap decision theory. Their method for dealing with the challenges exposing these myths for what they are is to ... intensify the fog, spin, and rhetoric.

Remarks

Review # 13 (Posted: February 19, 2010; Last update: February 25, 2010)

Reference: Tracy M. Rout, Colin J. Thompson, and Michael A. McCarthy
Robust decisions for declaring eradication of invasive species
Journal of Applied Ecology 46, 782–786, 2009.
Summary

1. Invasive species threaten biodiversity, and their eradication is desirable whenever possible. Deciding whether an invasive species has been successfully eradicated is difficult because of imperfect detection. Two previous studies [Regan et al., Ecology Letters, 9 (2006), 759; Rout et al., Journal of Applied Ecology, 46 (2009), 110] have used a decision theory framework to minimize the total expected cost by finding the number of consecutive surveys without detection (absent surveys) after which a species should be declared eradicated. These two studies used different methods to calculate the probability that the invasive species is present when it has not been detected for a number of surveys. However, neither acknowledged uncertainty in this probability, which can lead to suboptimal solutions.

2. We use info-gap theory to examine the effect of uncertainty in the probability of presence on decision-making. Instead of optimizing performance for an assumed systemmodel, info-gap theory finds the decision among the alternatives considered that is most robust to model uncertainty while meeting a set performance requirement. This is the first application of info-gap theory to invasive species management.

3. We find the number of absent surveys after which eradication should be declared to be relatively robust to uncertainty in the probability of presence. This solution depends on the nominal estimate of the probability of presence, the performance requirement and the cost of surveying, but not the cost of falsely declaring eradication.

4. More generally, to be robust to uncertainty in the probability of presence, managers should conduct at least as many surveys as the number that minimizes the total expected cost. This holds for any nominal model of the probability of presence.

5. Synthesis and applications. Uncertainty is pervasive in ecology and conservation biology. It is therefore crucial to consider its impact on decision-making; info-gap theory provides a way to do this. We find a simple expression for the info-gap solution, which could be applied by eradication managers to make decisions that are robust to uncertainty in the probability of presence.

Acknowledgement Many thanks to Yakov Ben-Haim and Dane Panetta for helpful advice and discussions. Also thanks to Michael Bode, Mark Burgman, Peter Baxter and three anonymous reviewers for comments on this manuscript. This research was supported by an Australian Postgraduate Award, the Commonwealth Environment Research Facility (AEDA), the Australian Centre of Excellence for Risk Analysis and an Australian Research Council Linkage Grant to MMcC (LP0884052).
Scores TUIGF:100%
SNHNSNDN:100%
GIGO:90%

Overview

My decision to review this article was triggered by the discovery that the info-gap model proposed in the paper as a framework for the modeling, analysis, and solution of the problem under consideration is not in fact a "proper" info-gap model. Or, to put it more accurately, strictly speaking the proposed model is not an info-gap model because it violates the Nesting Axiom of info-gap decision theory.

But as I began to examine the paper more carefully, it turned out that there are other matters that require critical comment.

I wish to point out, though, that at present I am working on a number of more urgent projects, so the amount of time I can devote to this review is quite limited. I shall therefore have to come back to it from time to time to enlarge on the current discussion.

This is a draft of Work in Progress

Last update: Wednesday, 24-Feb-2010 20:17:09 MST

At this stage I focus on the following points:

I now turn to a more detailed discussion of these points.

Errata

The mathematical analysis in the paper is not sufficiently careful. As a consequence, some of the results are incorrect. Some of the assertions may eventually prove to be correct, but the arguments supporting them are incorrect and/or incomplete.

For the purposes of this discussion it is sufficient to mention the following:

In a word, the results presented in this paper require corrections. This can be done either by imposing additional conditions on the problem so as to validate the results. Or by re-working the mathematical analysis properly to obtain results that are valid under the assumptions postulated in the paper.

The problem

It is important to be clear on what I mean when I say that the problem studied in this paper is a trivial problem.

Obviously, by this I do not want to suggest that the problem is unimportant in the disciplines of applied ecology, conservation biology, and so on. Indeed, the problem may have enormous consequences for applied ecology and conservation biology.

The point I am making is entirely different. My point is that when the "real world" situation that is investigated in this paper is cast as a decision-making problem and given a mathematical formulation, it becomes immediately clear that the mathematical problem is trivial in the sense that its solution flows (more or less) directly from its mathematical statement. As a matter of fact, the essential part of the problem can be solved by inspection.

Granted, you may contend that this does not argue against using info-gap for this purpose. Indeed, why not use info-gap to solve a problem that readily lends itself to solution by inspection? To which I would say, paraphrasing a reply once given by the renowned mathematician Richard Bellman (founder of dynamic programming): you may just as well tie a chair to your leg and attempt to swim across the river in this state.

My point is then that not only is the use of info-gap as a framework for modeling analysis and solution totally superfluous in this case. It is in fact counter-productive, if not harmful, because it gives a distorted picture of the problem in question. Worse, it gives a totally distorted picture of the real issues that we face in robust decision-making in the face of uncertainty.

To set the stage for the discussion, recall that the elements of the eradication problem under consideration are as follows:

To simplify the notation, we assume (with no loss of generality) that Cs=1, and we let E=Ce, and C=Nc. Also assume that C is an integer and let k=C+1.

So,

Statement of the problem:

Find the most robust decision d in D={1,2,3,...,k} against the uncertainty in the true value of u ∈[0,1] with respect to the performance requirement

(1)           d + uE ≤ C+1

where E and C are given positive numeric scalars, typically much greater than 1. The value of k is relatively small, say much smaller than 1000.

Note that because D is a discrete set, this problem is a discrete optimization problem.

Finally, it is important to stress that the uncertainty in the true value of u is assumed to be "considerable" (page 783) and that no probabilistic or likelihood assessments of this uncertainty are provided.

The essence of the problem

Roughly, in this context, decision d is robust if it satisfices the performance requirement d + uE ≤ C+1 for a large number of values of u∈[0,1]. The larger the number, the more robust d is. That is, we are searching for a decision whose feasible region of u, namely

(2)           F(d):= {u∈[0,1]: d + uE ≤ C+1} , d∈D
is large. The larger this region is the more robust the decision.

So, the point to note here is that the robustness that this problem aims to determine is not merely that of robustness against the uncertainty in the probability of presence. Rather, the idea here is to identify decisions that are robust against the impact of the uncertainty in the true value of this parameter on the budgetary constraint stipulated in (1).

Admittedly, as things stand, the problem is not sufficiently defined. That is, we still lack a formal definition of 'robustness'. Also, information is required regarding the nature of the uncertainty in the true value of u. Note that the latter affects the former.

And yet, despite all this, it is clear that this problem is trivial. To wit, it involves:

Thus, from the perspective of robust optimization, this is a "trivial problem".

But more than this, one need not even be a "professional" mathematician or an expert in robust optimization to figure out that

(3)           F(d) = [0,u*(d)] , d∈D
where
(4)           u*(d):= (C+1-d)/E

This follows directly (by inspection) from the definition of the performance requirement (1). For obvious reasons, we shall refer to u*(d) as the critical value of u associated with decision d.

It is immediately clear that the critical value of u would be pivotal in the definition of robustness against the uncertainty in the true value of u.

So:

In short, as can be gathered from the above, the formal definition of robustness would express "adjustments" made in these critical values in relation to what we know about the uncertainty in the true value of u.

The solution

Having clarified the issues that bear on the solution of the problem under consideration, we can now outline its solution procedure. To this end let ρ(d) denote the robustness of decision d against the uncertainty in the true value of u.

Solution Procedure

Note that the first step is easy: we compute the critical values according to the formula given in (4). The third step is also easy because the set of feasible decisions D is small, hence we can conduct the maximization of ρ(d) over D by enumeration. This means that a spreadsheet can be easily set up to solve this problem.

Of course, in cases where the maximization of ρ(d) over D={1,...,k} is also amenable to solution by analytic methods, it might be possible to obtain a closed-form solution for the continuous counterpart of the (discrete) problem that is under consideration. This would depend on the definition of robustness used.

What emerges therefore is that the real issue is in Step 2: the definition of the robustness function ρ=ρ(d).

Robustness against uncertainty

Let us examine a number of cases, representing various degrees of uncertainty in the true value of u. These cases refer to an estimate of the true value of u. Since this estimate may depend on d, we denote it û(d), d∈D.

Case 1: Certainty.

In this case we assume that the estimates û(d),d∈D are "perfect" and are equal to the respective true values of u. Thus, there is no uncertainty in the true value of the parameter.

This means that any decision is either thoroughly robust or utterly fragile. That is, d is thoroughly robust iff û(d) ≤ u*(d). Otherwise it is utterly fragile (infeasible).

Conclusion:

d is robust iff û(d) ≤ (c+1-d)/E

d is fragile iff û(d) > (c+1-d)/E

Note that in this extreme situation -- a perfect estimate -- there is even no need to use the solution procedure outlined above. We simply do as follows:

(5)           min {d + û(d)E: d∈D}

If the estimates û(d) are given by a nice smooth formula, it may be possible to identify robust decisions analytically by minimizing, over [1,k] the expression d + û(d)E, namely

(6)           min {d + û(d)E: d∈[1,k]}

If the optimal value of this expression is not greater than C+1, then the optimal value of d is the most robust decision. If this value is not a positive integer, then the nearest integer neighbors will have to be compared. And if the optimal value of this expression is greater than C+1, then all the decision are fragile.

Case 2: The estimates are very good

If the estimates û(d),d ∈ D, are very good, we would argue that it is in fact unnecessary to explore values of u that are substantially different from (much smaller or larger than) these estimates.

So, the following definition of robustness would be appropriate

(7)           ρ(d):= u*(d) - û(d) , d∈D

That is, the robustness of decision d is the "distance" between the estimate and the sub-region of [0,1] where u violates the performance requirement for this decision. For d to be robust, this distance should be large. The larger this distance the greater the robustness.

Thus, to find the optimal decision, we solve the following optimization problem:

(8)           max { ρ(d): d∈ D} = max {[C+1-d - û(d)E]/E: d∈D}

which is equivalent to

(9)           min {d + û(d)E: d∈D}

which is equivalent to the problem in Case 1: Certainty.

Case 3: The Estimates are very poor

In this case the estimates can be substantially wrong, namely no more than wild guesses, or perhaps just rumors. It may therefore be best to ignore them altogether.

Under the circumstances we may let the robustness ρ(d) be defined as follows

(10)           ρ(d):= u*(d) , d∈D

By inspection, in this case the most robust decision is d=1. Its robustness is equal to C/E. Note that if E≤C, then d=1 is super-robust: it satisfies the performance constraint for all u∈ [0,1].

Again, the situation is so simple that we need not even use the procedure outlined above.

Remarks:

  • Should you decide to incorporate the estimates in the definition of robustness knowing that the estimate is very poor, then make sure that you do not give the estimates too much 'weight'.

  • And do not forget to examine the impact of errors in the estimates on the results.

The Moral of the Story

We can continue in this vein: formulating "sensible" definitions of robustness for our small problem ad infinitum ...

For example, we may want to "scale" the critical values u*(d),d∈D by dividing them by the corresponding estimates û(d),d∈D; or we may even want to "normalize" the critical values and consider [u*(d)-û(d)]/û(d) as the measure of robustness of decision d -- as done in the paper. Note that these two alternatives are equivalent.

More generally, we can incorporate "weights", call them w(d), d∈D, to refine the robustness function ρ. In particular, in cases where the estimates are very good, we can let

(11)           ρ(d) := [u*(d) - û(d)]/w(d) = [C+1-d - û(d)E]/w(d)E

where w(d) >0 is the weight associated with decision d.

The point is then that the simplicity of the performance constraint implies that the critical values u*(d),d∈D can be easily determined by inspection, and then be invoked in the definition of robustness.

Remarks:

  • Note that by definition, (11) allows ρ(d) to take negative values. This occurs in cases where u*(d) < û(d), namely when decision d is fragile in the immediate neighborhood of û(d). In other words, ρ(d) is negative when d violates the performance requirement d + uE ≤ C+1 for values of u that are ever-so-slightly greater than û(d).

    This means that û(d) is a measure of the robustness and fragility of d: if &rho(d) > 0, then d satisfices the performance requirement in the immediate neighborhood of û(d); and if &rho(d) < 0, then d violates the performance requirement in the immediate neighborhood of û(d).


  • The estimates û(d),d∈D can be quite small for large values of d. For example, in the article û(d)=0.136d, hence û(5) = 0.00004652587. Hence, one has to be careful when using robustness measures such as (11) with w(d)=û(d).

Let us now examine how robustness is defined in the article.

Proposed Info-Gap Robustness Models:

In sharp contrast to the effortless manner in which the definitions of robustness are derived above - directly from the statement of the problem, the derivations of robustness in the article get unnecessarily complicated by the requirement to be expressed in terms of an info-gap robustness model.

In fact, the derivations are suffiently complicated that they are not explained in full in the body of the article. You may plough through the article but you will not find the expression (formula) used to measure the robustness of decisions in the context of the problem under consideration. For this you'll have to read Supplement S1.

But more than this, the derivations of robustness in the article and in the two supplements make no reference to the intuitive notion critical value of u(d). As we have seen above, this concept in fact brings out what robustness is all about in the problem studied in the article.

When you finally figure out what it is (eqn A5 in Appendix S1), you'll discover that it is the instance of the robustness model specified by (11) that corresponds to

(12)           w(d) = û(d)     ---->     ρ(d) = [C+1-d]/[û(d)E] - 1

The info-gap robstness model specified in eqn A2 in Appendix S2 corresponds to

(13)           w(d) = 1 - û(d)     ---->     ρ(d) = [C+1-d - û(d)E]/[E(1-û(d))]

The point to note about these two info-gap models is that they correspond to the definitions of robustness falling under what we refer to above as Case 2: The Estimates are Very Good.

Indeed, this is what is so interesting about these two info-gap models: the fact they that they correspond to the definitions falling under what we refer to above as Case 2: The Estimates are Very Good. Namely, for the models to make sense the estimates should be asssumed to be good, but ... the authors assume that the true value of u " ... is subject to considerable uncertainty and ignoring this uncertainty may lead to suboptimal solutions ..." (p. 783; emphasis is mine).

No comment whatsoever is made to explain the blatant incongruity between the fact that (as assumed in the paper) the estimates are poor (due to the considerable uncertainty in the true values of the parameters) and the fact that they are pivotal in the determination of robustness. Moreover, no sensitivity analysis is conducted on the estimates themselves!

Stay tuned ...

The 64K$ question

Given then the utter simplicity with which the problem under consideration can be analyzed and solved, I repeat the question raised above: why use info-gap decision theory to solve this problem?

Surely the authors should explain the rational, the point, the merit of using info-gap decision theory as a framework for the modeling, analysis and solution of a problem whose solution is simplicity itself.

I might add in this regard that over the past five years I have written a number of letters to authors of papers on info-gap decision theory. Occasionally, the letter is about the triviality of the problem under consideration.

So I prepared a generic letter, which I modify according to the circumstances. Here is a version intended for cases where the problem is "trivial":

Dear ??????:

I read with interest your paper entitled ??????.

Note that the essence of the problem investigated in this paper boils down to this:

Determine the critical value of ?????, namely the worst (largest) value of ?????? that satisfies the performance constraint ????? or equivalently ?????.

Therefore, my immediate reaction to the analysis in your paper was sheer amazement!

After all, by inspection, the answer is obviously ??????. Therefore, one cannot help but wonder how Info-Gap suddenly appears on the scene ?!

This is yet another example of what can happen when instead of trying to model and solve a given problem, one tries -- by hook or by crook -- to use a given methodology to model and solve this problem.

On a number of occasions I alluded to this danger. But here I must be more forthright.

This article is a good example of how easily one can end up focusing almost exclusively on manipulating the formulation of a given problem so as to fit it into the paradigm of a Beloved methodology. So much so that one may fail to see that the problem is actually so trivial that it can be easily solved by inspection.

Isn't it time, ??????, that we asked ourselves:
Are we in the business of developing, using and promoting scientific methods for decision-making under uncertainty in the area of ??????, or are we in the Info-Gap business? I cannot see how we can make progress on the important issues that we identified if we keep ourselves busy trying to fix conceptual and technical Info-Gap bugs.

Best wishes

    Moshe
Melbourne (date:??????)

This generic letter applies to the paper under review.

Stay tuned for more ...

The Continuing Maximin Saga

As indicated above, now that Bryan Beresford-Smith and Colin J. Thompson (2009) have conceded that Info-Gap's robustness model is a Maximin model (see Review 11), what is the point of withholding this fact from scientists in the field of applied ecology?

Stay tuned more ...

The Ongoing Severe Uncertainty Saga

The authors concede that info-gap decision theory is unsuitable for situations where the estimate is likely to be substantially wrong. For consider this:

Although info-gap theory is relevant for many management problems, two components must be carefully selected: the nominal estimate of the uncertain parameter, and the model of uncertainty in that parameter. If the nominal estimate is radically different from the unknown true parameter value, then the horizon of uncertainty around the nominal estimate may not encompass the true value, even at low performance requirements.
Rout et al (2009, p. 785)

However, their explanation of this fact is totally wrong.

To begin with, a distinction must be drawn between:

Insofar as the problem statement is concerned, the uncertainty is described by the uncertainty space under consideration, call it U, and the estimate, call it û. Needless to say, the estimate û and the true value of u are assumed to be elements of U. Obviously, if the estimate is poor the complete uncertainty space can be vast. This explains why, according to Ben-Haim (2006, p. 210), the most commonly encountered info-gap uncertainty models are unbounded.

Enter info-gap.

Given this, the info-gap model of uncertainty is constructed so that its regions of uncertainty, call them U(α,û),α≥0, are centered at û, and at least one of them contains U. Thus, if the uncertainty space U is vast, so would be the regions of uncertainty U(α,û) for large values of α.

That said, it is clear that the real trouble with info-gap's robustness analysis is not that the true value of u may not be contained in the uncertainty space of U.

Comment

I should add that I have yet to come across an info-gap publication where it is not immediately obvious that the uncertainty space U contains the (unknown) true value of u. So, for our purpose here it would be best to leave it at that: the true value of u is unknown, but it is contained in the complete uncertainty space. This means that it is contained in at least one of the regions of uncertainty centered at the estimate.

Indeed, it is ironic that the authors should raise this issue at all in this paper. After all, in the case of the problem they investigate, the unknown parameter under consideration is a probability, hence the (unknown) true value of the parameter of interest is definitely in the bounded interval [0,1].

The real trouble with info-gap's analysis lies elsewhere. It lies in info-gap's localized robustness analysis.

That is, info-gap's robustness model conducts the robustness analysis, in the first instance, in the immeidate neighborhood of the given estimate. This means that a decision that violates the performance constraint at a u near the estimate is deemed fragile regardless of its performance in neighborhoods of the uncertainty space that are further way from the estimate.

By definition, therefore, info-gap decision theory does not seek decisions that are robust against uncertainty over the given uncertainty space U. It seeks decisions that are robust in the neighborhood of the given estimate û.

So, the difficulties that info-gap's analysis would run into would remain even if the (unknown) true value of the parameter would be contained in the uncertainty space stipulated by the problem statement. In a word, the trouble is with info-gap's lame "search methodology" which a-priori undermines its ability to properly explore the uncertainty space especially under conditions of --- what the authors term --- "considerable" uncertainty.

The following picture illustrates this point.

It shows the rewards R(q,u) generated by two decisions, q' and q'', as a function of some parameter u. The estimate of the true value of u is û = 0, the uncertainty space is U=(-∞,∞) and the performance requirement is R(q,u) ≥ 0.

According to Info-Gap's robustness model, q'' is more robust than q' with respect to R(q,u) ≥ 0 because the closest u to û that violates the constraint R(q',u) ≥ 0 is at a distance α'=1.08 from û, whereas the closest u to û that violates the constraint R(q'',u) ≥ 0 is at a distance α''=1.429 from û. Hence, q'' is more robust than q'.

Note, however, that

This example also illustrates why Info-Gap's robustness analysis cannot handle ordinary, plain, white Swans, let alone genuine (Australian) Black Swans.

See the discussion on this issue at Second Opinion on Info-Gap Decision Theory.

And how about this:

Thus, the method challenges us to question our belief in the nominal estimate, so that we evaluate whether differences within the horizon of uncertainty are 'plausible'. Our uncertainty should not be so severe that a reasonable nominal estimate cannot be selected.
Rout et al (2009, p. 785)

Since info-gap decision theory is non-probabilistic and likelihood-free, info-gap users are in no position to quantify levels, or degrees, or what have you, of "good". "reasonable", or "bad" that are applicable to the estimate. Nor are they in any position to determine what is more or less "plausible" within the horizon of uncertainty even when the plausible is only 'plausible'.

Add to this the fact that info-gap decision theory does not even begin to deal with the question of how the estimates are obtained. Namely, info-gap decision theory does not bother to give us so much as a clue on how to check/verify whether the estimate is bad, poor, good, excellent, perfect and it is clear that determining the quality of the estimate is an external issue. In other words, you come to info-gap decision theory with an estimate in hand. And in this case as well, there is nothing in info-gap decision theory itself that would enable it to distinguish between the quality of various estimates.

So what are the authors telling us?

The authors seem to be saying the obvious: unless you have good reasons to believe that the estimate you have is "pretty reasonably good" (whatever that means), it makes no sense to focus the robustness analysis on the immediate neighborhood of the estimate. In other words, it makes no sense to do what info-gap prescribes doing. In this case the authors agree with my criticism of info-gap decision theory.

But more than this, are the authors willing to stick their necks out and declare that an estimate subject to "considerable" uncertainty (be it a rumor or gut feeling or whatever) is so good that it makes sense to confine the robustness analysis to a given neighborhood of the estimate and call it a day?

In this case, the authors would have to do as follows:

  1. Stipulate the uncertainty space of the problem, call it U. That is, they would have to specify the smallest set that (the decision-maker is reasonably confident) contains the true value of the parameter of interest.

  2. Specify the value of the estimate.

  3. Conduct a robustness analysis that seeks decisions that are robust on the given uncertainty space U.

But this, one need hardly point out, is not what info-gap decision theory does!

Info-gap decision theory does not seek decision that are robust on U. It seeks decisions that are robust in the neighborhood of the given estimate, to wit: a decision that is not robust in the immediate neighborhood of the estimate is eliminated from any further consideration even if it performs exceptionally well in other neighborhoods of U.

So, ... how are the authors going to use info-gap decision theory to identify decisions that are robust on U rather than in the immediate neighborhood of the estimate?

In any case, suppose that the uncertainty is not so severe and we have in hand a reasonably good estimate. How then can the local analysis in the neighborhood of this estimate enable dealing with rare events, catastrophes etc? Hence, how about this:

For ecological management in the face of uncertainty, managers may use info-gap to gain some protection against catastrophic outcomes by answering the question: how wrong could this model be before outcomes are unacceptably bad?
Rout et al (2009, p. 785)

Note that info-gap decision theory does not -- indeed, is in principle unable to -- answer the question: how wrong could this model be before outcomes are unacceptably bad?

This is so because the true value of the parameter of interest is unknown and is subject to considerable uncertainty. There is therefore no way of knowing how wrong the model is.

Info-gap robustness model answers a completely different question, namely: what is the largest region of uncertainty around the estimate over which the performance constraint is satisfied?

Strictly speaking, this question has nothing to do with uncertainty as such. Moreover, the answer to this question does not depend on the "quality" of the nominal point, or estimate, used as the center point of the regions of uncertainty.

See discussion on this topic in my response to Burgamn's comments on my criticism of info-gap decision theory.

In short, info-gap decision theory does not -- much less is it able to -- deal with the question stated by the authors and it therefore cannot help managers gain protection against catastrophic outcomes.

As a matter of fact, info-gap decision theory constitutes the precise antithesis of what a theory for modeling, analyzing and managing severe uncertainty ought to be. Instead of exploring thoroughly the given complete uncertainty space, info-gap decision theory focuses its robustness analysis in the neighborhood of a point estimate.

Remarks:

This is definitely a move in the right direction, but ... it does not go far enough!

Stay tuned for more ...

The State of the Art Saga

The paper shows complete disrespect for the state of the art in decision-making under uncertainty. The discussion section basically amounts to an uncritical endorsement of info-gap decision theory. Well established methods that have become the "bread and butter" approaches to decision-making subject to uncertainty are not even mentioned. The paper thus provides an extrememly distorted picture of the area of decision-making under severe uncertainty.

It also ignores relevant publications that are critical of info-gap decision theory.

Stay tuned for more ...

Unsubstantiated and/or misleading statements

The paper is riddled with "problematic" statements. I shall mention just a few.

Stay tuned for more ...

Conclusions

The problem examined in the article is trivial, so much so that its essence can be solved by inspection.

The main results, namely (p. 782):

3. We find the number of absent surveys after which eradication should be declared to be relatively robust to uncertainty in the probability of presence. This solution depends on the nominal estimate of the probability of presence, the performance requirement and the cost of surveying, but not the cost of falsely declaring eradication.

4. More generally, to be robust to uncertainty in the probability of presence, managers should conduct at least as many surveys as the number that minimizes the total expected cost. This holds for any nominal model of the probability of presence.

are unsubstantiated and certain assertions made in the article are (technically) wrong.

On the whole, this article is a typical info-gap article. The only point of difference between this article and other info-gap publications is that the authors concede that (despite all the rhetoric in the info-gap literature) it is obvious that this theory is unsuitable for the treatment of severe uncertainty where the estimate is likely to be substantially wrong.

Otherwise, the paper follows the established info-gap line that info-gap decision theory is distinct and radically different from "common" theories for decision under uncertainty. Particularly jarring in this respect is its omission of the fact that info-gap robustness model is in fact a Maximin model.

Consequently the paper gives a thoroughly distorted account of the state of the art in robust decision-making under uncertainty.

Welcome to Factland

This contribution is dedicated to the Info-Gap people at Wikipedia. They were searching for a formal proof that ....

 

Fact 1: Info-Gap is a simple instance of Wald's Maximin model [1945].

 

Fact 2: Info-Gap does not deal with severe uncertainty: it simply ignores it.

Recent Articles, Working Papers, Notes

Also, see my complete list of articles
    '
  • Sniedovich M. (2009) Modeling of robustness against severe uncertainty, pp. 33- 42, Proceedings of the 10th International Symposium on Operational Research, SOR'09, Nova Gorica, Slovenia, September 23-25, 2009.

  • Caserta, M., Voss, S., Sniedovich, M. (2009) Applying the corridor method to a blocks relocation problem, OR Spectrum in press.

  • Sniedovich M.(2009) A Critique of Info-Gap Robustness Model. In: Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 2071-2079, Taylor and Francis Group, London.
    • .
  • Sniedovich M.(2009) A Classical Decision Theoretic Perspective on Worst-Case Analysis, Working Paper No. MS-03-09, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)

  • Caserta, M., Voss, S., Sniedovich, M. (2008) The corridor method - A general solution concept with application to the blocks relocation problem. In: A. Bruzzone, F. Longo, Y. Merkuriev, G. Mirabelli and M.A. Piera (eds.), 11th International Workshop on Harbour, Maritime and Multimodal Logistics Modeling and Simulation, DIPTEM, Genova, 89-94.

  • Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)

  • Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)

  • Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08
    This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file )

  • Sniedovich, M. (2008) A call for the reassessment of Info-Gap decision theory, Decision Point, 24, 10.

  • Sniedovich, M. (2008) From Shakespeare to Wald: modeling wors-case analysis in the face of severe uncertainty, Decision Point, 22, 8-9.

  • Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!, Journal of Risk Finance, 9(3), 287-291.

  • Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)
    In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model.

  • Sniedovich. M. (2008) The Mighty Maximin! (PDF File)
    This paper is dedicated to the modeling aspects of Maximin and robust optimization.

  • Sniedovich, M. (2007) The art and science of modeling decision-making under severe uncertainty, Decision Making in Manufacturing and Services, 1-2, 111-136. (PDF File) .

  • Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)
    In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory.

  • My reply (PDF File) to Ben-Haim's response to one of my papers. (April 22, 2007)

    This is an exciting development!

    • Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.

    • Ben-Haim is not familiar with the fundamental concept point estimate. He does not realize that a function can be a point estimate of another function.

      So when you read my papers make sure that you do not misinterpret the notion point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.

      Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!


  • A formal proof that Info-Gap is Wald's Maximin Principle in disguise. (December 31, 2006)
    This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)
    It shows that Info-Gap is not a new theory but rather a simple instance of Wald's famous Maximin Principle dating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context of Game Theory (1928).

  • A proof that Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)
    This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File) .
    It shows that because Info-Gap deploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust.

  • A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)
    This is a very short article entitled The GAP in Info-Gap (PDF File) .
    It is a math-free version of the paper above. Read it if you are allergic to math.

  • A long essay entitled What's Wrong with Info-Gap? An Operations Research Perspective (PDF File) (December 31, 2006).
    This is a paper that I presented at the ASOR Recent Advances in Operations Research (PDF File) mini-conference (December 1, 2006, Melbourne, Australia).

Recent Lectures, Seminars, Presentations

If your organization is promoting Info-Gap, I suggest that you invite me for a seminar at your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use — let alone promote — Info-Gap as a decision-making tool.

Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.

Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.


Last modified: Wednesday, 24-Feb-2010 23:47:35 MST