Overview
Over the past eight years, a number of myths enunciating certain (erroneous) positions about Info-Gap's place and role in decision theory, its capabilities, and its mode of operation, have taken root in the Info-Gap literature. Recently, some of these myths have been modified somewhat, in response to my criticism of this theory. As a result, newly formulated myths are now beginning to circulate in the Info-Gap literature.
So, given this new development, I decided to dedicate a separate page to a running commentary on it. My objective is not only to set the record straight on the already established myths, but to forewarn readers of this literature that the new ones — apparently aimed at resolving the difficulties in which Info-Gap is entangled — hardly do the job.
To make this discussion accessible to the widest readership possible — readers with limited mathematical background, as well as readers with little or even no knowledge of Info-Gap decision theory itself — I divide the discussion into two parts.
In the first part I discuss the myths and facts in a general non-technical fashion. In the second part ( Appendix) I discuss some of the technical aspects of these myths.
My intention is to follow these developments as they come to light. So, if you have come across something that you believe is relevant to this discussion, but which I had missed, please let me know about it.
By way of introduction, especially for the benefit of readers who have not encountered Info-Gap decision theory before, I need to preface the discussion with these two points:
- First, Info-Gap decision theory was developed by Prof. Yakov Ben-Haim to provide — on his testimony — a non-probabilistic, likelihood-free methodology for robust decision-making under severe uncertainty (Ben-Haim, 2001, 2006). Also, right from its inception Info-Gap has been portrayed and promoted as a distinct, novel, revolutionary theory that is radically different from all current theories for decision under uncertainty (Ben-Haim, 2001, 2006).
- Second, the main ingredients of this theory are: a robustness model and an opportuneness model. It is important to note that both models are by definition local in that they focus on an estimate of the parameter of interest, and confine the entire (robustness) analysis to this estimate and its immediate neighborhood. The working assumption is that, in view of the severe uncertainty, the estimate is a wild guess, a poor indication of the true value and is likely to be substantially wrong.
For a detailed description, analysis, and evaluation of this theory consult the WIKIPEDIA entry and the FAQs about Info-Gap decision theory.
Myths
Topping the list of myths that are advanced about this theory is of course the one which enunciates Info-Gap's role and place in decision theory in general and in robust decision-making in particular.
Myth 1: Info-Gap decision theory is distinct, novel, revolutionary, and radically different from all current theories for decision under uncertainty. The difference is real and deep.
What is particularly striking about this characterization of Info-Gap is not only that it is remarkably bold, but that it is put forward without the slightest argument to corroborate it. Nowhere in his two books on Info-Gap, or in his subsequent publications, does Ben-Haim so much as bother to discuss the state of the art in classical decision theory and/or robust decision-making. Thus, no sustained effort is made to elucidate Info-Gap's relationship to other decision theories so as to bring out in what way is Info-Gap a distinct, different, revolutionary decision theory for the management of severe uncertainty!
And so, the only conclusion that one can possibly reach about this matter — of course taking into account the various assertions in these books and in later publications (or more generally, the Info-Gap literature) — is that this mistaken position about Info-Gap could only be arrived at as a result of a profound misjudgment about the relationship of Info-Gap decision theory to classical decision theory, or a lack of familiarity with the subject, or something to this effect.
And to make matters worse, this erroneous characterization of Info-Gap has been further compounded — apparently, to meet persistent challenges to its validity — by the advancement of another erroneous position denying outright that Info-Gap's robustness model is a Maximin model or that it performs a worse-case analysis.
As you will recall, the preeminent tool offered by classical decision theory and by robust optimization, for the treatment of severe uncertainty, is Wald's famous Maximin Model (circa 1940). And yet, from the start, the position advanced in the Info-Gap literature was that Info-Gap's robustness model is not a Maximin model. Hence,
Myth 2: Info-Gap's robustness model is not a Maximin model.
In response to my proof demonstrating that Info-Gap's robustness model is indeed a Maximin model, we now begin to witness an attempt to adjust the position proclaiming Info-Gap a distinct approach to robust decision-making in the face of severe uncertainty. The trouble is, however, that this readjusted position puts forward a new myth in the Info-Gap literature, namely:
Myth 3: Info-Gap's robustness model is equivalent to Wald's Maximin model.
Also, this newly adjusted position is not necessarily universal. In fact, Ben-Haim continues to hold that Info-Gap's robustness model is not a Maximin model!
As for the position proclaiming Info-gap a distinct method, the irony for Info-Gap is that the only feature that actually sets it apart from other methods of robust decision-making under severe uncertainty, is its fundamentally flawed approach to the management of severe uncertainty. This fundamental flaw is due to its failure to distinguish between local and global robustness.
For, in sharp contrast to other methods in this area, Info-gap's robustness analysis is by definition local. That is, while all other methods proceed on the obvious assumption (without even theorizing about it) that the severe uncertainty enjoins an analysis of the entire uncertainty space or an approximation thereof; Info-Gap, has the (dubious) distinction of being the only method that prescribes a robustness analysis that is confined only to the neighborhood of a (poor) point estimate of the parameter of interest. As a consequence, Info-Gap's prescription for the management of severe uncertainty is to ignore the severity of the uncertainty altogether. And so, while this effectively means that Info-Gap is in principle unable to tackle severe uncertainty, the myth propagated in the Info-Gap literature is that
Myth 4: Info-Gap's robustness analysis seeks decisions that are robust against severe uncertainty.
And if this were not enough, the greater irony is that Info-Gap proponents make a huge fuss about Info-Gap's purported ability to handle unbounded regions of uncertainty. So much so that Ben-Haim finds it necessary to point out that such regions are the most commonly encountered case in applications of Info-Gap decision theory.
The fact that unbounded regions of uncertainty only exacerbate the already deep rooted flaw, namely the "localness" of Info-Gap's analysis and results, has gone unnoticed. That is, the fact that the larger the uncertainty space the more pronounced Info-Gap's failure at managing severe uncertainty becomes, is totally lost on Info-Gap scholars. And so the following is being made much of:
Myth 5: Info-Gap decision theory is particularly well suited to handle unbounded regions of uncertainty.
More generally, Info-Gap decision theory has been hailed, right from its launch, as a non-probabilistic, likelihood-free theory. The term "true" Knightian uncertainty is often used to describe the extreme lack of knowledge that characterizes the uncertainty that Info-Gap decision theory is (purportedly) capable of handling.
Myth 6: Info-Gap decision theory is well suited to handle true Knightian uncertainty.
So, the other plank of my criticism of this theory has been that given these defining characteristics of the theory (and given the severe uncertainty), Info-Gap's prescription for the management of severe uncertainty has no rhyme or reason. That is, as I have been arguing, Info-Gap's prescription to fix on a point estimate, which given the severe uncertainty amounts to a wild guess of the true value of the parameter of interest, and to confine the entire robustness analysis to the neighborhood of this estimate, is utterly senseless.
This criticism has apparently been noted so that attempts are being made to correct this glaring flaw. And so, the statements that we now begin to encounter in the Info-Gap literature maintain that using Info-Gap decision theory — which in the same breath, is being hailed a non-probabilistic, likelihood-free methodology — one is able to impute likelihood to non-probabilistic, likelihood-free, models, so as to obtain the most likely results.
That this entangles Info-Gap in an even deeper contradiction has also gone unnoticed, for the latest inadvertently claim that:
Myth 7: Info-Gap uncertainty model possesses a significant likelihood structure.
Another alleged attribute of Info-Gap decision theory that is being made much of in the Info-Gap literature, is that Info-Gap decision theory has an advantage over what is dubbed in this literature "direct optimization" methods. The claim is that the solutions obtained by methods based on "direct optimization" of reward have zero robustness to uncertainty. In contrast — so the argument goes — solutions obtained by Info-Gap robustness analysis attain the maximum possible robustness. Hence:
Myth 8: Solutions generated by "direct optimization" of rewards have zero robustness to uncertainty.
What is particularly inexcusable about these assertions about Info-Gap's advantage over other optimization methods in obtaining robustness against severe uncertainty is that they are made in a vacuum. No attempt whatsoever is made to substantiate this claim through a serious comparative analysis of Info-Gap decision theory with other optimization methods that were designed specifically for robust decision-making. Worse, no reference whatsoever is made to the rich literature on Robust Optimization — that field in optimization theory whose business is precisely that: obtaining optimal solutions that are robust to uncertainty and/or variability. In a word, Info-Gap proponents hold forth about robustness against uncertainty in total disregard to the advances made over the past forty years in the field of Robust Optimization, thus giving the impression that:
Myth 9: There is no such area of expertise/research as Robust Optimization.
My contention is then that these myths are pure ... myths.
But what are the facts?
Facts
Topping the list of facts are of course those that pull the plug on the first three myths, namely
Fact 1: Info-Gap's robustness model is a simple instance of Wald's famous Maximin model (circa 1940) and Info-Gap's opportuneness model is a simple instance of the classic Minimin model (circa 1950).
Fact 2: Info-Gap's robustness model is a simple instance of Wald's famous Maximin model (circa 1940).
Fact 3: Info-Gap's robustness model is not equivalent to Wald's Maximin model, it is a simple instance of it.
The claim that Info-Gap's robustness model is equivalent to Wald's Maximin model is as erroneous as the claim that the class of polynomials of degree 3 is equivalent to the class of polynomials of degree n (n=positive integer), or the claim that the family of exponential probability distribution functions is equivalent to the family of Weibull distribution functions.
The next three facts pull the plug on Myth 4, Myth 5 and Myth 6. They demolish the thesis that Info-Gap decision theory is particularly suitable for decision-making under severe uncertainty.
Fact 4: Info-Gap's robustness analysis does not tackle the severity of the uncertainty under consideration: it simply takes no notice of it.
Fact 5: Info-Gap's robustness analysis does not tackle the severity posed by unbounded uncertainty spaces. It simply takes no notice of it.
Fact 6: Info-Gap's robustness analysis is utterly unsuitable for the treatment of true Knightian uncertainty.
And to make vivid this point, let us now examine what Info-Gap's much vaunted capability to "handle" unbounded regions of uncertainty actually come down to. As noted above, Info-Gap's robustness analysis is conducted entirely in the neighborhood of a point estimate of the parameter of interest, to the exclusion of the rest of the region of uncertainty. This implies of course that Info-Gap's robustness analysis is, as a matter of principle, oblivious to what happens outside the estimate's immediate neighborhood on which the analysis is conducted. It is therefore totally immaterial how large the region of uncertainty under consideration is. Better yet, it is totally immaterial whether the region of uncertainty under consideration is unbounded or not. And so, for all one cares, the region of uncertainty may just as well be bounded, as the region of uncertainty being unbounded in effect comes to naught. The overall conclusion is then that Info-Gap's analysis does not (much less can it) tackle the severity of the uncertainty, it simply takes no notice of it.
The picture is this:
| No Man's Land | û | No Man's Land |
 |
| -∞ <-------------- | Complete region of uncertainty under consideration | --------------> ∞ |
where û denotes the estimate of the parameter of interest, the black area represents the complete region of uncertainty under consideration, the red area around û represents the region of uncertainty that actually affects the results generated by Info-Gap's robustness analysis, and the vast No Man's Land represents that part of the complete region of uncertainty that has no impact whatsoever on the results generated by Info-Gap's robustness model.
Yet, the claims made in this literature is that Info-Gap decision theory provides reliable means for achieving essential goals (color is mine):
If "rationality" means choosing an action which maximizes the best estimate of the outcome, as is assumed in much economic theory, then info-gap robust-satisficing is not rational. However, in a competitive environment, survival dictates rationality. In section 11.4 we will show that, for a wide range of situations, the robust-satisfier is more likely to survive than the direct optimizer. If "rationality" means selecting reliable means for achieving essential goals, then info-gap robust-satisficing is extremely rational.
Ben-Haim (2006, pp. 100-101)
This is yet another reason for Info-Gap decision theory qualifying for the title Voodoo Decision Theory. The picture speaks for itself.
Turning now to the recent attempts to assign "likelihood" to the results generated by Info-Gap's robustness model.
Much as the motivation for statements such as " ... info-gap theory seeks decisions that are most likely to achieve a minimally acceptable (satisfactory) outcome in the face of uncertainty ..." is clear, it is even clearer that these (recent) statements are without any foundation. Worse, such statements exhibit a profound misconception as to how the uncertainty is quantified, modeled and managed by Info-Gap decision theory. For, it cannot be emphasized enough that in his books and early papers Ben-Haim goes out of his way to point out that Info-Gap decision theory is non-probabilistic and likelihood-free. Furthermore, that Info-Gap's generic model of uncertainty lacks any mechanism for imputing likelihood to the analysis. Thus,
Fact 7: Info-Gap's uncertainty model, hence its robustness model, hence the results generated by these models, are non-probabilistic and likelihood-free.
As indicated above, the attempts to impute likelihood to Info-Gap's uncertainty model are designed to deal with the criticism that, focusing as it does on a (poor) point estimate, Info-Gap's robustness analysis, under conditions of severe uncertainty, is manifestly absurd.
But the point to note here is that to impute a proper likelihood structure to Info-Gap's uncertainty model will require a complete revamping of Info-Gap's resume which will remove its raison d'étre. Specifically, to be able to justify Info-Gap's local definition of robustness, it will take a great deal more than a simple recitation of statements such as the one quoted above. Indeed, it will require the introduction of an assumption that will assert, in so many word, that the true value of the parameter of interest is in the neighborhood of the given estimate. But in this case it will be ludicrous to designate Info-Gap decision theory a theory for decision-making in the face of severe uncertainty.
In any event, this solution will prove costly as it will require a global recall of numerous publications.
As for Myth 8.
This myth is rather puzzling, because, as attested by the Info-Gap literature, it is common knowledge among Info-Gap scholars, including Ben-Haim, that in many cases Info-Gap decision theory and "direct optimization" generate the same decisions. In other words, as documented in the Info-Gap literature, often decisions selected by "direct optimization" are as robust as decisions generated by Info-Gap's robustness model. In short,
Fact 8: The claim that solutions obtained by "direct optimization" of reward always have zero robustness to uncertainty is groundless.In fact, in cases where the robustness of the solutions generated by Info-Gap decision theory is greater than zero, the robustness of solutions generated by the so-called "direct optimization" is also typically greater than zero.
And there are definitely many cases where the solutions obtained by "direct optimization" are as robust as the solutions obtained by Info-Gap decision theory.
Regarding Myth 9, perhaps one of the most objectionable on this list:
Fact 9: Robust decision-making has been, for the last forty years, the central topic of study of the thriving area of Robust Optimization. Info-Gap scholars would therefore do well to consult this literature as they stand to gain a great deal from it.
So what are we to conclude from this list of Myths and Facts?
Before I can answer this question I need to address a question that is perhaps more puzzling than all the others namely,
The Ultimate Question?
I have often been asked the following intriguing question:
How is it that Info-Gap scholars do not see that Info-Gap decision theory does not, indeed cannot, address the severity of the uncertainty under consideration? Isn't it crystal clear that Info-Gap's robustness analysis totally ignores the severity of the uncertainty rather than confront it and deal with it?
My usual reply is the following:
I do not have a definite answer to this intriguing question. I have some explanations and I shall be happy to discuss them with you over a cup of coffee (skinny latte, no sugar, please!).
And to be sure, over the past three years I have had quite a few Info-Gap inspired skinny lattes ...
All I can say here is this.
It is important to keep in mind the background against which this intriguing question arises. This question is, as a rule, raised by persons who have been able to benefit from my presentations on Info-Gap whose main thrust is to clear away the heavy fog, spin and rhetoric so as to bring Info-Gap's true nature into full view. So, now that I have succeeded in accomplishing this task and have managed to make the truth about Info-gap apparent to all, it is easy to sit back and argue that the flaws in Info-Gap decision theory are so obvious that it is unclear what the fuss is all about. Isn't it as clear that Info-Gap decision theory does not address the challenges posed by severe uncertainty? Indeed, isn't it clear that Info-Gap decision theory ignores the severity of the uncertainty altogether?!
Although this reaction should no doubt be construed as a compliment to my ongoing effort to disambiguate readers of the Info-Gap literature regarding the myths that are being advanced in this literature about Info-Gap decision theory, the reaction on the part of Info-gap adherents is, as might be expected, quite different.
For obvious reasons, Info-Gap scholars/adherents show a great reluctance to part with it, so that rather than admit to the fact that this theory is riddled with grave errors, mistakes, and misconceptions, they prefer to further compound the errors by intensifying the fog, spin and rhetoric. In a word, the preference in Info-Gap circles is to create ... new myths. As noted above, this is beginning to happen, hence this discussion.
It is particularly important therefore that I take up in this discussion what seems to be a newly emerging myth, that at present is being given the final touches by info-Gap scholars. This myth, which to the best of my knowledge is so far only being stated "behind the scenes", has to do with the concept of severe uncertainty, its meaning, its implications, and so on.
I am therefore going to address it in the context of a discussion on what I regard as Info-Gap's defining characteristic — its being a Voodoo decision theory. This will enable me to explicate (yet again) the clear connection between the mistreatment that the whole notion of "severe uncertainty" is subjected to in the Info-Gap literature, and the fact that it is precisely this mistreatment that renders Info-gap a Voodoo decision theory par excellence.
Voodoo Decision-Making
Info-Gap scholars take great umbrage at my labeling Info-gap decision theory a voodoo decision theory. It is important, therefore, to begin this discussion by clarifying first of all the meaning of this concept.
Observe then that according to the Encarta online Encyclopedia,
Voodoo n
- A religion practiced throughout Caribbean countries, especially Haiti, that is a combination of Roman Catholic rituals and animistic beliefs of Dahomean enslaved laborers, involving magic communication with ancestors.
- Somebody who practices voodoo.
- A charm, spell, or fetish regarded by those who practice voodoo as having magical powers.
- A belief, theory, or method that lacks sufficient evidence or proof.
My usage of the term "voodoo decision theory" and "voodoo decision-making" is in line with the last meaning listed above. So roughly, in this discussion voodoo decision-making is a decision-making process that is guided and/or inspired by a voodoo decision theory, which is a theory that lacks sufficient evidence or proof and/or is based on utterly unrealistic and/or contradictory assumptions, spurious correlations, and so on.
This reading is also in line with the widely used terms Voodoo Economics, Voodoo Science and Voodoo Mathematics.
I should point out, though, that the term "Voodoo Decision Theory" is not my coinage (what a pity!):
The behavior of Kropotkin's cooperators is something like that of decision makers using the Jeffrey expected utility model in the Max and Moritz situation. Are ground squirrels and vampires using voodoo decision theory?Brian Skyrms (1996, p. 51)
Evolution of the Social Contract
Cambridge University Press.
There are basically three reasons for my labeling Info-Gap decision theory a voodoo decision theory par excellence:
- Info-Gap decision theory violates the universally accepted Garbage In — Garbage Out (GIGO) Axiom.
- Info-Gap decision theory violates the universally accepted Axiom that the results generated by a model can be only as good as the estimate on which the model is based.
- Info-Gap's treatment of severe uncertainty makes a mockery of the real challenge posed by decision-making under severe uncertainty.
All three points are interconnected. However, as an explication of the third will bring out the implications of the first two points, I shall focus primarily on the third.
Info-Gap's mistreatment of severe uncertainty
The great difficulty facing decision-making in the face of severe uncertainty is due to the fact that the true value of the parameter of interest is not only unknown, but is subject to severe uncertainty. The central point here is that our ignorance about the true value of this parameter is so profound that any estimate of the parameter's true value that we may come up with would be no more than a wild guess, a poor indication of the true value, and likely to be substantially wrong. Or, to put it in somewhat more technical terms, the parameter's true value can be anywhere in the given uncertainty space.
It is immediately clear then that given this state-of-affairs, the task facing decision-making under severe uncertainty, is to devise methods that will be able to conduct a global analysis of the uncertainty space. The essential issue here is preference reversal: the preference of decisions varies depending on the location of the true value of the parameter of interest in the uncertainty space. That is, a decision that performs well in one neighborhood of the uncertainty space may perform poorly in other neighborhoods of the uncertainty space and vice versa. Add to this the fact that given the severe uncertainty in the true value of the parameter of interest, we have not the slightest inkling in what neighborhood to conduct the analysis, it is clear that a global exploration of the uncertainty space is of the essence.
And to be sure, methods that are designed specifically to identify decisions that are robust to severe uncertainty aim precisely at that: a global analysis of the uncertainty space, or a suitable approximation thereof. It is simply taken for granted that anything short of a global analysis cannot be deemed reliable. Or, to put it differently, a local analysis is not even contemplated.
There is one exception to this rule: Info-Gap decision theory.
This theory is unique in that it inexplicably has no qualms to propose that robust decision-making in the face of severe uncertainty be based on a local analysis of the neighborhood of a wild guess. That this proposition demonstrates a profound lack of understanding and appreciation of what decision-making under severe uncertainty is all about is self-evident.
For, to reiterate, the challenge posed by the severity of the uncertainty is not that of finding how robust decisions are in a given point in the uncertainty space and its neighborhood. The challenge is to determine how robust decisions are relative to the entire uncertainty space.
The task therefore is to explore the uncertainty space under consideration by means of a suitable global technique so as to ensure that the uncertainty space under consideration is properly represented in the robustness analysis. As can be gathered then from the discussion thus far, it is precisely this incomprehensible proposition to base robust decision-making under severe uncertainty on an analysis that is confined entirely to an estimate (that amounts to a wild guess) and its immediate neighborhood that earns Info-Gap decision theory its title voodoo decision theory.
Of course, reading the Info-Gap literature one may not immediately discover this glaring flaw because this literature bursts with high rhetoric on the severe (true Knightian) uncertainty that Info-Gap (presumably) takes on, on how vast its uncertainty space can be, on its being a non-probabilistic and likelihood-free methodology, and so on ...
Discovering that all this talk comes to naught, indeed that it has nothing in common with the approach that Info-gap actually takes with regard to severe uncertainty, hence the methodology that it puts forward for its treatment, takes some doing ...
For instance, consider this piece of vintage Info-Gap rhetoric:
Rare events in probabilistic models are described by the tails of the distribution, while probability distributions are usually specified in terms of mean and mean-variation parameters. This makes probabilistic models risky design tools, since it is rare events, the catastrophic ones, which must underlie the reliable design.
Ben-Haim (2006, pp. 330-331)
In other words, Info-Gap decision theory claims that probabilistic models are risky and unreliable design tools because they give too little weight to catastrophic rare events, thereby distorting the robustness analysis.
Suppose that for the purposes of this discussion we accept this criticism of probabilistic models of uncertainty as well as the assertion that rare events must underlie reliable robust design.
The question then arising is: how does Info-Gap decision theory deal with rare events? How does Info-Gap decision theory incorporate rare events in its robustness/opportuneness analysis?
And the answer is very simple: Info-Gap decision-theory does not deal with rare events: it simply ignores them.
And the nice thing about this simple answer is that you need not be a mathematician to be able to put it across or to grasp it.
All you need to do is to consult again the picture presented above as it speaks volumes about the failings of Info-Gap's robustness/opportuneness models.
| No Man's Land | û | No Man's Land |
|
| -∞ <-------------- | Complete region of uncertainty under consideration | --------------> ∞ |
where, as you recall, û denotes the estimate of the parameter of interest, the black area represents the complete region of uncertainty under consideration, the red area around û represents the region of uncertainty that actually affects the results generated by Info-Gap's robustness analysis, and the vast No Man's Land represents that part of the complete region of uncertainty that has no impact whatsoever on the results generated by Info-Gap's robustness model.
The obvious question is of course: how can Info-Gap's robustness analysis possibly deal with rare events if it is confined to the neighborhood of a given estimate? Are we to assume that rare events would be located in the neighborhood of the estimate û? Or, are we perhaps to assume that the estimate itself represents a rare event?
The truth, of course, is that raising these questions is pointless because there are more basic questions about the estimate û that require a prior clarification, yet they remain unattended to and hence unanswered. For one thing, Info-Gap decision theory does not even bother to address the more basic question of how the value of the estimate û is determined. As you will recall, this estimate is the most crucial element of its uncertainty, robustness, and opportuneness models. Info-Gap simply assumes that the value of û is given — end of story.
And while we are at it, it is important to connect the remarks on the estimate û to my clarification of the reasons rendering Info-Gap a Voodoo decision theory.
It is important to realize that my labeling Info-Gap a Voodoo decision theory is not to condemn its use of an estimate that in effect boils down to a wild guess. After all, the general expectation is that the estimates of values subject to severe uncertainty should not be considered better than wild guesses because this is the nature of the beast and often there is little we can do to change this state-of-affairs.
Rather, the point of my labeling Info-Gap a Voodoo decision theory is to draw attention to a far more disturbing fact. The fact that, despite it being fully recognized that the severe uncertainty renders the estimate a wild guess, no qualms whatsoever are shown in prescribing a robustness analysis that fixes only on this wild guess and its immediate neighbourhood, to the exclusion of the rest of the uncertainty space. Namely, that no provisions (eg. scenario generation*) whatsoever are made to explore neighbourhoods of the uncertainty space which are distant from this wild guess.
In so doing, Info-Gap decision theory puts itself at loggerheads with the universally accepted Garbage In — Garbage Out (GIGO) Axiom and the well known dictum that the results of an analysis can be only as good as the estimate on which they are based. So, the point is that in the case of Info-Gap decision theory the results yielded by its analysis can be no better than ... wild guesses.
To suggest otherwise amounts to suggesting that Info-Gap decision theory possesses mysterious (magical) powers that enable it to translate an analysis around a wild guess into a reliable methodology for the treatment of severe uncertainty.
| Conventional Science | Info-Gap Decision Theory | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
The point of course is that Info-Gap's robustness analysis in fact makes an implicit claim to an extraordinary ability to generate reliable robust decisions out of a wild guess.
So, if we accept this claim, we may as well wind-up the discipline of Decision-Making Under Severe Uncertainty and declare it redundant. For, dealing with decision-making problems subject to severe uncertainty would now amount to child's play:
1-2-3 fool-proof recipe for decision-making under severe uncertainty
- Ignore the severity of the uncertainty.
- Focus instead on the neighborhood of your best estimate of the parameter of interest.
- Don't worry if you lack an estimate, a wild guess will do**.
Wouldn't this be great?!
*The following quotes are representative of the literature on scenario generation in robust decision-making in the face of uncertainty.
RAND undertook an analysis of future scenarios to help inform EERE’s planning process. Scenarios are used as descriptions of alternative futures, not as forecasts or predictions. They enable policymakers to systematically consider uncertainties inherent in energy planning and to select strategies that are robust. Robust strategies perform adequately over a range of conditions, in contrast to those that do very well under some conditions, but fail under others.
Richard Silberglitt and Anders Hove
A. Scenario
Analyses and Papers Prepared for the E-Vision 2000 Conference (p. 303)
E-Vision 2000: Key Issues That Will Shape Our Energy Future
October 11 - 13, 2000
The Ronald Reagan Building and International Trade Center, Washington, D.C.
The intention is not simply to move beyond a single, canonical view of the future, but to confront uncertainty as realistically as possible — conceiving the full "possibility space." To be sure, even the most heroic efforts are unlikely to be fully successful, as suggested by the difference between the dark and white areas in the middle of Figure 1.1. We aspire, however, to identify as much of the possibility space as possible. We may choose later to dismiss portions of it as insufficiently plausible to worry about or as irrelevant to most planning (e.g., a comet might destroy the earth). Further, we most certainly do not need high levels of detail for all of the points in the possibility space. Nonetheless, we need first to see the possibilities, at least in the large.
Enhancing Strategic Planning with Massive Scenario Generation: Theory and Experiments
Paul K. Davis, Steven C. Bankes, Michael Egner
Technical Report TR-392, RAND Corporation, 2007.
**Should you need it, the recipe for obtaining a wild guess is simplicity itself:
- Wet your index finger and put it in the air.
- Think of a number and double it.
See it on-line at wiki.answers.com/Q/What_is_best_estimate_and_how_do_i_calculate_it.
Local vs global robustness
Having tied the main reason for my labeling Info-gap a Voodoo decision theory to its "localizing" its robustness analysis to the immediate neighborhood of a (poor) point estimate, I now turn to a brief discussion on the fundamental difference between global and local robustness.
Of course, one can argue that a discussion on this topic is superfluous (if not slighting) as the terminology speaks for itself, so that anyone with a minimal mathematical education would instinctively have a pretty good idea of what these terms connote. Furthermore, that applying these two characterizations (of an analysis, results etc) in the context of a given problem situation would be a straightforward affair. Indeed, that the very need to draw a distinction between these two properties would be manifestly clear to all ... and so on and so forth....
And yet, the Info-Gap experience has shown that "it ain't necessarily so!" In fact, if there is one lesson to be learned from the Info-Gap experience it is that the fundamental difference between these two properties must indeed be pointed out. Specifically, that it is important to call attention to the fact that one ought to be clear on whether the robustness yielded by an analysis is local or global, because the implications of a local analysis can be worlds away from those of a global analysis.
And to bring out the difference between local and global robustness the most obvious thing to do is to draw an analogy with optimization theory. As is well known, there is a well-established, long-standing convention in optimization theory where a clear distinction is drawn between a local and a global optimum. The same type of distinction applies in the case of robustness. So, in parallel to optimization, where the ramifications of a local optimum are vastly different from those of a global optimum, the ramifications of local robustness are vastly different from those of global robustness.
And the consequences of this fact for decision theory in general and decision-making under severe uncertainty in particular are self-evident.
Informally, global robustness is the robustness that is obtained relative to the entire given uncertainty/variability space associated with the problem under consideration. That is, if the uncertainty/variability of a parameter of interest is expressed in terms of a set containing the "possible" values of this parameter, then robustness is global if it is measured over the prescribed set in question. And so, if we let U denote the given uncertainty/variability space of a parameter of interest, then a global robustness of a decision is a measure of how well/poorly a decision performs when the parameter of interest, u, is varied over U.
In contrast, a local robustness is robustness relative to a neighborhood of a given point, call it ü, in U.
Let U(ü,ε)⊆U denote a neighborhood of "size" ε≥0 around ü. Then a local robustness of a decision at ü∈U is the robustness of the decision over U(ü,ε) for some given ε.
Needless to say, if U(ü,ε) is a relatively small subset of U, then in general, there is no guarantee that a decision that is robust over U is also robust over U(ü,ε) and vice versa. Similarly, the fact that a decision is not robust over U(ü,ε) does not imply that it is not robust relative to U, and vice versa
Now, it is immediately clear that a methodology for decision-making under severe uncertainty, employing models of uncertainty that are non-probabilistic, and likelihood-free, must set itself the task of establishing the performance of decisions relative to the entire set U. In other words, the goal must be to determine the global robustness of decisions.
Because, even if it is argued (as Info-Gap proponents indeed do) that one can always come up with an estimate, the fact remains that this estimate is a wild guess of the true value of u and the uncertainty model remains and will remain non-probabilistic, and likelihood-free. After all, the mere fact that one claims to have an estimate is no basis for claiming that the true value of u is more/less likely to be in the immediate neighborhood of this wild guess. This means of course that it is imperative to adopt a global, rather than local, approach to robustness.
The "weight" that one gives this wild guess in the definition, hence the evaluation and analysis, of robustness can reflect, among other things, one's confidence — or lack of it — in the quality of this guess. But this does not alter by one iota the basic fact that under conditions of severe uncertainty the estimate is a wild guess and, a wild guess is... a guess, is a wild guess! Therefore, under severe uncertainty robustness must be sought with respect to U.
The whole trouble with the Info-Gap literature is that its rhetoric is misleading. Info-Gap publications announce to the world that the methodology that they are going to outline provides a reliable basis for robust decision-making etc. Clearly the impression given by this rhetoric is that the robustness promised is global robustness. But the methodology actually outlined in these publications of course has got nothing to do with these promises, as it is the usual Info-Gap's local approach to robustness.
For the benefit of Info-Gap scholars, I provide an illustrative example of how a local approach to robustness should be introduced, explained and discussed.
Example
Consider the paper entitled Local robustness analysis: Theory and application by Brock, Steven N. Durlauf (Journal of Economic Dynamics & Control 29, 2005, 2067–2092).
The first thing to note is the qualifier "Local" in the title of the article. You cannot miss it even if you tried hard.
Next, let us have a quick look at the abstract. The opening sentence is as follows:
Abstract
This paper develops a general framework for conducting local robustness analysis. By local robustness, we refer to the calculation of control solutions that are optimal against the least favorable model among models close to an initial baseline.
Let us now read the opening section of the discussion on the proposed theory:
2. Basic theory
In this section, we describe a general framework for local robustness analysis. By local robustness analysis, we refer to the idea that a policymaker may not know the 'true' model for some outcome of interest, but may have sufficient information to identify a space of potential models that is local to an initial baseline model. This may be regarded as a conservative approach to introducing model uncertainty into policy analysis, in that we start with a standard problem (identification of an optimal policy given a particular economic model) and extend the analysis to a local space of models, one that is defined by proximity to this initial baseline. The local model uncertainty assumption, in our judgment, is naturally associated with minimax approaches to policy evaluation. When a model space includes nonlocal alternatives, we would argue that one needs to account for posterior model probabilities in order to avoid implausible models from determining policy choice.
Finally, let us read the opening part of the summary section:
5. Summary and conclusions
This paper has provided an initial outline of a theory of local robustness. We have provided some general conditions that describe how, in the presence of local model uncertainty, robustness policy rules may be analyzed using basic calculus tools. This analysis provides explicit characterizations of the robust analog to an optimal control solution and the implied least favorable model for which the robust solution is an optimal control.
In short, the authors make it crystal clear that the robustness under consideration is local in nature.
The local "neighborhood" around the given baseline that Brock and Durlauf (2005) use is the conventional Euclidean "ball" of "radius" ε around the baseline point ã, namely
B(ã,ε):= {a∈ℜn: ||a - ã|| ≤ ε}
where ℜ denotes the real line and ||a - ã|| denotes the Euclidean distance between two points a and ã in ℜn.
Note that in the context of Brock and Durlauf's (2005) model, the size of the "ball" is specified a priori. In the framework of Info-Gap decision theory, the size of the neighborhood is not fixed in advance, rather it is yielded a posteriori by the robustness analysis. That is, the neighborhood considered by Info-Gap's robustness model is the largest "ball" around the estimate û over which the given performance requirement is satisfied for each point in the "ball".
All one can say is that had Info-Gap scholars been as clear as Brock and Durlauf (2005) are on the local nature of the robustness obtained, they would not have proclaimed to the world that Info-Gap decision theory seeks decisions that are robust against severe uncertainty.
Also, one need hardly point out the widespread use of the qualifiers "local" and "global", in countless contexts, to distinguish between the scope, range, domain etc. of a phenomenon, an activity, a process, and so on. Thus: local vs global economy; local vs global climate change, local vs global impact, local vs global analysis, local vs global search; local vs global availability, local vs global issues, local cs global groups, local vs global perspective, local vs global knowledge, local vs global networks, local vs global attributes, local vs global benefit, local vs global properties, local vs global solutions, local vs global needs, local vs global coverage, and the list goes on ...
The fact that this vital distinction is totally absent from the Info-Gap analysis, furthermore that its absence is (nearly) impossible to detect due to the heavy fog spin and rhetoric, leads uncritical readers of this literature to buy into the claims of the Info-Gap literature that Info-Gap decision theory provides a reliable methodology for robust decision-making under severe uncertainty. The disservice that this rhetoric in fact does to readers of this literature requires no further elaboration.
Remark
It is important to take note that in the book "Robust Reliability in Mechanical Sciences" Ben-Haim (1996) makes it crystal clear that the proposed theory — later called Information Gap — is designed to deal, first and foremost, with small deviations from a given nominal value. For instance, (color is mine):
1.5 Summary
Table 1.1 recapitulates the examples of robust reliability discussed in the previous four sections. In this book, reliability is assessed in terms of robustness to uncertain variation. The system or model is reliable if failure is avoided even when large deviations from nominal conditions occur. On the other hand, a system is not reliable if small fluctuations can lead to unacceptable performance. Robustness and fragility describe opposite extremes of reliability..
Ben-Haim (1996, p. 6)
Robust Reliability in the Mechanical Sciencesand
Chapter 3
Robust Reliability of Static Systems
3.1 Introduction
Mechanical systems are hardly ever designed to fail. Failure occurs because the system differs from its nominal design, or because the operational environment changes, or the system is altered in some way, or unanticipated or extraordinary loads are applied. The robust reliability of a system is a measure of its resistance to these uncertainties. The system is reliable if it can tolerate a large amount of uncertainty without failing. On the other hand, a system is not reliable if it is fragile with respect to uncertainty; it is unreliable if failure becomes a possibility as a result of small deviations from nominal circumstances.
Ben-Haim (1996, p. 31)
Robust Reliability in the Mechanical SciencesNot only is the term Knighitan uncertainty not mentioned even once in this book, the general impression is that the uncertainty under consideration is mild. What is more, the parameter û does not play the role of an "estimate" of the true value of u. Rather, it is viewed first and foremost as a "nominal" value, or the "center" of the nested regions of uncertainty.
The question therefore is: how can a theory that was originally designed to tackle "small deviations from nominal circumstances" possibly be able to handle severe, or "true" Knightian uncertainty, or unbounded regions of uncertainty?
The answer is: it can't.
Yet, five years later, essentially the same theory — now called Information-Gap (Ben-Haim 2001) — is presented to the world as one that is capable of handling severe, "true" Knightian uncertainty and even unbounded regions of uncertainty!
In summary then, in the case of Info-Gap decision theory, the title Voodoo decision theory reflects a combination of all these facts: the absurd formula prescribing a local analysis for the treatment of severe uncertainty; the proclamation to the world that this absurd formula actually provides a reliable basis for robust decision-making under severe uncertainty; the hollow spin and rhetoric that obscure the truth about Info-Gap decision theory from unwary readers.
Conclusions
It seems to me that the following conclusions are inevitable:
- Considering that Info-Gap's robustness to severe uncertainty is determined on grounds of a local robustness analysis of the neighborhood of a wild guess of the true value of the parameter of interest, Info-Gap decision theory cannot escape being dubbed a Voodoo decision theory par excellence.
- Given that Info-Gap's robustness and opportuneness models are simple instances of well-known, well-established, models of classical decision theory, Info-Gap decision theory cannot escape being dubbed a reinvention of the wheel. And given the first conclusion, a square wheel at that.
- Info-Gap's flaws cannot be rectified because they stem from fundamental conceptual errors. These errors are therefore at the core of this theory so that correcting them will require a total revamping of the theory. And in any case, this will lead us back to classical decision theory and robust optimization.
As I suggested some time ago, scholars/analysts who propose Info-Gap as a methodology for robust decision-making under severe uncertainty should attach a caveat to their reports/articles stating the following, or something similar:
|
In our discussion we make much of the severity of the uncertainty under consideration, the poor quality of the estimate we use, and the vastness of the uncertainty space. Note, however, that the robustness/opportuneness analysis that we propose here is local: it applies only to the estimate in question and its immediate neighborhood. It therefore, does not/cannot address the severity of the uncertainty under consideration, the poor quality of the estimate we use, the vastness of the uncertainty space, rare events, black swans, and so on. The results of our local analysis therefore apply only to the locale of this estimate and must therefore not be used as a basis for conclusions about the performance of decisions relative to the given uncertainty space. If you are interested in methodologies that do address the severity of the uncertainty under consideration, the estimate being poor, and the vastness of the uncertainty space, consult the literature on Robust Optimization and Decision-Making in the Face of Severe Uncertainty. |
They should also attach the following, or a similar, picture:
| No Man's Land | û | No Man's Land |
|
| <-------------- | Complete region of uncertainty under consideration | --------------> |
and they should explain the implication of the No Man's Land syndrome.
Appendix: A theorem a day keeps the doctor away!
Appendix: A theorem a day keeps the doctor away!
Recall that Info-Gap's generic robustness and opportuneness models are formulated as follows:
| Robustness of decision d: | α(d,û):= max {α≥0: r(d,u) ≥ r*, for all u∈U(α,û)} , d∈D | |
| Opportuneness of decision d: | β(d,û):= min {α≥0: r(d,u) ≥ r*, for some u∈U(α,û)} , d∈D | |
And this is how the famous Maximin and Minimin models of classical decision theory are formulated:
| ||||||||||||||||||||||||||||
The implications of the following theorems do not require much discussion:
Maximin Theorem Info-Gap's robustness model is a simple instance of Wald's Maximin model. Specifically,
where a◊ b = 1 iff a ≥ b and a◊ b = 0 iff a < b.
α(d,û):= max {α≥0: r(d,u) ≥ r*, for all u∈U(α,û)} ≡ max min α·(r(d,u)◊r*) α≥0 u∈U(α,û)
Minimin Theorem Info-Gap's opportuneness model is a simple instance of the generic Minimin model. Specifically,
where a◊ b = 1 iff a ≥ b and a◊ b = ∞ iff a < b.
β(d,û):= min {α≥0: r(d,u) ≥ r*, for some u∈U(α,û)} ≡ min min α·(r(d,u)◊r*) α≥0 u∈U(α,û)
The Maximin Theorem asserts that Info-Gap's robustness model is a simple instance (specific case) of Wald's Maximin model. The Minimin Theorem asserts that Info-Gap's opportuneness model is a simple instance (specific case) of the famous Minimin model.
For the benefit of newcomers to the field, the Maximin model is a central pillar of classical decision theory and robust decision-making.
More details on these theorems, including their formal proofs, can be found in Wikipedia and in FAQs about Info-Gap decision theory.
The theorem below enunciates Info-Gap's robustness model's failure to tackle the severity of the uncertainty under consideration. It makes explicit that the results generated by this model are invariant with the size of the complete region of uncertainty under consideration.
This is illustrated in the following picture:
where α' > α(d,û) for all decisions d∈D and U' denotes the complete region of uncertainty under consideration.
Because Info-Gap's regions of uncertainty are nested, it follows that there is a u in U(α',û), call it u*, such that r(d,u*) < r* for all d∈D and u* is in U(α,û) for all α > α'. This means that an intensification in the severity of the uncertainty — the size of the region of uncertainty growing from U' to U'' — has no affect whatsoever on the results generated by Info-Gap's robustness analysis. In other words, Info-Gap's robustness analysis will generate the same results for all complete regions of uncertainty that contain U(α',û). In fact, we can even let the complete region of uncertainty be unbounded: and this will have no affect whatsoever on the results generated by Info-Gap's robustness analysis.
Invariance Theorem
The results generated by Info-Gap's robustness analysis are invariant with the size of the complete region of uncertainty U. That is, the analysis generates the same results regardless of how large/small this region is, as long it contains a region of uncertainty U(α',û) such that α' > max {α(d,û): d∈D}.
The picture is this:
| No Man's Land | û | No Man's Land |
| ||
| U(α',û) |
| -∞ <-------------- | U Complete region of uncertainty under consideration U | --------------> ∞ |
where the black area represents the complete region of uncertainty U and the red area represents the set U(α',û). Note that α' can be any number that is greater than max {α(d,û): d∈D}.
The picture speaks for itself.
Indeed, it is difficult to imagine a more imprudent approach to robustness against severe uncertainty than the blurring of the lines between robustness to severe uncertainty and robustness to a very mild uncertainty.
More details on this theorem, including its formal proof, can be found in Wikipedia and in FAQs about Info-Gap decision theory.
