| Reference: | Hall, J., R. J. Lempert, K. Keller, A. Hackbarth, C. Mijere, and D. J. McInerney |
| Abstract |
This study compares two approaches for robustness analysis of decision problems: the Info-‐gap method originally developed by Ben-Haim and the RDM (robust decision making) approach originally developed by Lempert, Popper, and Bankes. This study uses each approach to evaluate alternative paths for climate-‐altering greenhouse gas emissions given the potential for non-‐linear threshold responses in the climate system, significant uncertainty about such a threshold response and a variety of other key parameters, as well as the ability to learn about any threshold responses over time. Info-‐gap and RDM share many similarities. Both represent uncertainty with sets of multiple plausible representations of the future and seek to identify robust strategies whose performance is as far as possible insensitive to uncertainties. Yet they also exhibit important differences, as they arrange their analyses in different orders, treat losses and gains in different ways, and take different approaches to imprecise probabilistic information. This study finds that the two approaches reach similar but not identical policy recommendations and that their differing attributes raise important questions about their appropriate roles in decision support applications. The comparison not only improves understanding of these specific methods, it also suggests some broader insights into robustness approaches and a framework for comparing them. |
| Scores | TUIGF:100% SNHNSNDN:500% GIGO:126% |
Introduction
This article is of particular interest to me because its first author is: Hall, J.
Let me explain.
For the benefit of readers who are not familiar with this collection of reviews, I should begin by pointing out the following. In the past two years I reviewed two articles (see Review 6 and Review 15) by Hall in which he advocated info-gap decision theory as a methodology for decision under severe uncertainty. In my reviews of these articles, I pointed out that apparently, in response to my sustained criticism of info-gap decision theory, Hall had made a brave attempt to introduce a fundamental fix in this theory, so as to correct a deep rooted flaw in it, in fact a flaw that renders info-gap decision theory a voodoo decision theory par excellence. Of course, the main objective of my reviews was to show that Hall's attempts at fixing info-gap decision theory's fundamental ills were themselves very problematic, so that rather than providing a remedy, they exacerbated the problem even more!
So, having now come across this new article, I was curious to find out whether my response to Hall's previous attempts to revamp info-gap decision theory would be reflected in the new article.
But, not only did I not find so much as a remote echo of my critique of Hall's failed attempts at "amending" info-gap decision theory. I did not find so much as an echo of his previous implied admission of the necessity to introduce some such correction in info-gap decision theory, and what is more of the necessity to state these corrections explicitly.
This seems to suggest that in this article Hall has given up on his attempts to fix the theory, which means of course that this new article calls for harsher criticism than the criticism that was directed at the previous articles (see Review 6 and Review 15).I want to assure all those readers who might find this story a bit complicated that it is in fact quite simple to follow. Simply read on!
However, before I can proceed to unravel the relevant details that will make sense of this story, it is important that I call attention to a no less important fact about this article. This has to do with what seems to be an attempt on the part of the authors to "repackage" info-gap decision theory, again I suspect in an effort to circumvent the censure that info-gap decision theory is so obviously vulnerable to! And by this I mean that, by using a clearly modified rhetoric, the authors are now seeking to dress info-gap decision theory in new clothes.
So, contrary to what, to date, has been trumpeted in the entire info-gap literature (some of Hall's previous articles included) as info-gap decision theory's great forte, namely its capabilities as a method yielding robust decisions under severe uncertainty, in this article, is (deliberately?) being played down.And so, in this article info-gap decision theory is no longer claimed to provide the decision-maker with a mechanism for ranking decisions! No, not at all!
According to the rhetoric in this article, all that info-gap decision theory furnishes the decision maker is some sort of general approach which gives him some (indeterminate?) counsel about the complicated business of decision under uncertainty. In a word, going by the rhetoric in this article, one would be hard pressed to recognize the info-gap decision theory described here as an "approach", as the methodology that in the entire Info-gap literature to date (some of Hall's previous articles included) is being hailed as a reliable methodology for identifying robust decisions, to be precise identifying those decisions that are (the most) robust to uncertainty.
The trouble is, however, that I am unable to give you a complete assessment of the so-called "Info-gap approach" outlined in this paper, because the authors did not bother to specify the details of the performance function. Consequently, it is impossible to validate the results reported on in this paper.
The missing Main Assumption
And now to the details of the story that is at the center of this review. I propose to unravel it in three stages. In the first I remind the reader of the fundamental flaws that render info-gap decision theory a voodoo decision theory. In the second, I explain how in previous articles, Hall had sought to find a way around some of these flaws. Following that I examine how the new article deals with this issue.
The fundamental flaw
For the benefit of readers who are not familiar with the "Info-gap decision theory story" I should point out that the reason that I have branded this theory: a voodoo decision theory par excellence is essentially due to its prescription for robust decision-making under severe uncertainty. In fact, you need not even be a risk analyst to immediately see that like any other "voodoo theory" this theory is of the "too good to be true" ilk. Which means of course that it is a theory whose groundless propositions can be easily exposed for what they are by means of simple examples and counter-examples.
Of particular interest to us in this discussion is info-gap decision theory's prescription for the treatment of severe uncertainty. This prescription basically instructs the following:
Ignore the severity of the uncertainty.
Of course, not in so many words, still this is what this prescription comes down to.
And to appreciate my claim that you need not be a certified risk-analyst to immediately see that it is this prescription that renders this theory a voodoo decision theory par excellence, simply keep in mind that this prescription is given by a theory claiming to provide the means for tackling the severest uncertainty imaginable.
How SEVERE?
To begin with, the uncertainty is claimed to be non-probabilistic and likelihood-free, meaning that it cannot be quantified by means of "conventional" models of uncertainty or, by means of fuzziness. The quantification of the uncertainty is therefore austere in the extreme. Indeed, it comprises two elements:
- An uncertainty space, U.
- A point estimate û of the true value of the parameter of interest, call it u.
The immediate implication of the uncertainty being likelihood-free, is that there are no grounds to assume that the true value of u is more/less likely to be in the neighborhood of the estimate than in the neighborhood of any other value of u in U. And for similar reasons, the estimate must be assumed to be poor to the effect that it may well be substantially wrong indeed, no more than a wild guess of the true value of u.
And to illustrate, suppose that the uncertainty space U is this page. This means that, as the uncertainty is likelihood-free, we have not a clue where the true value of u is on this page. All we know is that it is somewhere on this page.
And even if we were to assume, for argument's sake, that the point estimate of the true value of u is exactly in the middle of this page. Given that the uncertainty is likelihood-free, this additional stipulation would not alter even by one iota the basic fact that we have no clue where the true value of u is in the page.
The implication is then that there are no grounds whatsoever, to assume that the true value of u is more/less likely to be in the neighborhood of the estimate than in the neighborhood of any other value of u in U.
And all this leads to the inevitable conclusion that to determine the robustness of a decision to uncertainty where the uncertainty is described in these terms, it is imperative to evaluate the performance of the decision over the uncertainty space U.
Yet, this is not what info-gap decision theory prescribes doing!
To the contrary, info-gap decision theory prescribes conducting a robustness analysis only in the neighborhood of this (highly questionable, doubtful etc.) estimate and nowhere else. And so, info-gap decision theory's prescription for measuring robustness against severe uncertainty is based on the following question:
how much can this (poor) estimate be perturbed (in all directions) without causing a violation of the performance requirement?
Thus, if the answer is for instance 12cm, then any perturbation of size 12cm or less (in any direction) will not violate the performance requirement and will therefore be deemed "acceptable". Whereas, a slightly larger perturbation (in some direction) will be deemed unacceptable as it will cause a violation of the performance requirement.
This definition and means of measuring robustness is known universally as Radius of Stability.
For obvious reasons, the Radius of Stability is treated universally (eg. in the robust control theory literature, economics literature, etc.) as a model of local robustness. This means of course that the accepted convention is that as a measure of local robustness it cannot be counted on to provide the global robustness of a system a decision, or whatever. In other word, it cannot be counted on to indicate how well, or how poorly, a decision performs over the entire uncertainty space U. Indeed, it is elementary to devise examples demonstrating that a decision that is locally robust in the neighborhood of the estimate is very fragile globally over U, and vice versa.
In short, all this goes to show that, as a radius of stability model, info-gap's robustness model is the wrong model for the treatment of a severe uncertainty of the type that info-gap decision theory is claimed to address.
It is extremely important to note in this regard that info-gap decision theory in fact has the dubious distinction of being the only decision theory in the trade to propose that the robustness of a decision against severe uncertainty be measured by a Radius of stability model.
So, to repeat, you need not be a risk analyst to see that a theory purporting to offer a reliable methodology for robust decision to uncertainty proposing this prescription for this purpose, is fundamentally flawed.
Hall definitely saw it and this is the reason for the fixes that he proposed in his recent articles (see Review 6 and Review 15).
The Fix
In response to my criticism of info-gap decision theory, Hall and Harvey (2009) have hit on an easy quick fix.
Although their paper is riddled with TUIGF, my concern here is only with the following remarkable statement made immediately after the description of info-gap's regions of uncertainty, the horizon of uncertainty α and the estimate û of the parameter of interest:
An assumption remains that values of u become increasingly unlikely as they diverge from û.
Hall and Harvey (2009, p. 12 , emphasis added)
In other words, this assumption indicates that the estimate û is the most likely value of the parameter u and that the likelihood of u decreases as u deviates from û. As I pointed out in my review, it was clear that Hall and Harvey's rationale for introducing this assumption was to justify Info-Gap's fixing on the estimate as the focus of the robustness analysis.
What I particularly wanted readers to take note of was the phrasing of this assumption. I therefore, called readers attention to the word remains by raising the following (rhetorical) question:
"What exactly are we to make of remain? Does it mean that in the context of Info-Gap, which boasts of being a non-probabilistic likelihood-free theory, this assumption was all along the case and it thus remains? If so how does it square with the "official" Info-Gap decision theory? Or, is this a "new" assumption? One that the authors decided to append to the "official" theory? If it is a newly added assumption then surely this must be made clear. Whatever the case, the authors must explain how this assumption tallies with the claim that the uncertainty under consideration is severe? "
More on this in Review 6
A second attempt was made by Hine and Hall (2010), where a different assumption was introduced to get rid of, or mitigate, the above mentioned absurd in info-gap decision theory's prescription for the management of severe uncertainty. So, on page 2-3 we read the following:
The main assumption is that u, albeit uncertain, will to some extent be clustered around some central estimate û in the way described by U(α,û), though the size of the cluster (the horizon of uncertainty α) is unknown. In other words, there is no known or meaningfully bounded worst case. Specification of the info gap uncertainty model U(α,û) may be based upon current observations or best future projections.
Hine and Hall (2010, pp.2-3, emphasis added)
Although this phrasing is an improvement on Hall and Harvey's (2009) quick-fix, just like its predecessor, this assumption amounts to a major indictment of info-gap decision theory. And, strictly speaking, it is misleading.
Let me explain.
- What the authors refer to here as the "main assumption", thereby creating the impression that this assumption is implicit in info-gap decision theory, is not in fact an info-gap decision theory assumption. It is not an integral part of info-gap decision theory -- as the theory is described in Ben-Haim (1996, 2001, 2006, 2010). There is no trace of it in any of these publications.
- This "main" assumption is an innovation introduced ad hoc by the authors in order to justify info-gap's absurd proposition to focus the analysis on a wild guess.
- Interestingly, there is no trace of this "main" assumption in Hall's earlier publications on info-gap decision theory.
A detailed critique of the errors, misconceptions, misinformation etc. demonstrated in the wording of this assumption is available in Review 15.
The question is then: where is this "main Assumption" in Hall et al. (2011)?
Good question!
A repackaging of info-gap decision theory
But the really intriguing question that this article by Hall et al. (2011) raises is why the "sudden need" to repackage info-gap decision theory? In other words, the question that the authors must answer is this: how does their claim that info-gap decision theory does not provide a strict ranking of decisions square with the standard depictions of info-gap decision theory in all info-gap publications to date?
To enable readers who are not familiar with this literature, to see for themselves, I juxtapose Hall et al's. (2011) statement with a sample of statements --- illustrating the standard depiction of this theory --- by other info-gap scholars, including the Father of the theory (Ben-Haim) and ... Jim Hall himself!
| Hall et al. 2011 | Other info-gap scholars |
|
Neither Info-gap nor RDM provide a strict ranking of alternative decisions. Rather, both provide decision support, summarizing tradeoffs for decision makers to help inform their judgments about the robustness of alternative decision options.
Hall et al. (2011, p. 2)
|
The best decision is then chosen as the one that is most robust to uncertainty, i.e. is guaranteed to give acceptable outcomes under the greatest degree of uncertainty.
Halpern, Regan, Possingham and McCarthy (2006, p. 3)
The best decision is the one that is most robust to uncertainty, by guaranteeing an acceptable outcome under the greatest degree of uncertainty.
McCarthy and Lindenmayer (2007, p. 554)
The robustness can be evaluated even though there is no known worst case. Furthermore, the robustness function generates preferences on the decisions, q: a decision which is more robust for achieving aspiration rc is preferred over a decision which is less robust. Robust-satisficing decision making maximizes the robustness and satisfices the reward at the value rc, without specifying a limit on the level of uncertainty: 
Davidovitch and Ben-Haim (2010, p. 268)
The robustness function generates a preference ordering on the available decisions: a more robust decision is preferred over a less robust decision. Satisficing means doing well enough, or obtaining an adequate outcome. A satisficing decision strategy seeks a decision whose outcome is good enough, though perhaps sub-optimal. A robust-satisficing decision strategy maximizes the robustness to uncertainty and satisfices the outcome.
Schwartz, Ben-Haim and Dasco (2011, p. 213)
We now have a general mathematical formulation of the problem at hand including a model, which incorporates uncertainties in the preliminary data, and a method to choose the best decision.
Sisso, Shema and Ben-Haim (2010, p. 1035)
As we have noted before, this means that ``bigger is better'' for the robustness function. Consequently, a decision maker will usually prefer decision option q over an alternative decision q' if the robustness of q is greater than the robustness of q' at the same value of critical reward rc. ... ... Let Q be the set of all available or feasible decision vectors q. A robust-satisficing decision is one which maximizes the robustness over the set Q of available q-vectors and satisfices the performance at at the critical level rc.
Ben-Haim (2006, p. 45)
Info-gap analysis allows the decision maker to identify solutions that perform satisfactorily well under the widest possible range of conditions. This is a departure from the conventional approach to decision making, that seeks to maximise performance, but under conditions of severe uncertainty, a guaranteed level of performance (up to some horizon of uncertainty) may well be more attractive than optimal performance that is vulnerable to the unexpected.
Hall and Ben-Haim (2009, p. 7)
In this way the robustness function generates preferences among available decisions. When choosing between two options, the robust-satisficing decision strategy selects the more robust option.
Ben-Haim (2010, p. 8)
|
Surely, this blatant distortion by Hall et al. (2011) of what info-gap claims to be cannot be an accident?!
The question is: what is its object?
Because to date, info-gap decision theory has been presented as a theory that ranks decisions according to their robustness: the larger the better. Hence, according to (what is known to the public as) info-gap theory, the best decision is that whose robustness is the largest (for the desired level of performance, rc).
Are we to conclude from this paper that this is no longer the case?
I suspect that this distortion may well be another attempt at a quick fix, aimed at getting around info-gap decision theory's fundamental ills. And this, by the way, is the norm in the info-gap literature: a repackaging of the rhetoric as a means of glossing over the fundamental ills of the theory!
Remarks
-
Ben-Haim, the Father of info-gap decision theory, is crystal clear about how his theory establishes preferences among decisions. In the 2006 edition of his book on the theory we read:
For given values of critical or windfall reward, rc or rw, each immunity function induces a preference ranking on the set of available decisions. More importantly, the immunity functions enable the decision maker to explore the desirability of different options q and different requirements rc and rw , and thus to alter earlier preferences.
Ben-Haim (2006, p. 45)And in the 2010 book we read:
Goals which are satisficed (sub-optimal but good enough) can be achieved by many alternative policies. Choose the most robust from among these alternatives.
Ben-Haim (2010, p. 10)In short, info-gap decision theory gives specific instructions on how to rank alternative decisions, policies, options, etc: choose the most robust!
Can there be any doubt about that?
So, the message is crystal clear, leaving no room for the imagination: contrary to the authors' claim, info-gap decision theory provides ranking of alternative decisions.
It should therefore be most interesting to hear from Ben-Haim himself about this attempt to revamp his theory.
So, we shall have to wait and see.
- The point raised by Hall et al. (2011) about the "tradeoff" issue has no bearing whatsoever on the fundamental fact that, as dictated by info-gap decision theory, for any given value of the critical reward rc, the best decision is a decision that maximizes the robustness function (for the specified value of rc). And it is on the basis of this determination that the "tradeoff" analysis is then conducted.
- The troubles besetting info-gap's robustness analysis have got nothing to do with this "tradeoff". The trouble is due to the fact that info-gap's robustness model is a a model of local robustness which is being misapplied for the management of severe uncertainty. More on this in the Official Mobile Debunker of info-gap decision theory.
The missing performance (reward) function
On Fri, 10 June 2011 16:42:20 +1000 (EST), I requested from one of the co-authors details on the performance function of the info-gap robustness model because this function is not specified for the numerical examples presented in the article. The request was acknowledged and was forwarded to another author.
I was informed on Wed, 13 July 2011 11:17:16 +0100, by the first author, that the reward function is not specified in the paper. The rewards are generated by a computer program (MLK DICE) that is reported on elsewhere.
I find this most surprising because the uncertain parameter under consideration (call it u) takes only 2662 values, namely the uncertainty space is discrete and contains 2662 distinct values of u. This means that for each of the four strategies under consideration, there are only 2662 rewards. The implication is then that, the reward function can be easily "specified" (online) by a relatively small spreadsheet to enable interested users to easily download and examine it.
But more than this, in view of the fact that the uncertainty space is discrete and manifestly small, determining robustness is trivial --- it can easily be done by enumeration. Indeed, given the size of this discrete uncertainty space, it is hard to comprehend why the authors conduct no more than a local robustness analysis! For their money, they could have easily performed a global analysis (over the entire uncertainty space) to come up with far more meaningful results!!!!
Regrettably, the authors do not provide such a spreadsheet so that it is impossible to check/reconstruct their results.
It is interesting to note the following reference, in the article under review, to the DICE-07 model (page 4):
For this study, we use a modified version of DICE07 (McInerney, Lempert, and Keller (2009), henceforth “MLK”), that adds the possibility of a large-scale and economically costly collapse of the North Atlantic Meridional Overturning Circulation (MOC) triggered if and when atmospheric CO2 levels exceed an uncertain threshold (Keller et al. 2004).where the bibliographic details of "MLK" are as follows:
McInerney, D., R. Lempert and K. Keller (2009). "What are robust strategies in the face of uncertain climate threshold responses?" Climatic Change, in revision, available at http://www.geosc.psu.edu/~kzk10/publications.html
I found a copy of this paper at http://www.geosc.psu.edu/~kzk10/McInerney_et_al_CC_09.pdf, but I could not find in it the details that are required to evaluate the reward function in question, in the article under review.
In short, I cannot see how the results reported on in the article under review, can possibly be reconstructed/checked/examined.
Summary and conclusions
The uncertainty space of the robustness model under consideration in this article consists of 2662 values of a parameter comprising 4 components. Since only four strategies are considered in this case, the robustness issue is trivial to the extent that it can easily be handled by enumeration.
This means that both methodologically and practically, it is unclear why a model of local robustness is used here to begin with.
Furthermore, as the authors do not specify the reward function, it is impossible to check the results generated by the proposed local robustness model so as to determine whether these results are consistent with the strategies’ global robustness over the uncertainty space.
Finally, there are no signs, in this article, of Hall's previous attempts to revamp info-gap decision theory (see Review 6 and Review 15).
