| Reference: |
Hall, J. and Harvey, H. Decision making under severe uncertainty for flood risk management: a case study of info-gap robustness analysis. Eighth International Conference on Hydroinformatics January 12-16, 2009, Concepcion, Chile. (PDF file) |
| Abstract | Flood risk analysis is subject to uncertainties, often severe, which have the potential to undermine engineering decisions. This is particularly true in strategic planning, which requires appraisal over long periods of time. Traditional economic appraisal techniques largely ignore this uncertainty, preferring to use a precise measure of performance, which affords the possibility of unambiguously ranking options in order of preference. In this paper we describe an experimental application of information-gap theory, or info-gap for short to a flood risk management decision. Info-gap is a quantified non-probabilistic theory of robustness. It provides a means of examining the sensitivity of a decision to uncertainty. Rather than simply presenting a range of possible values of performance, info-gap explores how this range grows as uncertainty increases. This allows considerably greater opportunity for insight into the behaviour of our model of option performance. The information generated may be of use in improving the model, refining the options, or justifying the selection of one option over the others in the absence of an unambiguous rank order. Secondly, we demonstrate the possibility of exploring the value of waiting until improved knowledge becomes available by constructing options that explicitly model this possibility. |
| Acknowledgement | The work described in this publication was supported by the European Community's Sixth Framework Programme through the grant to the budget of the Integrated Project FLOODsite, Contract GOCE-CT-2004-505420. This paper reflects the authors' views and not those of the European Community. Neither the European Community nor any member of the FLOODsite Consortium is liable for any use of the information in this paper. |
| Scores | TUIGF:90% SNHNSNDN:100% GIGO:50% |
I do not know for a fact that this is a refereed paper. Still, I consider it important for this project because, to the best of my knowledge, this is the first paper by Info-Gap proponents in which some sort of an attempt is made to deal with the fact that Info-Gap's robustness model engages in a ... voodoo analysis. I suspect that this attempt at rectifying the model is due to my criticism of Info-Gap — which, I am confident, is known to the authors.
So what exactly is it that I find so interesting in this paper?
Although this paper is afflicted with TUIGF, of particular interest is the following remarkable statement made immediately after the description of Info-Gap's regions of uncertainty, the horizon of uncertainty α and the estimate û of the parameter of interest (page 2, emphasis is mine):
An assumption remains that values of u become increasingly unlikely as they diverge from û.
In other words, this assumption indicates that the estimate û is the most likely value of the parameter u and that the likelihood of u decreases as u deviates from û. The inference clearly seem to be that this assumption is introduced to justify Info-Gap's fixation with the estimate as the focus of the robustness analysis.
Particularly noteworthy in the phrasing of this assumption is the word remains. What exactly are we to make of remain? Does it mean that in the context of Info-Gap, which boasts of being a non-probabilistic likelihood-free theory, this assumption was all along the case and it thus remains? If so how does it square with the "official" Info-Gap decision theory? Or, is this a "new" assumption? One that the authors decided to append to the "official" theory? If it is a newly added assumption then surely this must be made clear. Whatever the case, the authors must explain how this assumption tallies with the claim that the uncertainty under consideration is severe?
Three comments on this remarkable assumption:
- This assumption is not an integral part of info-Gap's generic model of uncertainty. The fact is that this assumption represents a radical departure from the "official" Info-Gap decision theory. Note that the founder of Info-Gap goes to great lengths to emphasize that this theory is not only probability-free, but also likelihood-free (emphasis is mine):
Furthermore, this assumption is not postulated in previous Info-Gap articles by Jim Hall, for instance the paper discussed in my Responsible Decision-Making page.In info-gap set models of uncertainty we concentrate on cluster-thinking rather than on recurrence or likelihood. Given a particular quantum of information, we ask: what is the cloud of possibilities consistent with this information? How does this cloud shrink, expand and shift as our information changes? What is the gap between what is known and what could be known. We have no recurrence information, and we can make no heuristic or lexical judgments of likelihood.
Ben-Haim (2006, p. 18)
Info-Gap Decision Theory: Decisions Under Severe uncertainty
Academic Press. - This assumption contradicts the authors' initial assumption that the uncertainty under consideration is severe. For, how can we possibly treat the uncertainty as "severe" and at the same time assume that values of u become increasingly unlikely as they diverge from û? Furthermore, how does this assumption sit with Ben-Haim's (2006, pp. 280-281) characterization of severe uncertainty (2006, pp. 280-281) where it is posited that under severe uncertainty the estimate û "... is a poor indication of the true value ..." and " ... is likely to be substantially wrong ...".
In fact, in his FAQs about Info-Gap Ben-Haim (2007) states in so many words that under severe uncertainty the estimate û can be a wild guess. Are we to understand then that the above assumption departs from Ben-haim's position about the nature of the estimate û to contend that it is not a wild guess? Of course this assumption is unable to invalidate the fact that under severe uncertainty the estimate is a wild guess. Still, let us strike compromise and assume that here the estimate is only a mildly wild guess.
- This assumption does not even come close to correcting the basic flaw in Info-Gap's uncertainty model. The reason is very simple: it does not even remotely give the slightest guarantee that the true value of the parameter u is very likely to be in the immediate neighborhood of the estimate û. But what is more, it is easy to construct examples where this assumption is satisfied, yet the true value of u is very unlikely to be in the neighborhood of the estimate û. The implication is then that this assumption comes to naught as it fails to meet the central difficulty besetting decision-making under severe uncertainty. The fact that the estimate û is a poor indication of the true value of u and is likely to be substantially wrong.
In any event, despite its failure, this attempt at dealing with the fundamental flaw in Info-Gap's uncertainty model attests to the fact that some Info-Gap scholars begin to realize that -- in the absence of such assumptions -- Info-Gap decision theory cannot escape being a manifestly voodoo decision theory.
Remark: Attempts of this kind are doomed to fail simply because it is untenable to justify a local robustness analysis in the neighborhood of the estimate û and at the same time assume that the estimate is a poor indication of the true value of u and is likely to be substantially wrong. Info-Gap scholars would thus be well advised to consult the literature on Robust Optimization to see how severe uncertainty is modeled and managed without a priori localizing the analysis around a poor estimate. I also suggest that they remind themselves of the GIGO Axiom, and of course, the second challenge cited on the list of top challenges for making robust decision (emphasis is mine):2. Managing uncertainty
Decisions depend on the best estimates of past performance, assessments of the current situation and visions into the future. Where the past performance may be known, the current is clouded by its immediacy and the future is a best guess. The robustness of any decision and the risk incurred in making that decision is only as good as the estimates on which it is based. Making estimation even more challenging, virtually all estimates that affect decisions are uncertain. Uncertainty can not be eliminated, but it can be managed.Top Ten Challenges for Making Robust Decisions
The Decision Expert Newsletter™ -- Volume 1; Issue 2But it is important to take note that one cannot postulate that the estimate û is so good that it is sufficient to conduct the robustness analysis in its neighborhood, and at the same time contend that the uncertainty in the true value of u is severe.
For the record, I should also point out that the article maintains the SNHNSNDN attitude towards the Maximin/Info-Gap connection. It continues to propagate the myth that Info-Gap decision theory is unique in its non-probabilistic, likelihood-free approach to the quantification and managing of uncertainty. Not only does this article fail to make the slightest mention of the famous Maximin paradigm, it takes what seems to be a rather evasive stance in this regard as born out by the assertion (page 2):
" ... All of these approaches rely upon some normalised measure being applied over the space of possibilities. Info-gap theory, but [sic] contrast does not employ normalised measure at all, so does not fit anywhere within the information-theoretic hierarchy of theories of uncertainty developed by Klir [9]. Rather, uncertainty is represented by a family of nested sets bounding the variation of system behaviour about some nominal value û. ..."
Why do the authors make do with the reference to Klir's article in the journal Reliability Engineering and Systems Safety? Why don't they refer to the theories developed by the founders of modern decision theory in the 1950s? In particular, why don't the authors refer to standard textbooks in decision theory, operations research, and robust optimization, where Wald's Maximin paradigm (circa 1940) is portrayed as the primary paradigm for decision-making under severe uncertainty? And why don't the authors refer to Sniedovich's (2007) proof that Info-Gap's robustness model is in fact a simple Maximin model?
How long can senior Info-Gap scholars keep their heads buried in the sand with regard to the Maximin/Info-Gap connection?
My view on this is rather cynical.
I believe that there is considerable pressure these days on academics to portray the methods that they use/develop as new and revolutionary. This is especially the case in applications for research grants. Indeed, what is the likelihood of your being awarded a grant if you state clearly in your grant application that your research will be based on an old mainstream theory that is described in undergraduate textbooks?
