What drew my attention to this Policy brief (henceforth Brief) was the information stated at the bottom of Table III.2 (page 44) whose caption reads as follows:
Here is a reproduction of this part of the table.
| 2. Method applicable when exact probabilities are not known | |||||
| Decision method | Decision criteria | Preference assumptions | Information assumptions | Additional requirements | Reference Annex A |
| Maximin expected utility | As for expected utility | As for expected utility + Extreme ambiguity aversion (act as if the worst plausible probability distribution were correct) | Multiple plausible probability distributions. | 6.i | |
| Smooth ambiguity model | As for expected utility | As above, but allows for any attitude to ambiguity. | Multiple plausible probability distributions, and weights on each of these distributions | 6.i | |
| Maximin | Any | Ordinal ranking of outcomes24 | No likelihood information | 6.iii | |
| Minimax regret | Any | Cardinal ranking of outcomes25 | As above | 6.iii | |
| Info-gap decision theory | Various | Does not rigorously account for preferences. Assumes satisficing26 thresholds, i.e. acceptable levels of minimum performance/maximum windfall. | A "best guess" model of the decision environment, and a set of models that are "close" to this best guess. | A method for measuring the distance between different models (an "uncertainty model" | 6.iii |
25 Cardinal rankings allow us to say how much better one outcome is than another. Differences between outcomes that are ranked cardinally are meaningful.
26 A satisficing threshold is the value of a decision criterion at which an adaptation option is considered "good enough". See Annex A, section 6.3.
As pointed out in the table, details of these decision methods are discussed in Annex A. So, to assess the analysis presented in this Brief we need to examine Annex A. However, before we do this, I want to call attention to a number of points arising from specific claims made in the table itself and to comment on them briefly:
- The caption of the table puts all the listed methods on an equal footing treating them all as standard decision methods. My question to the authors is therefore this: does this correctly reflect the situation in the area of decision-making?
- The table does not indicate what scope of valid applications do these methods have.
- The characterization of info-gap decision theory is downright incorrect.
For obvious reasons, in this review report I focus exclusively on info-gap decision theory.
In greater detail:
The point that I want to highlight through the question that I am putting above to the authors is this. Are the authors earnestly claiming that decision theory specialists, or more generally, specialists in disciplines concerned with robust decision-making regard info-gap decision theory as a "standard decision method" in the same manner that they would regard say, Wald's Maximin model as a standard method?! Surely, judged on this criterion alone, info-gap decision theory cannot by any stretch of the imagination be described as a "standard decision method".
As for the characteristics attributed to info-gap decision theory.
The first point to note is that the statement in the table regarding info-gap's account of preferences is simply in error. Info-gap decision theory does most definitely seek to establish a rigorous preference relation among decisions. Indeed, preferences are determined as follows: the more robust the decision (for the critical performance level under consideration) the better it is. And if decisions are evaluated on the basis of their opportuneness, then the smaller the opportuneness the better the decision. In other words, info-gap decision theory claims to offer decision models that stipulate a preference relation whereby decisions are ranked according to their robustness (or, opportuneness) levels. Hence, the optimal decision is that with the highest (largest) robustness, or the lowest (smallest) opportuneness.
In the same vein, the statement in the table claiming that info-gap's uncertainty sets are `close' to the `best guess' is also incorrect. The fact of the matter is that info-gap's uncertainty sets can be vast so as to encompass areas of the uncertainty space that are significantly distant from the `best guess'. Indeed, according to Ben-Haim (2001, 2006, 2010), info-gap's uncertainty sets are very often unbounded.
Third, to set aright the (wrong) impression given by the fact that this table places info-gap decision theory on an equal footing with other methods, I remind the reader that ... info-gap's robustness model and info-gap's decision model are in fact ... simple Maximin models (more on this below).
In short, the information provided by this table about info-gap decision theory misrepresents the facts about this method!.
One can of course argue that this Brief was written by analysts/scholars specializing in Climate Change, and not in decision theory, and that this fact explains the errors in this table and in Annex A.
But consider this statement on page 13 (red color added):
This research was conducted by the Centre for Climate Change Economics and Policy (CCCEP) and the Grantham Research Institute on Climate Change and the Environment at the London School of Economics and Political Science (LSE), in consultation with a number of experts on decision-making and sector-level adaptation. The Ecosystems case study was contributed by Alice Hardiman from the Royal Society for the Protection of Birds. The research was sponsored by the secretariat to the ASC.
For the sake of color-blind readers, I note that I show the clause: "in consultation with a number of experts on decision-making" in red so as to highlight this fact.
In other words, I want to call attention to the fact that the authors did the right thing: they consulted experts in decision-making. The trouble is, however, that at least insofar as info-gap decision theory is concerned, they were either ill advised, or they misunderstood the advice given to them, or both. The end result is that the discussion on info-gap decision theory in this Brief, especially in Annex A, is --- to put it mildly --- highly problematic and requires attention (correction?).
With this in mind, let us now examine the discussion in Annex A on info-gap decision theory.
Annex A's perspective on info-gap decision theory
On page 18 of Annex A we read:A related approach, known as Info-gap decision theory (Ben-Haim 2006), provides methods that yield qualitative information about the robustness of decisions to uncertainty around a best guess parameter estimate. "Robustness" and "opportuneness" curves are generated, which measure the maximum amount of uncertainty the decision maker can be exposed to and still ensure that losses do not exceed a given level (in the case of robustness), or the minimum amount of uncertainty the decision-maker must expose herself to in order to have the possibility of achieving a windfall of a certain level (in the case of opportuneness). These curves are then fed into an informal decision process (info-gap does not account for preferences rigorously), in which the decision maker is asked to specify acceptable levels, i.e. the largest loss she is willing to sustain and the smallest windfall she wishes to have the possibility of achieving, and picks options based on their trade-offs between robustness and opportuneness at these levels. A vital part of the info-gap method is the choice of uncertainty model, i.e. how uncertainty is defined and measured. This is an ad-hoc modelling choice, and much of the art in the analysis may lie in picking a model that is suited to the application at hand. It should be emphasized that Info-gap is in one respect less general than the methods employed by (Hansen & Sargent 2008), since it does not achieve robustness to changes in model structure, but only to uncertain parameters or a model.
There are a number of errors/misrepresentations in this profile of info-gap decision theory. Some are more glaring than others. This therefore leads me to conclude that the authors of this Brief misunderstood the explanations given to them (by experts?) of info-gap decision theory and what it aims to do.
For now I discuss only a number of those points that must be corrected immediately. My plan is to come back to this discussion every so often and to expand on it some more.
Comment 1:
Take note of the assertion that info-gap decision theory " ... provides methods that yield qualitative information about the robustness of decisions to uncertainty around a best guess parameter estimate. ..." My question to the authors is as follows:
Is this a typo or do the authors actually mean qualitative rather than quantitative?
Because, if the authors indeed mean "qualitative", then they should take note that according to info-gap decision theory, the robustness and opportuneness of decision q∈Q are yielded by these quantitative mathematical models:
| Robustness model | Opportuneness model | |
| α(q,rc) = max {α≥0: r(q,u) ≥ rc ∀u∈U(α,û)} | β(q,rw) = min {α≥0: r(q,u) ≥ rw for at least one u∈U(α,û)} |
Are the authors seriously claiming that these models are qualitative ?!!!?!?!!
Technical Remark: with no loss of generality assume that r(q,û) > rc and r(q,û) < rw for all q∈Q.
And consider this (emphasis added):
In such situations we are dealing with “true uncertainty” in the sense of Knight (1921) who was the first to distinguish between “risk” based on known probability distributions and true uncertainty when the underlying statistical distributions are unknown. Knight’s ideas have been further developed by several authors over the years and in particular by Ben-Haim (2006) who has developed a quantitative formulation known as information-gap decision theory. This theory has recently been shown by Sniedovich (2008) to be formally equivalent to Wald’s maximin model in classical decision theory (French, 1988).
Beresford and Thompson (2009, p. 278)
Managing credit risk with info-gap uncertainty, Journal of Risk Finance, Vol. 8 No. 1, pp. 23-34.
See Comment 3 for more details on the maximin connection.
Comment 2:
Info-gap decision theory claims to provide decision-makers two decision models (Ben-Haim 2001, 2006, 2010). A robustness model that is aimed for decision-makers whose concern is robust decision. And an opportuneness model aimed for decision-makers whose concern is with opportuneness. Decision-makers concerned with robustness select a decision from the set of decisions whose robustness is high (large), whereas decision-makers concerned with opportuneness select decisions whose opportunenss is low (small). In short, info-gap decision theory instructs the former to select decisions whose robustness is the highest (largest), and the latter to select decisions whose opportuneness is the lowest (smallest). To this end, info-gap decision theory puts forth two decision models formulated as follows:
| Decision model for robustness | Decision model for opportuneness | |
| max {α(q,rc): q∈Q} | min {β(q,rw): q∈Q} |
Now, the info-gap literature (notably its primary texts (Ben-Haim 2001, 2006, 2010)) indicates in no uncertain terms that decision makers dealing with situations that are subject to severe uncertainty are concerned mostly with robustness rather than with opportuneness. Indeed, this literature overwhelmingly discusses applications dealing with robustness rather than with opportuneness.
The implication therefore is that the following narrative in the Brief:
These curves are then fed into an informal decision process (info-gap does not account for preferences rigorously), in which the decision maker is asked to specify acceptable levels, i.e. the largest loss she is willing to sustain and the smallest windfall she wishes to have the possibility of achieving, and picks options based on their trade-offs between robustness and opportuneness at these levels.
misrepresents info-gap decision theory's basic claims and its intended application.
I want to particularly single out the phrase "...and picks options based on their trade-offs between robustness and opportuneness at these levels." Because, this phrase attests to an astounding misapprehension of the info-gap methodology.
According to Ben-Haim (2006, pp. 169-171),
Comment 3:
The reference to the methods employed by Hansen and Sargent (2008) is grossly misleading. The Brief claims that
It should be emphasized that Info-gap is in one respect less general than the methods employed by (Hansen & Sargent 2008), since it does not achieve robustness to changes in model structure, but only to uncertain parameters or [sic] a model.
The trouble with this comparison is that it conceals a great deal more than it reveals about info-gap decision theory.
To begin with, if the intention is to point at the differences between info-gap decision theory and other methods, to indicate which method is more general, then surely the most obvious candidate for such a comparison to top the list would be Wald's maximin model. And to appreciate why this is so consider this fact:
Theorem 1: Info-gap robustness model and info-gap decision model for robustness are simple instances of Wald's Maximin model (circa 1940).
To be even more accurate:
Theorem 2: Info-gap robustness model is a very simple instance of the famous Radius of Stability model (circa 1960) itself a simple instance of Wald's Maximin model.
You can find formal proofs of these theorems in two recent papers on this topic:
- Sniedovich, M. (2010) A bird's view of Info-Gap decision theory, Journal of Risk Finance, 11(3), 268-283.
- Sniedovich, M. (2012) Black swans, new Nostradamuses, voodoo decision theories and the science of decision-making in the face of severe uncertainty, International Transactions in Operational Research, available on line.
But if you want immediate access to such proofs, have a look at these:
- Mobile Maximin Theorem (show/hide).
- Mobile Radius of Stability Theorem (show/hide).
These two theorems debunk info-gap decision theory outright (see why ....). The bottom line is that info-gap's robustness model is a re-invention of a very famous wheel (universally known as Radius of Stability), and ... a square one at that!
I should also set the record straight on the information conveyed by the phrase: "... it does not achieve robustness to changes in model structure, but only to uncertain parameters or [sic] a model." Hence,
Comment 4:
As correctly indicated by the authors (elsewhere in the above quoted paragraph), info-gap's robustness analysis is indeed conducted around a best guess parameter estimate. Hence, as (almost) correctly indicated by this phrase, info-gap's robustness analysis seeks to "...achieve robustness ... only to uncertain parameters ... ".
But ... given that this is the case, surely this should have indicated to the authors that it is their duty to explain how such a method can possibly be suitable for robust decision-making under conditions of deep uncertainty? On what grounds can it possibly be claimed that info-gap decision theory provides the requisite methodology that is called for by this task? Are the authors suggesting that a theory of local robustness (to a "best guess") is the appropriate tool for dealing with the deep uncertainty that characterizes climate change phenomena of the type examined in the Brief?
Comment 5:
On page 16-17 of Annex A we read (emphasis added):
Aside from their lack of dependence on probabilities, a feature of many robust decision methods is that they do not attempt to maximize anything. Rather they aim to find decision strategies that satisfice, i.e. achieve an acceptable level of some, possibly conflicting, objectives. This is true of Info-gap decision theory, as well as the global robustness methods advocated by Lempert (2002) and Lempert et al. (2004), and applied to adaptation decisions by Dessai & Hulme (2007).
I want to single out for special attention the hair raising statement: "... a feature of many robust decision methods is that they do not attempt to maximize anything." I leave it to the reader to reflect on the claim made by this statement. At this stage all I do is put the following question to the authors: are the authors seriously claiming that info-gap decision theory does not maximize anything?!
In that case, how do they square this claim with the following statement in Dessai et al. (2009, p. 74) which is one of their main references (emphasis added)?
Info-gap decision theory is a non-probabilistic decision theory seeking to optimize robustness to failure, or opportunity of windfall. This differs from classical decision theory, which typically maximizes the expected utility.
And this:
We use info-gap theory for satisficing (not minimizing) the probability of detection, while at the same time maximizing the robustness to uncertainty.
Lior Davidovitch, Richard Stoklosa, Jonathan Majer, Alex Nietrzeba, Peter Whittle, Kerrie Mengersen, Yakov Ben-Haim.
Info-gap theory and robust design of surveillance for invasive species: The case study of Barrow Island
Journal of Environmental Management, Volume 90, Issue 8, Pages 2785-2793, 2009.
And how do the authors square their claim with this statement (color added)?
Satisfice profit, maximize robustness. We now consider a robust-satisficing strategy which both guarantees no less than a specified minimum profit (if possible) and which maximizes the robustness or immunity to uncertainty.And with this statement (emphasis added):
Ben-Haim, Y. Info-Gap Decision Theory (2006, p. 92).
This is the allocation which maximizes the robustness while satisficing the capital reserve requirement
Ben-Haim, Y. Info-Gap Economics: an Operational Introduction (2010, p. 132).
Those readers who are not familiar with info-gap decision theory are advised that Prof. Y. Ben-Haim is the Father of this theory.
The question is of course, how could such a (bewildering) misconception of the very nature of info-gap decision theory, possibly have infiltrated this Brief?
Readers seeking answers to this intriguing question should obviously direct it to the authors of the Brief. All I am going to say here, at this stage, is that this seems to be an interpretation of the incessant (misleading) babble in the info-gap literature about the difference between optimizing and satisficing gone wrong. In particular, this seems to be a misconstruction of the incessant (misleading) babble in the info-gap literature claiming that info-gap decision theory is a robust-satificing theory rather than a utility-maximizing theory.
All the same, I want to point out that I'd be delighted to discuss this matter with anyone interested in this issue, over a cup of coffee (soy latte, please).
And there is more ...
Environment Agency, UK
It is surprising that the authors seem to be unaware of the 2009 Department for Environment Food and Rural Affairs (DEFRA) report. To be precise, what I am particularly referring to is the following paragraph on page 75 of the report:
More recently, Info-Gap approaches that purport to be non-probabilistic in nature developed by Ben-Haim (2006) have been applied to flood risk management by Hall and Harvey (2009). Sniedovich (2007) is critical of such approaches as they adopt a single description of the future and assume alternative futures become increasingly unlikely as they diverge from this initial description. The method therefore assumes that the most likely future system state is known a priori. Given that the system state is subject to severe uncertainty, an approach that relies on this assumption as its basis appears paradoxical, and this is strongly questioned by Sniedovich (2007).Mervyn Bramley, Ben Gouldby, Anthony Hurford, Jaap-Jeroen Flikweert
Marta Roca Collell, Paul Sayers, Jonathan Simm, Michael Wallis
Delivering Benefits Through Evidence
PAMS (Performance-based Asset Management System)
Phase 2 Outcome Summary Report
Project: SC040018/R1
Product code: SCHO1209BRRA-E-E
SubTitle: Report SC040018/R1
Pages: 132
ISBN: 9781849111638
Environment Agency -- December 2009
UK
Department for Environment Food and Rural Affairs
Note: The URL of the report keeps changing. Try this one (downloaded on March 6, 2011):
http://publications.environment-agency.gov.uk/epages/eapublications.storefront/4d72dd4801b700e6273fc0a80296066c/Product/View/SCHO1209BRRA-E-E#
Deep uncertainty
The term Deep Uncertainty seems to be used in this Brief to put across the severity, or extremity, of the uncertainty that this document is concerned with. Indeed, the impression one gets is that the term Deep Uncertainty is intended to convey the message that the uncertainty that the authors are concerned with is in some way different from the, if you will, Knightian (non-probabilisitc) uncertainty that is considered in classical decision theory. Yet, there is no discussion in this document to explain in what sense is the uncertainty designated by Deep Uncertainty different, if at all, from the (non-probabilisitc) uncertainty that is postulated by classical decision theory. In brief what is it that qualifies the uncertainty that this policy brief is concerned with for the designator deep uncertainty?
I would therefore like to call the authors' and the readers' attention to the following paragraph:
... Another concern of the committee regarding the content of this chapter involves the use of the concept of "robustness." The committee finds that this term is insufficiently defined. A plausible argument can be made that there is no meaningful distinction from usual optimality analysis and that the concept discussed in this report is a matter of a poorly defined utility function. If indeed there is a real technical distinction to be made, the authors should consider expanding and supporting the discussion of this concept. Furthermore, the committee suggests that the authors address the concept of adaptive management in conjunction with discussions of robustness and in particular address how different sources of uncertainty affect different kinds of decisions. Finally, the committee would appreciate a further elucidation of what the author considers to constitute "deep uncertainty" (page 34 and other locations). The committee understands that there is overlap between this concept and the others defined in this section (e.g., "robust"), but nevertheless finds that it is not entirely clear when the author considers the situation inappropriate for use of conventional methods for characterizing uncertainty.
Review of the U.S. Climate Change Science Program's Synthesis and Assessment Product 5.2
"Best Practice Approaches for Characterizing, Communicating, and Incorporating
Scientific Uncertainty in Climate Decision Making"
pp. 17-18, 2007
http://www.nap.edu/catalog/11873.html
My suggestion to the authors is that they would do well to reflect on this statement, as well as on Golomb's (1970) 9-th don't of mathematical modeling, quoted below, before they next set out to write on this topic:
Don't apply terminology of subject A to the problems of subject B if it is to the enrichment of neither. Catch Phrase: 'New names for old.'
Golomb, S.W., 1970. Mathematical Models - Uses and Limitations. Simulation, 4 (14), 197-198.
What's next?
Despite the crystal clear technical assessments of info-gap decision theory that are easily accessible to the public, for some odd reason, analysts/scholars in the areas of applied ecology and climate change seem to have great difficulties in comprehending the basic facts about info-gap decision theory.
Considering then that the basic facts about this theory continue to elude scholars/analysts in a number of research centers in the UK, I am tempted to extend my info-gap campaign to the UK!
Stay tuned, as there is more to come ...
