In this page I address some of the comments made in Burgman (2008) in response to issues that I had raised in Sniedovich (2008) regarding info-gap decision theory and its use in applied ecology and conservation biology.
I plan to post here a full formal article on this topic, hopefully before the end of the year (2009).
To put this discussion in a proper perspective, I should make it clear at the outset that Mark Bugman's assessment of info-gap is fundamentally different from mine. Whereas Mark Burgman holds info-gap decision theory in high regard, I am a harsh critic of this theory.
And to give you an indication of the gap between our positions, consider this: Whereas I argue that info-gap decision theory is a voodoo decision theory par excellence, Mark Burgman is the co-author of a paper published in Ecological Modelling whose last paragraph reads as follows (Moilanen et al 2006, p. 124):
In summary, we recommend info-gap uncertainty analysis as a standard practice in computational reserve planning. The need for robust reserve plans may change the way biological data are interpreted. It also may change the way reserve selection results are evaluated, interpreted and communicated. Information-gap decision theory provides a standardized methodological framework in which implementing reserve selection uncertainty analyses is relatively straightforward. We believe that alternative planning methods that consider robustness to model and data error should be preferred whenever models are based on uncertain data, which is probably the case with nearly all data sets used in reserve planning.
Mark Burgman also believes that info-gap decision theory is " ... Of fundamental importance to all
applied sciences ..." (see flyer of the 2nd edition (2006) of the Info-Gap book).
So it should come as no surprise that I dispute many of the comments made in Burgman (2008) about info-gap.
In brief:
Burgman (2008) does not question the validity of my two basic critical characterizations of info-gap decision theory, namely
- Info-gap's robustness model is neither new nor radically different from current models for decision under severe uncertainty -- as persistently claimed in the info-gap literature. Info-gap's robustness model is in fact a simple instance of Wald's famous Maximin model (circa 1945).
- Info-gap's robustness model neither does nor is it able to deal with severe uncertainty. Info-Gap's robustness model simply takes no notice of it. That is, this model conducts a Maximin analysis around an estimate, which means that it is local in nature. Therefore, methodologically speaking, it neither does nor is it able to tackle the severity of the uncertainty (size of the total region of uncertainty) under consideration.
- Burgman's (2008) stated aim is to challenge Sniedovich's (2008) alleged misrepresentation of decision problems in the conservation biology and applied ecology literature.
But it turns out that this challenge amounts to no more than taking issue with Sniedovich's (2008) specific phrasing of a question asked in this literature.
And, alas, even this semantic challenge is a non-issue. In fact, it represents no more than Burgman's (2008) subjective choice from a variety of available phrasings of this question. Indeed, Burgman's (2008) phrasing of the question is significantly different from the one given in Burgman (2007).
So far so good.
However.
The comments in Burgman (2008) give a very distorted view of the status of info-gap decision theory in conservation biology and applied ecology.
- The fact is that in the conservation biology and applied ecology literature info-gap is viewed as a methodology for decision-making under severe uncertainty and for this purpose alone is it (mis)used in this literature.
- Burgman's (2008) comments suggest that info-gap is used in this literature for a completely different purpose. Namely to no more than determine how robust a decision is in the neighborhood of a given nominal value of the parameter of interest.
The irony is that it was Moshe Sniedovich who drew Mark Burgman's attention to the following:
- Myth: Info-gap's robustness model generates decisions that are robust under severe uncertainty.
- Fact: Info-gap's robustness model seeks decisions that are robust in the neighborhood of a given estimate.
- Myth: Info-gap's robustness model addresses the question: How wrong can one be and still get an acceptable result?
- Fact: Info-gap's robustness model does not address such questions. It address the question: How large is the 'safe' region around a given estimate?
Here are the main points made in Burgman (2008) and my response to them.
|
Point
|
My long response
|
|
1. Sniedovich's definition of severe uncertainty is too narrow to be useful.
My short response: It is not for me to define "severe uncertainty" in the context of Info-Gap decision theory. I have to adopt the working assumption stipulated by the Father of Info-Gap decision theory regarding the meaning of "severe uncertainty" in the framework of this theory -- and I do. |
I am at a loss to see on what grounds Burgman (2008) could be so wrong on this matter.
Sniedovich's definition of severe uncertainty cannot possibly be too narrow because ... it does not exist.
The fact is that Moshe Sniedovich is not in the habit of ascribing his views and definitions to others. So, as a rule, he does not provide his own definition of severe uncertainty when discussing info-gap matters. He lets the info-gap literature speak for itself. Accordingly, in Sniedovich (2008) he proceeds on the assumption that the meaning of severe uncertainty is that the estimate we have is
- a wild guess, and/or
- a poor indication of the true value of the parameter of interest, and/or
- likely to be substantially wrong.
This is precisely the working assumption stipulated by the Father of info-gap (eg. Ben-Haim 2006, 2007). It is therefore, the working assumption that Moshe Sniedovich uses in his discussions on info-gap, including Sniedovich (2008).
Burgman (2008) should have known better.
This fact is carefully pointed out in my articles and presentations at a number of which Mark Burgman was present, eg. SRA 2007 Conference where we discussed this issue in public.
So, if Burgman (2008) is unhappy with Ben-Haim's working assumption regarding the implications of severe uncertainty, he should communicate his concern to Ben-Haim.
Note that this assumption implies that, under severe uncertainty, an analysis conducted exclusively around the estimate generates results that, methodologically speaking, are wild guesses, and/or poor indications of the true values sought, and/or likely to be substantially wrong.
I suspect that this is why Burgman (2008) is so unhappy with this assumption.
For the record, this working assumption is fine with me.
Also for the record, I remind Mark Burgman of his very own working assumption regarding the meaning of "severe uncertainty". For instance, in Moilanen et al (2006, p. 118) we read (bold text is used here for emphasis)
Under severe uncertainty (Ben-Haim, 2006), we do not know the horizon of uncertainty, that is, we do not
have a meaningful estimate of the error, but info-gap theory still allows us to specify an uncertainty model.
It is therefore unclear why Burgman (2008) should object to Ben-Haim's working assumption.
|
|
2. Sniedovich's (2008) phrasing of the question asked in the applied ecology and conservation biology literatures to explain the problem addressed by info-gap's robustness model is wrong.
My short response: My phrasing of the question is not wrong. Burgman (2008) is playing with words here. |
Burgman (2008) should know better.
The applied ecology and conservation biology literatures are spotted with various verbal formulations of essentially one and the same generic question. The important thing is that all these formulations have the same basic feature in common: they are not addressed, let alone answered, by info-gap's robustness model.
Here are six phrasings of this generic question:
- Halpern et al (2006, p. 2):
How wrong can one be and still get an acceptable result?
- McCarthy and Lindenmayer (2007, p. 554):
How wrong can one be and still get an acceptable result?
- Burgman (2007):
How wrong could this model be before I should change my decision?
- Fox et al (2007, p. 192):
How wrong can a model and its parameters be without jeopardising the
quality of decisions made on the basis of this model?
- Sniedovich (2008):
How wrong can I be, yet get an acceptable level of performance?
- Burgman (2008):
How wrong can this model be, without jeopardizing an acceptable level of performance?
Ironically then, Burgman's (2007) very own version of this question reads quite differently from the one he had chosen to discuss in Burgman (2008).
Note that the formulation used in Sniedovich (2008) is based (deliberately) on the one stated in Halpern et al (2006) and on that in McCarthy and Lindenmayer (2007).
In short, Burgman (2008) does not challenge my assertion that the (real) question on the agenda is not addressed by info-gap's robustness model. Instead, he plays with words in what seems to be an attempt to transform the substantive discussion into a "semantic debate" (see 11).
So, for the record, let us remind ourselves, again, of the real issue here.
The real issue -- as indicated in Sniedovich (2008) -- is that info-gap's robustness model neither addresses nor can it answer questions such as the one stated in Burgman (2007, 2008) or in Halpern et al (2006), or in McCarthy and Lindenmayer (2007), or in Fox et al (2007).
The inevitable conclusion therefore is that the applied ecology and conservation biology literatures are spotted with misinterpretations of the question addressed by info-gap's robustness model.
No amount of playing-with-words can change this fact.
|
|
3. One should distinguish between two perspectives on info-gap's robustness model. One representing the large world of the mathematics of decision-making. The other representing the small applied world of environmental decision-making.
My short response: This distinction between the two worlds is not a license for applied environmental analysts to misinterpret the question addressed by info-gap's robustness model. |
The important point here is that by committing themselves to info-gap, whether they like it or not, info-gap users are committed to info-gap's robustness model which is a mathematical object.
Hence, the meaning ascribed to its elements and the relations between them in the "small applied world" of environmental decision-making must be consistent with the universal rules, conventions and terminology that govern the "large world" of mathematics in general and mathematics of decision-making in particular.
Consequently, whatever the mathematical findings about the mathematical model, these findings must apply in the "small world" of environmental studies and ecology.
In short, the distinction between the two worlds is not a license for applied environmental analysts to misinterpret the question addressed by info-gap's robustness model.
Therefore, Burgman's (2007) assessment of info-gap (see abstracts of the SRA 07 Program)
It is useful mainly because it answers a very specific question: how wrong could this model be, before I should change my decision
is doubly mistaken.
It is mistaken in the framework of the large world of the mathematics of decision-making AND it is mistaken in the framework of the small applied world of environmental decision-making. |
4. Sniedovich (2008) misrepresents decision-making problems in the conservation biology and applied ecology literature.
My short response: Unfortunately, my comments on the (mis)use of info-gap decision theory in the conservation biology and applied ecology literature are not misrepresentations. The fact is that in this literature the question addressed by info-gap's robustness model has been misinterpreted consistently and persistently.
|
Since Burgman (2008) does not make clear what problems Sniedovich (2008) misrepresents and in what way, one is at a loss to determine what exactly is behind this comment.
Perhaps this comment refers to the claim (Sniedovich 2008) that the conservation biology and applied ecology literatures are spotted with misinterpretations of the question addressed by info-gap's robustness model -- a claim that Burgman (2008) does not refute but takes issue with.
In this case, the offending words are "not so interesting" in the following paragraph (Sniedovich 2008, p. 9):
Rather, Info-Gap's robustness model addresses the following, completely different, much easier, and -- under severe uncertainty -- not so interesting question: How much can I deviate from a given estimate so that the performance requirement is satisfied throughout the region of uncertainty (around the estimate) which is contained within this deviation?
Burgman (2008) interprets this paragraph as though it claimed that the problem associated with this easier question is of small importance/relevance in the small applied world of environmental decision-making and proceeds to argue that this is not so (see 7).
I should therefore point out that this is not the way the text should be understood. The key words here are "severe uncertainty" and the correct interpretation of the text is as follows:
Methodologically speaking, under severe uncertainty robustness in the neighborhood of a wild guess is at best a poor approximation of robustness with respect to the total region of uncertainty. Hence, under severe uncertainty, the question "How much can I deviate ... within this deviation?" is not so interesting from the view point of the task requiring to determine robustness with respect to the total region of uncertainty under consideration.
|
| 5. " ... The reason why this question matters in a world of real decisions is that there comes a time when a specific decision (a single model from among the universe of possibilities) has to be accepted, when the reserve system has to be selected and purchased, when the investment has to be made to build a fence rather than translocate a population. This decision becomes the focal point for thinking.
When that time comes, and irrespective of how we arrived there, we want to know how much can we deviate from the given estimate (the model for the thing we are actually going to do) so that the performance requirement is satisfied throughout the region of uncertainty around the nominal estimate ... "
My short response: Burgman's (2008) assertion is wrong.
It is clear that in the conservation biology and applied ecology literature the stated goal of info-gap applications is to seek robust decisions under severe uncertainty, rather than to evaluate the robustness of a given decision around a given estimate. |
By laboring this point Burgman (2008) attempts to create the impression that in conservation biology and applied ecology info-gap decision theory is used deliberately for the purpose of determining how well a decision performs in the neighborhood of a given nominal value of the parameter of interest.
To put it bluntly, this assertion is wrong.
In fact, I am not familiar with a single publication in the area of conservation biology and applied ecology where info-gap decision theory is used expressly for the purpose of investigating the robustness of an already arrived at decision in the neighborhood of a given nominal value of the parameter of interest.
Rather, the stated goal of info-gap applications in conservation biology and applied ecology is to identify robust decisions for problems subject to conditions of severe uncertainty. And the rationale given in these publications for the use of Info Gap is very clear. Info Gap is used precisely because it is claimed to be -- both by its founder and its users -- a methodology designed specifically for identifying robust decisions under conditions of severe uncertainty.
That this is indeed the case is borne out by statements made in the conservation biology and applied ecology literature.
For example, consider this (McCarthy and Lindenmayer, 2007, p. 555):
Because the parameters of the problem, such as the risk of fire and the effect of pines on water supply, are highly uncertain, we also analyze the problem using info-gap methods (Ben-Haim 2001). The info-gap approach to decision theory is particularly well-suited to this decision problem because there is severe uncertainty that cannot be reliably described probabilistically within the time available for making the decision.
In fact, Burgman is a co-author of a paper (Regan et al, 2005) entitled
"Robust decision-making under severe uncertainty for conservation biology"
where we read this (p. 1471):
In conservation biology it is necessary to make management decisions for endangered and threatened species under severe uncertainty. Failure to acknowledge and treat uncertainty can lead to poor decisions.
and this (p. 1472)
Information-gap theory was invented by Ben-Haim (2001) to assist decision-making when there are severe knowledge gaps and when probabilistic models of uncertainty are unreliable, inappropriate, or unavailable.
and this (p. 1476)
Info-gap decision theory provides a platform extending decision theory into a broad range of conservation decision problems. For instance, it may be applied to decisions related to translocation strategies (Haight et al. 2000), probabilistic risk assessments of invasive species (Johnson et al. 2001), species management (Peterman and Anderson 1999), reserve design, and habitat management (Haight et al. 2002). In all of these contexts, it will inform us of the action that gives a satisfactory outcome, and that provides the greatest immunity against parameter and model uncertainty. This will improve flexibility in decision-making under severe uncertainty and will foster more reliable conservation management decisions.
Burgman is also a co-author of a paper (Moilanen et al 2006) where we read this (p. 118):
Under severe uncertainty (Ben-Haim, 2006), we do not know the horizon of uncertainty, that is, we do not have a meaningful estimate of the error, but info-gap theory still allows us to specify an uncertainty model.
and this (p. 115)
Having two conservation options of apparently equal biological value, one would prefer the option whose value is relatively insensitive to errors in planning inputs. In this work we show how uncertainty analysis for reserve planning can be implemented within a framework of information-gap decision theory, generating reserve designs that are robust to uncertainty.
and this (p. 116)
Furthermore, we would like to be confident that this outcome will be achieved even if the models upon which our decision is based are substantially flawed. This approach to reserve planning is entirely different from the common approach, which chooses the optimal set of conservation areas based on best-estimated models, while risking the possibility of large failure due to model
error.
and this (p. 124):
In summary, we recommend info-gap uncertainty analysis as a standard practice in computational reserve planning. The need for robust reserve plans may change the way biological data are interpreted. It also may change the way reserve selection results are evaluated, interpreted and communicated. Information-gap decision theory provides a standardized methodological framework in which implementing reserve selection uncertainty analyses is relatively straightforward. We believe that alternative planning methods that consider robustness to model and data error should be preferred whenever models are based on uncertain data, which is probably the case with nearly all data sets used in reserve planning.
In short, Burgman (2008) must no doubt be aware of the fact that the stated objective of info-gap applications in conservation biology and applied ecology is not to examine how a given decision performs in the neighborhood of a given nominal value of the parameter. Rather, in these applications info-gap is used precisely because it is viewed -- mistakenly of course -- as a methodology that provides the wherewithal for identifying robust decisions for situations subject to severe uncertainty.
So what is the net result of Burgman's contention (2008) that info-gap can be (and is) used in conservation biology and applied ecology to do something completely different, from what it was envisaged to do?
The net result is a double whammy.
Burgman thereby concedes that Info Gap is in effect unable to perform the task it was designed for, namely to perform as a methodology for decision-making under severe uncertainty. But what is more, his contention flies in the face of what the Father of info-gap has to say about this (Ben-Haim, 2007, p. 2 ):
Info-gap theory is useful precisely in those situations where our best models and data are highly uncertain, especially when the horizon of uncertainty is unknown. In contrast, if we have good understanding of the system then we don't need info-gap theory, and can use probability theory or even completely deterministic models. It is when we face severe Knightian uncertainty that we need info-gap theory.
In other words, pursuing the logic of Ben-Haim's argument, the inference is clear: one does not need info-gap theory to accomplish the objectives that Burgman (2008) claims are sought to be accomplished in conservation biology and applied ecology.
So what exactly do we need info-gap theory for?!
|
6. No method can guarantee that, under severe uncertainty, the true value is in the 'safe' area around the estimate.
My short response: This issue is a non-issue. The issue is info-gap's local approach to robustness notwithstanding the severe uncertainty.
|
The true value of the parameter of interest in the total region of uncertainty under consideration is determined entirely by Nature. It has got nothing to do, in any way shape or form, with the method we use to solve the problem. This is not an issue.
The issue here is Info-Gap's local approach to robustness notwithstanding the severe uncertainty. As I have been arguing all along, the fundamental difficulty presented by severe uncertainty is that there is no way of knowing the location of the true value. Since Info gap's prescription for robustness is to fix on an estimate and its immediate neighborhood it cannot possibly tackle situations subject to severe uncertainty. This is so because, given the severe uncertainty it is very unlikely that the true value will be in the neighborhood of the estimate. The net result is clear. Info-gap's robustness analysis does not indeed, cannot deal with the severe uncertainty. This results in its robustness model being thoroughly oblivious to the performance of a decision outside the 'safe' region around the estimate.
Thus, a decision that is hardly safe around the estimate can be highly safe elsewhere in the region of uncertainty. Conversely, a decision can be safe in the immediate neighborhood of the estimate but hardly safe elsewhere in the region of uncertainty.
That said, it is important to keep in mind that, the only test that Info Gap must be put to is its ability to deal with severe uncertainty. We have no other choice but to evaluate it on this grounds because this -- as pointed out by the profuse rhetoric in the Info-Gap literature -- is Info Gap's raison d'etre: providing a methodology for decision under severe uncertainty.
Indeed, this is what Ben-Haim (2007, p. 2 ) has to say about this point in his FAQs about info-Gap:
Info-gap theory is useful precisely in those situations where our best models and data are highly uncertain, especially when the horizon of uncertainty is unknown. In contrast, if we have good understanding of the system then we don't need info-gap theory, and can use probability theory or even completely deterministic models. It is when we face severe Knightian uncertainty that we need info-gap theory.
But how can one seriously propose to tackle severe Knightian uncertainty by confining an analysis to what happens only around a wild guess?
This is what one would expect from a voodoo decision theory, not from a scientific theory that seriously seeks to find robust decisions under severe Knightian uncertainty.
In sum, Info-Gap cannot have it both ways: if its claim to fame is its ability to tackle severe uncertainty, then it cannot ignore the performance of decisions on large sub regions of the total region of uncertainty. And if its claim is to determine only what happens around a given estimate, then it cannot possibly claim to be a methodology for decision-making under severe uncertainty. |
7. In the small applied world of environmental decision-making, the question addressed by info-gap's robustness model is life-and-death.
|
Suppose that the question addressed by info-gap's robustness model is indeed life-and-death in the context of applied environmental decision-making. And as implied by this contention: a local analysis is therefore a requisite.
Then the inevitable implication is that the associated life-and-death decision-making problems have got nothing to do with robustness against SEVERE uncertainty. And in any case, the analysis you conduct for these problems is a run-of-the-mill Maximin analysis. |
8. Experience shows that the Maximin Principle is not sufficient for decision-making under uncertainty.
|
Suppose that the Maximin Principle is indeed insufficient for decision-making under uncertainty.
Then surely so is info-gap's robustness model.
After all, this model is a simple instance of Wald's Maximin model. The inference therefore is that this is precisely the reason why it is of the first importance for applied environmental analysts to realize that info-gap's robustness model is a Maximin model in disguise. |
| 9. The strategy you choose should depend on your personal circumstances, your attitude to the outcomes, how many such games you intend to play, and so on. |
The strategy you choose should indeed depend on many things, including your personal circumstances, your attitude to the outcomes, and so on.
But this has nothing to do with the discussion on the flaws in info-gap's robustness model and the misinterpretations of this model in the literatures of applied ecology and conservation biology. |
10. Info-gap has a role to play, and can help, in addressing a variety of issues/situations.
My short response: The fact remains that info-gap's robustness model is fundamentally flawed in its primary role, namely in its role as the robustness model of a methodology that is designed SPECIFICALLY for decision-making under SEVERE uncertainty.
And the fact is that it is in this capacity that it is used in conservation biology and applied ecology. |
Suppose that the specific instance of Wald's Maximin model known in the info-gap literature as "info-gap's robustness model" can indeed have a role to play, and can help, to address a variety of issues/situations.
So what?
Recall that even a dead watch shows the correct time once -- sometimes even twice -- a day and can therefore be helpful (sometimes).
But a dead watch is a dead watch and a flawed model is a flawed model.
In the case of Info-Gap, the question, of course, is what "role" can it play and in what "situations" can it "help".
The fact remains -- as I have formally shown -- that this model is fundamentally flawed in its primary role, namely in its role as the robustness model of a methodology that is designed SPECIFICALLY for decision-making under SEVERE uncertainty.
This means that it is unsuitable for the purpose of solving problems subject to severe uncertainty. That is, it is unsuitable for solving problems such as those described in the conservation biology and applied ecology. Much less can it help to address the issues of severe uncertainty dogging the situations described in the conservation biology and applied ecology literature.
And if the idea here is that Info Gap's robustness model can be used or, can be helpful in situations that are not subject to severe uncertainty then recall again what Ben-Haim (2007, p. 2 ) has to say about this:
Info-gap theory is useful precisely in those situations where our best models and data are highly uncertain, especially when the horizon of uncertainty is unknown. In contrast, if we have good understanding of the system then we don't need info-gap theory, and can use probability theory or even completely deterministic models. It is when we face severe Knightian uncertainty that we need info-gap theory.
We do not confront severe Knightian uncertainty when seeking to determine robustness of decisions in the immediate neighborhood of a given point of the region of uncertainty.
According to Ben-Haim (2007) we do not need info-gap theory for this purpose. |
11. The discussion that Sniedovich (2008) initiated on info-gap decision theory is a "semantic debate".
My short response: The debate is not about semantics, it is about the use and promotion of a "voodoo" decision theory in Australia.
|
As indicated above, in contrast to my harshly critical view of info-gap, Mark Burgman believes that info-gap decision theory is "... of fundamental importance to all applied sciences ..." (see the info-gap book's flyer).
It is not surprising therefore that Burgman's (2008) comments try to brash away my criticism as a "semantic debate".
I find it amusing, though, that my arguments ("complaints" according to Burgman (2008)) which are all based on a rigorous formal analysis, should be shrugged off as tangential and semantic.
The truth -- unwelcome though it is to info-gap users -- is of course that my explanations of the relationship between info-gap and Maximin, the failings of info-gap's robustness model, and the misinterpretation of the question addressed by this model, are all rooted in meticulously argued explanations and formal proofs (see references in my info-gap page.).
Hence, an attempt to belittle my arguments will not cover up the fact that we are dealing here with serious issues concerning the validity of info-gap decision theory and the way it is used and promoted in general and in the small world of applied environmental decision-making in particular.
In view of Burgman's (2008) comments, it is important to remember why I initiated this "debate" in the first place, and what the central issue on the agenda is.
I initiated this "debate" as part of my "campaign" to contain the spread of info-gap decision theory in Australia. I took this step because in my view info-gap decision theory is fundamentally flawed.
The debate is then about the use in Australia of a theory that does not subscribe to fundamental, universal, scientific axioms. More specifically, a theory that is supposed to tackle severe uncertainty yet ignores it, a theory that presents its robustness model as new and radically different whereas in actual fact this "revolutionary" model is a simple instance of the most famous model in decision-making under severe uncertainty and robust optimization.
It is about a methodology that is supposed to deal with severe uncertainty, including unbounded regions of uncertainty, yet goes about it by conducting a local analysis around a wild guess.
In a word, the debate is not about semantics. It is about the use and promotion of a "voodoo" decision theory in Australia.
|
In the next section I discuss in more detail two of the points raised by Burgman (2008), as I believe that they deserve special attention.
Math vs Applied Environmental Decision-Making
The distinction made in Burgman (2008) between
- the mathematics of Info-gap's robustness model, and
- specific applications of this model in applied environmental decision-making
calls for some further elaboration.
Apparently the intended implication is that the failings of info-gap's robustness model, particularly its in principle inability to tackle severe uncertainty, are primarily of mathematical concern, for when it comes to using this model in the area of applied environmental decision-making such questions are of marginal (or no) concern as they are JUST mathematical issues.
But if this is the case then the further implication is that analysts in the small applied world of environmental decision-making, using the Info-gap model, in fact turn a blind eye to this model's inability to tackle severe uncertainty. Does this mean then that severe uncertainty is not an issue as far as they are concerned? If so, isn't this in stark contradiction to the declared aim of analysts in this area, who as attested by the applied environmental decision-making literature, point out clearly that the problems they are striving to solve using info-gap are subject to severe uncertainty?
Furthermore, if the conclusions arrived at in the mathematical analysis are relegated to the larger world of mathematics and not taken on board in the application of the info-gap model, does this mean that analysts in the small world of applied decision-making are indifferent to the severity of the stipulated uncertainty as measured, for instance, by the "size" of the (total) region of uncertainty and, that their attention is focused exclusively on the immediate neighborhood of the given estimate?
So,
- What happens under SEVERE uncertainty where, according to the working assumption in the info-gap literature, the estimate is a wild guess, and/or a poor indication of the true value, and/or likely to be substantially wrong?
- How can environmental analysts be interested only in what happens in the neighborhood of a wild guess?
- How would applied environmental analysts justify their decisions given that they are based on such a local approach to robustness against an uncertainty that is known to be severe?
- What is the justification for ignoring the performance of a decision over most of the stipulated total region of uncertainty?
These questions require careful consideration and cannot be swept under the carpet. The picture is this:
Treasure Hunt
 |
- The island represents the total region of uncertainty
under consideration (the region where the treasure is
located).
- The tiny black dot represents a wild guess of the
parameter of interest (location of the treasure).
- The large white circle represents the region of
uncertainty affecting info-gap's robustness analysis.
- The small white square represents the true
(unknown) value of the parameter of interest.
|
|
Clearly then, under severe uncertainty Info-Gap
may conduct its robustness analysis in the vicinity of
Brisbane, whereas for all we know the true location of the treasure may be somewhere in the middle of the
Simpson desert or perhaps in down town Melbourne.
Perhaps.
This is a vivid illustration of the importance of
mathematical modeling in decision-making and the
validity of the dictum: modeling is more art than
science, Sniedovich (2008).
|
My general impression is that analysts using info-gap's robustness model restrict their analysis to the immediate neighborhood of a wild guess not because this is required/dictated by the decision-making problem itself. They do so because this is dictated by the info-gap model.
This is precisely what prompted me to publish my paper in Decision Point and why my advice there was -- and is here -- that:
Whatever the case, it is important to remember that one must never fall in love with one's favorite (mathematical) model! A model is just . . . a model.
Before vs After the choice
Burgman (2008) argues that the question addressed by Info-Gap's robustness model --- namely what is happening in the neighborhood of the estimate -- is relevant and of interest -- after the choice is made:
" ... there comes a time when a specific decision (a single model from among the universe
of possibilities) has to be accepted, when the reserve system has to be selected and purchased, when the investment has to be made to build a fence rather than translocate a population. This decision becomes the focal point for thinking.
When that time comes, and irrespective of how we arrived there, we want to know how much can we deviate from the given estimate (the model for the thing we are actually going to do) so that the performance requirement is satisfied throughout the region of uncertainty around the nominal estimate. ... ''
and later on
" ... Once we've settled on a (tentative) choice, the question invariably arises, how robust is THIS choice, how far wrong can I be in the choice of parameters and assumptions for THIS model, before the outcome would be unacceptably bad. Info-gap can help. ... "
In other words, the argument is that while my criticism is valid BEFORE the choice is made, it is not valid AFTER the choice is made. That is, by some miracle, my criticism is neutralized, after the choice is made.
This is a myth.
My criticism is valid BEFORE the choice is made, and is alive and well AFTER the choice.
Here is the picture (using standard Info-Gap notation) with respect to the task of evaluating the robustness of a given decision q∈Q:
|
BEFORE the CHOICE
|
| |
AFTER the CHOICE
|
|
Info-Gap Robustness Model
|
| |
Info-Gap Robustness Model
|
|
α(q,û):= max {α: R(q,u) ≥ rc, ∀u∈U(α,û)}
|
| |
α(q,û):= max {α: R(q,u) ≥ rc, ∀u∈U(α,û)}
|
None of the objects is given.
The user must specify all the objects.
α is a control (decision) variable.
|
| |
|
All the objects except α are given.
The task is to compute the value of α(q,û).
α is a control (decision) variable.
|
So AFTER the choice, Info-Gap's robustness model determines the largest value of α such that the performance requirement R(q,u) ≥ rc is satisfied for all u∈U(α,û), recalling that û denotes the estimate of the true value of u, and U(α,û) denotes the region of uncertainty of size α centered at û.
My criticism is that AFTER the choice, this model ignores how good/bad decision q performs outside the region of uncertainty U(α',û), where α' is slightly larger than α(q,û), say α' = α(q,û)+ε, for some very small, positive ε.
The picture is shown in in the next figure, where the the gray square represents the complete region of uncertainty and the No Man's Land represents the part of the complete region of uncertainty that Info-Gap's robustness model ignores. In other words, the performance of decision q on the No Man's Land part of the complete region of uncertainty has no impact whatsoever on the Info-Gap's robustness model evaluation of the robustness of q.
This is absurd.
The point is that under severe uncertainty, and AFTER the choice, the true value of u is very likely to be outside the region U(α(q,û),û). After all, under severe uncertainty the estimate û is a wild guess of the true value of u and therefore, methodologically speaking, we have to assume that the true value is unlikely to be in the neighborhood of û.
In other words, the flaw in Info-Gap's robustness model -- AFTER the choice -- is that it does not deal with the severity of the uncertainty under consideration, it simply ignores it and evaluates the performance of d only in the neighborhood of û.
The CHOICE itself -- of q, û and the other parameters of Info-Gap's robustness model -- does not diminish the severity of the uncertainty, hence, Info-Gap's robustness model remains as flawed AFTER the choice, as it were BEFORE the choice.
In short, Burgman's (2008) AFTER THE CHOICE issue is a non-issue.
Talking about AFTER.
Suppose that AFTER the big choice, you discover that actually the complete region of uncertainty is much larger than what you originally thought it was. How would that change your view about the robustness of the decision that you have chosen?
If you use Info-Gap decision theory, you don't have to worry about this. If AFTER the big choice you have to re-evaluate the robustness of the given decision because the complete region of uncertainty turns out to be much larger, then as long as the estimate does not change, the robustness of your decision will not change either!
The proof to my Invariance Theorem shows (eg. Sneidovich 2007) that the robustness of decision q will remain constant at α(q,û) as long as the complete region of uncertainty, U, contains the set U(α',û), α'=α(q,û)+ε for some &epsilon >0.
The picture is shown in the next figure, where α*= α(q,û), and the white circle represents the 'safe' region a la Info-Gap, namely U(α*,û). The robustness of decision q remains constant at α(q,û) no matter how much you increase the complete regions of uncertainty, say from U, to U', to U''', and so on.
This is absurd.
And this is why I consider Info-Gap decision theory to be a classical voodoo decision theory. It presents itself as a methodology that is designed specifically for decision-making under severe uncertainty, yet it actually does not tackle the severity of the uncertainty.
Conclusions:
- Not only does Burgman's (2008) challenge fail to refute the issues that I raised in Sniedovich (2008), the comments in Burgman (2008) confirm the validity, relevance and importance of these very issues.
- Bergman's (2008) comments also reinforce my general observation that a huge GAP exists in the info-gap literature between the rhetoric describing info-gap decision theory and what it actually is and does.
- The debate is not about semantics, it is about the use and promotion of a "voodoo" decision theory in Australia.
It is time for info-gap users to address these issues, in fact this is long overdue.
References:
- Ben-Haim Y (2006) Info-gap Decision Theory: decisions under severe uncertainty. Academic Press.
- Ben-Haim Y (2007) FAQs about info-gap, http://www.technion.ac.il/~yakov/IGT/faqs01.pdf
- Burgman M (2008) Shakespeare, Wald and decision making under uncertainty. Decision Point, 23, p. 10. http://www.aeda.edu.au/docs/Newsletters/DPoint_23.pdf
- Burgman M (2007) Info-gap: a user's evaluation of info-gap, SRA 2007. http://www.acera.unimelb.edu.au/materials/papers%2007/Mark_Burgman_SRA.pdf
- David R. Fox. D.R., Ben-Haim, Y., Hayes, K.R., McCarthy, M.A., Wintle, B., and Dunstan, P. (2007) Info-gap approach to power and sample size calculations, Environmetrics, 18, 189-203.
- Halpern, B.S., Regan, H.M., Possingham, H.P. and McCarthy, M.A. (2006) Accounting for uncertainty in marine reserve design, Ecology Letters, 9, pp. 2-11.
- McCarthy, M.A. and Lindenmayer, D.B. (2007) Info-Gap Decision Theory for Assessing the Management of
Catchments for Timber Production and Urban Water Supply, Environmental Management, 39, 553-562.
- McDonald-Madden, E., Baxter, P.W.J., and Possingham, H.P. (2008) Making robust decisions for conservation with restricted money and knowledge, Journal of Applied Ecology, 45(6), 1630-1638.
- Moilanen A, Rungeb M.C, Elith J, Tyred A, Carmel Y, Fegraus E, Wintle B.A, Burgman M, Ben-Haim Y (2006) Planning for robust reserve networks using uncertainty analysis. Ecological Modelling, 199, pp. 115-124.
- Regan, H.M., Ben-Haim, Y., Langford, Wilson, W.G., Lundberg, P., Adelman, S.J., and Burgman, M.A., (2005) Robust decision-making under severe uncertainty for conservation biology, Ecological Applications, 15(4), pp. 1471 - 1477.
- Sniedovich M (2008) From Shakespeare to Wald: Modelling worst-case analysis in the face of severe uncertainty. Decision Point, 22, pp. 8-9. http://www.aeda.edu.au/docs/Newsletters/DPoint_22.pdf
