The Spin Stops Here!
Decision-Making Under Severe Uncertainty  
Faqs | Help | @ | Contact | home  
voodoostan info-gap decision theory info-gap economics severe uncertainty mighty maximin robust decisions responsible decisions

The CSIRO Report (Keith Hayes, 2011)

Visitors interested in this topic should also be interested in my 200-page report to ACERA.

This page is dedicated to a discussion of the following important recent publication:

Hayes KR (2011). Uncertainty and Uncertainty Analysis Methods.
Final report for the Australian Centre of Excellence for Risk Analysis (ACERA),
CSIRO Division of Mathematics, Informatics and Statistics,
Hobart, Australia,
130 pp.
PDF file

I should point out, though, that my discussion here revolves only around the issues arising from the assessment of info-gap decision theory in this report -- henceforth REPORT.

Note that the REPORT per se does not make info-gap decision theory its primary concern. Still, it is of interest to me because it identifies and discusses some of the flaws in info-gap decision theory.

As indicated by the title, the REPORT was written for the Australian Centre of Excellence for Risk Analysis (ACERA). I should therefore note that ACERA has been a key player in the promotion of info-gap decision theory in Australia.

But, first ... a few words about the CSIRO.

The Commonwealth Scientific and Industrial Research Organization (CSIRO) is the federal government body for scientific research in Australia. It was founded in 1926, originally as the Advisory Council of Science and Industry. It now employs more than 6600 people all over Australia. Its Headquarters is in Canberra, Australian Capital Territory.

The author of the REPORT, DR Keith Hayes, is a senior research scientist at the CSIRO Division of Mathematics, Informatics and Statistics. His research on risk in relation to complex ecological problems is focused on the areas of:

Keith is located at CSIRO Marine and Atmospheric Research --- Hobart, Tasmania, Australia.

I commend Keith for this initiative. This is definitely a step in the right direction. However, ... parts of the discussion on info-gap decision theory in the REPORT must be re-examined. Because alas, a number of typical info-gap bugs have infiltrated even this REPORT!

I hope that this discussion will encourage Keith, and other scientists in Australia, to take an even closer look at the flaws in info-gap decision theory and the misconceptions that it propounds, especially from the pages of peer-reviewed journals (see my compilation of reviews of info-gap publications).

This is long overdue.

Table of contents

Significance of the report

The publication of the REPORT has considerable implications for my campaign to contain the spread of info-gap decision theory in Australia. Because, for the first time since I launched my campaign at the end of 2006, scientists in an Australian government organization have stated, in print, views that support my criticism of info-gap decision theory and challenge the validity of claims made in the info-gap literature. It is gratifying therefore to read views supporting what I have been arguing publicly for so long, stated on the pages of such a report.

I am afraid, though, that readers will have to read ... between the lines of the REPORT to get the full measure of the flaws in the theory that for so long has captivated risk analysts in Australia. For one thing, the REPORT does not spell out the obvious conclusions deriving from its analysis of info-gap decision theory --- conclusions that are spelled out clearly and unambiguously in my articles and presentations on this topic.

For instance, consider this:
Analysts who were attracted to IGT because they are very uncertain, and hence reluctant to specify a probability distribution for a model's parameters, may be disappointed to find that they need to specify the plausibility of possible parameter values in order to identify a robust management strategy.
Hayes (2011, p. 88)

So first, let me confess that I admire Keith's use of language here!

Still, I submit that for the benefit of all those who may contemplate turning to info-gap decision theory, and particularly for the benefit of info-gap scholars, it is imperative to be more forthright on the issues involved. Because, as I explain below, one of the hidden messages in this rather diplomatically phrased statement is this:

The good old days of ALCHEMY are long gone!

It is generally accepted nowadays that creating something out of nothing is well-nigh impossible.

This means that if a model of uncertainty is claimed to be probability-free, likelihood-free, plausibility-free, and so on, then the results yielded by this model are by necessity ... probability-free, likelihood-free, plausibility-free, and so on. Namely, no amount of rhetoric will possibly change this fact to induce the model to yield results that are not likelihood-free and so on.

But this universally accepted fundamental does not apply in info-gap decision theory. Thus, while a big fuss is made, in the info-gap literature, about info-gap's model of uncertainty being probability-free, likelihood-free, plausibility-free, info-gap's prescription for the management of uncertainty flies in the face of this fundamental stipulation.

This means that info-gap's recipe for the management of severe uncertainty is in fact ALCHEMY par excellence, but this fact is obscured from view by spin and rhetoric.

In greater detail, the rhetoric in info-gap publications describing the model and the results yielded by this model is utterly incongruous with what this model actually is and does, indeed what this model is capable of doing.

The implication is then that misleading rhetoric obscures from view that info-gap decision theory is utterly unsuitable for the treatment of severe uncertainty of the type (severity) that it claims to address.

Now, as I can afford to be less diplomatic than Keith, I shall spell out the fundamental trouble in info-gap decision theory more clearly, indeed more bluntly, as follows:

Info-gap decision theory cannot possibly deliver on what it claims to deliver.

This is so because info-gap decision theory is proclaimed to be a non-probabilistic and likelihood-free decision theory. But to justify the application of its prescription for the management of severe uncertainty, one must impose a specific plausibility/likelihood structure on the uncertainty space.

That is, one cannot even begin to implement the info-gap methodology without first of all stipulating a specific plausibility/likelihood structure on the uncertainty space. Because, short of such a stipulation, one would be unable to justify info-gap's prescription to focus the entire robustness analysis on a single (poor) point estimate and its neighborhood. In other words, short of such a stipulation, one would be unable to justify the inherently local nature of info-gap's notion of robustness.

Surely, it does not take a risk analyst to immediately see how utterly at odds this prescription is with the proclaimed objective of the theory, which is the treatment/management of a severe uncertainty that is quantified in a non-probabilistic, likelihood/plausibility free fashion.

But, more than this, it is important to appreciate that this contradiction is inescapable, because unless one imposes this specific plausibility/likelihood structure on the uncertainty space, applying the info-gap decision methodology would make no sense at all!

And the upshot of all this is clear: given this foundational contradiction, which lies at the very heart of this theory, the info-gap methodology cannot possibly be suitable for the treatment/management of severe uncertainty of the type (severity) it claims to manage. Because, the measures it prescribes taking are in stark contradiction to the severity of the assumed uncertainty.

It is important, therefore, that one's assessment/evaluation of info-gap decision theory be based in the first place on these two hard facts.

And now back to the REPORT, specifically to the comment on the "disappointment" awaiting unsuspecting info-gap users.

The important point to note here is that this "disappointment" is only one of many that are in store for analysts setting out to use info-gap decision theory. Namely, it is no doubt true that analysts are bound to be disappointed when they realize that info-gap's presumed attraction --- of exempting one from specifying a likelihood/plausibility structure on the uncertainty space U --- is no more than an illusion. But this, as well as other "disappointments" are an immediate consequence of a deeper problem. The "disappointments" in store are the immediate consequence of the fundamental flaw afflicting info-gap decision theory described above, which is its prescription to apply a model of local robustness to deal with a severe uncertainty that is characterized by

The trouble is, however, as attested by the info-gap literature, that info-gap scholars are clearly not disappointed at all by all this, which can only mean that info-gap scholars are unaware of the flaw identified in the REPORT, hence of the need to supplement the info-gap methodology with a likelihood/plausibility model.

As a matter of fact, Hall and Harvey (2009) and Hine and Hall (2010) even go so far as to (mistakenly) assume that such a likelihood/plausibility model is already posited by info-gap decision theory!!!! (see Review 6 and Review 15)

I hope that the discussion on info-gap decision theory in the REPORT will encourage other scientists in Australia to speak their minds openly on this matter so as to impress on applied ecologists in the Land of the Black Swan how flawed info-gap decision theory actually is.

This is long overdue.

But it is even more important to impress on info-gap scholars and users that there are other theories available for decision-making under severe uncertainty --- theories which, unlike info-gap decision theory, are suitable for this purpose because they do have the capabilities to perform that which a theory for severe uncertainty ought to be able to do! The point is then that these theories have the capabilities that info-gap decision theory clearly does not have. As pointed out in this discussion, the field of robust optimization offers a rich knowledge-base on this subject.

Scope

Info-gap decision theory is, on the testimony of its founder (Ben-Haim 2001, 2006, 2010), a ... decision theory. Its overriding objective is to determine the best decision under conditions of severe uncertainty. Therefore, in my discussions I assess it as a ... decision theory.

In contrast, the focus in the REPORT is on the classification and quantification of uncertainty, and on methods for its analysis, not on decision-making as such. Therefore, at first glance, it may appear somewhat surprising that a decision theory is included in this investigation to begin with, and that the chosen odd-fellow is ... info-gap decision theory.

But this only prima facie odd choice reflects the fact that info-gap decision theory is extremely popular among applied ecologists in Australia, so that it warrants a proper assessment.

What is important is that Hayes (2011) does discuss, clarify, and illustrate one of the major flaws in info-gap decision theory that I discuss in my articles and presentations (e.g. Sniedovich (2007, 2010, 2011). This, as the REPORT observes, is that info-gap decision theory does not provide a mechanism to deal with the poor quality of the estimate of the parameter of interest, while at the same time, it instructs focusing the local robustness analysis in the neighborhood of this point estimate. The reason that info-gap decision theory lacks this mechanism is due to the fact that the theory does not attribute any likelihood/probability structure to its uncertainty model.

The main part of info-gap decision theory's assessment in the REPORT reads as follows:

In biosecurity risk assessment one of the most severe forms of uncertainty is our limited understanding of complex ecological processes that manifests as model structure uncertainty. IGT does not provide a ready-made solution to this problem and, as with many other applications of uncertainty analysis, this form of uncertainty is typically not addressed in ecological applications. IGT provides an alternative non-probabilistic way to express uncertainty, but in most ecological applications it is applied to uncertain parameters of probabilistic models, such as the rate of a Poisson process, or the probability of detecting a pest in a trap. Its greatest strength is that it places uncertainty at the forefront of the decision selection problem.

An important point is that its recommendations could be sensitive to the initial estimates of the uncertain parameters. As a method of uncertainty analysis it is not unique in this regard, but, as Figure 4.11 demonstrates, small departures from an initial estimate can still lead to different conclusions.

This is important because IGT does not distinguish between the likelihood of different initial estimates. Hence, if recommendations based on an Info-gap analysis change with different initial estimates, and these estimates are highly uncertain (for example two equally credible experts have different views on the `best' initial estimate) then the theory may not be able to unambiguously identify the best course of action. If the robustness is low at the point where the preference order of the two decisions change (conditional on the required reward\index{reward}) then the theory highlights that the current level of understanding and information is insufficient for reliable decision-making. This insight, of course, presumes that analysts test for the effect of different initial conditions when using IGT.

Hayes (2011, p. 92)

A careful reading of this assessment reveals that the conclusion that should in fact be drawn is that info-gap decision theory does not even begin to address the difficulties encountered in view of the uncertainty under consideration being severe.

And I should add that the REPORT does not discuss at all the implications of info-gap's robustness model being a simple instance of the Radius of Stability (circa 1960) model and of Wald's Maximin model (circa 1940). In fact, there are no references in the REPORT to the literatures on the treatment and management of severe uncertainty by decision theory and robust optimization.

And this leads me straight to what the REPORT describes as a "debate" about two of my claims regarding info-gap decision theory.

The Debate

Consider this interesting statement in the REPORT (emphasis added):
There is, however, an on-going debate surrounding IGT that revolves around two claims: a) IGT is not a radically new theory but rather a reformulation of minimax analysis that has been known in the mathematical research literature for over 60 years; and b) its results are sensitive to initial estimates and are not therefore robust to "severe uncertainty" (Sniedovich, 2007, 2008, 2010).
Hayes (2011, p. 88)

The first point that must be made abundantly clear is that, as far as Sniedovich is concerned, there is no "debate" whatsoever about these two claims. And if it appears that such a debate is on-going, then ... this is unfortunate, because as shown by Sniedovich (2007, 2008, 2010, 2011) there is nothing that is "debatable" about these two claims.

In other words, as I show in Sniedovich (2007, 2010, 2011), indeed in all my articles and lectures on this issue, the fact that info-gap's robustness model is a simple instance of Wald's famous Maximin model (circa 1940) is a formally proved mathematical fact. Furthermore, I am unaware of any argument showing that my proofs are invalid. So what exactly would the debate be about?

Similarly, I proved long ago that info-gap's robustness model is a model of local robustness. The proof implies that info-gap decision theory, by definition, does not, indeed cannot, seek decisions that are robust against severe uncertainty. Rather, all that info-gap decision theory can by definition do is seek decisions that are robust against small perturbations in a nominal value of a parameter of interest. And in this regard as well, I am unaware of any argument showing that my proofs are invalid. Moreover, the discussion in the REPORT supports my "claim". So what exactly would the debate be about?

Readers who are not familiar with my formal, rigorous debunking of info-gap decision theory, may want to examine the mobile on-line version thereof.

Remark
Info-gap enthusiasts who continue to hold that there is a "debate" on the two issues raised in the REPORT should read my recent articles (Sniedovich 2010, 2011) as well as my Official Debunker of info-gap decision theory.

I should point out that there is no real debate on these facts about info-gap decision theory in the info-gap literature. Indeed, as I indicate above, info-gap adherents do not formally refute my proofs (debunking info-gap decision theory). All one can find in the info-gap literature is an unsubstantiated, erroneous rhetoric about these facts and other issues (see Myths and Facts about info-gap decision theory).

So, incredible though it may sound, more than four years after I published these incontestable proofs substantiating my criticism of info-gap decision theory, info-gap adherents continue to cling to the info-gap rhetoric. Worse, they continue to circumvent this criticism by means of unsubstantiated, erroneous claims.

Hence the importance of this REPORT.

I hope that this REPORT will encourage analysts to open their eyes and ... minds to Hayes' (2011) explanation and illustration of the validity of my criticism of info-gap decision theory with regard to point b) in his REPORT. It is a pity, though, that such discussion is lacking on point a).

To sum up then, pace the REPORT, the following two facts are "undebatable":

And this leads me straight to the commentary in the REPORT on the concept severe uncertainty, notably to the fact that the term "severe uncertainty" is placed in the REPORT in inverted commas.

About Severe Uncertainty

Before I can turn to a formal discussion of the issue of "severe uncertainty", I want to call attention to a basic fact that some info-gap scholars prefer to conveniently overlook.

The point is this.

The subtitle of the two books on info-gap decision theory, namely Ben-Haim (2001, 2006), is

Decisions Under Severe Uncertainty

Indeed, one of the books' main theses is that info-gap's forte is that it provides a methodology for the management of a severe uncertainty that is really severe.

Readers who are familiar with Taleb's Black Swan theory, might be a bit skeptical about such claims and would therefore be inclined to ask:

How severe is the uncertainty that info-gap decision theory is capable of handling?

This, it is important to note, is a question of the first importance for info-gap decision theory, hence of immediate relevance to the REPORT. Yet, surprisingly, it is not addressed at all in the REPORT.

So I ask again:

How severe is the uncertainty that info-gap decision theory is capable of handling?

However, before I can take up this question, I want to clarify to all those who are not familiar with the info-gap literature, how this issue is treated by info-gap scholars.

The first thing to note is that an answer to the question: "how severe is the uncertainty handled by info-gap decision theory?" will differ depending on who you approach for an answer.

It is clear that some analysts hold that info-gap decision theory has the capabilities to handle a severe uncertainty that is far beyond the capabilities of other decision theories. For example, Wintle et al. (2010) (see Review 17) have recently proposed the use of info-gap decision theory as a means for dealing with Black Swans and Unknown Unknowns ! Yes, you have it right:

Unknown Unknowns

My commentary on this amazing proposition is outlined in Review 17.

In contrast, if you put it to analysts who, after years of having used info-gap decision theory for the declared purpose of managing "severe uncertainty", that info-gap's robustness model is a simple Radius of Stability model, hence a model of local robustness, hence unsuitable for severe uncertainty, the answer will be quite different. Conveniently overlooking that they have been using info-gap's robustness model to manage "severe uncertainty", hence as a model of global robustness, they will tell you that models of local robustness of the Radius of Stability type, for instance info-gap's robustness model, are suitable for handling situations where the uncertainty under consideration pertains to small perturbations in the nominal value of a parameter. And what is more, that this is the reason that they turn to info-gap decision theory, not for the purpose of determining global robustness against severe uncertainty.

My point is then that info-gap scholars are in the habit of tailoring their rhetoric according to the needs of the moment ...

And all this goes to show that as far as the "debate" is concerned, it is of the first importance to clarify this question:

How does info-gap decision theory define, or describe, or quantify, "severe uncertainty"?

Because, unless one gives a correct picture of info-gap's conception of "severe uncertainty", one would be unable to give a correct assessment/evaluation of info-gap decision theory. And what is more, for such an assessment to have any merit at all, it must be conducted, in the first instance, in terms of the uncertainty that is postulated by ... the theory itself. To be precise, it cannot be based on one's own understanding of this concept!

I should therefore make it clear that all my discussions on info-gap decision theory, including my criticism of it, are based solely on the meaning that info-gap decision theory --- as described in Ben-Haim (2001, 2006, 2010) --- ascribes to the term "severe uncertainty".

That said, I want to make it clear that the term "severe uncertainty" is not defined formally in the info-gap literature (Ben-Haim 2001, 2006, 2010). But, there is no room for any doubt as to how this concept is perceived in the theory, hence there is no doubt as to the meaning ascribed to it. Surprisingly, the REPORT does not mention this fact. What is more, its placing the term "severe uncertainty" in " " , apparently seems to suggest that the term "severe uncertainty" is problematic in that it is too nebulous to be pinned down.

Now, the reason that the REPORT suggests that an "ambiguity" surrounds the concept of "severe uncertainty" in info-gap decision theory, is not due to the fact that the concept itself is indeed hopelessly ambiguous, hence open to debate. The reason that the REPORT takes this line is entirely different. It simply reflects a position (conveniently) adopted by info-gap adherents who make such claims in an attempt to avoid dealing with the fundamental flaws afflicting info-gap decision theory.

To explain. Even a superficial scan of the uncertainty that info-gap decision theory claims to address, immediately reveals that the theory is ill-equipped to deal with this type of uncertainty. What is more, that it "deals" with the severity of the uncertainty by ... ignoring it. To reiterate, info-gap decision theory instructs "dealing" with the severity of the uncertainty that it claims to address by ignoring it altogether!

Indeed, this fact can be made clear to experts and novices alike by means of simple graphic example (see below).

So, as might be expected, info-gap scholars are extremely reluctant to discuss this rather problematic (embarrassing) aspect of the theory. Instead, they prefer to waltz around the issue claiming that the term "severe uncertainty" is ambiguous, hence unyielding to a formal discussion/analysis, hence open to "debate". This position, they seem to believe, enables them to avert straight answers to queries about the fact that info-gap decision theory is utterly unsuitable for the treatment of the uncertainty that it stipulates.

And when these scholars are reminded that info-gap's primary texts (e.g. Ben-Haim 2001, 2006, 2010) do indeed make it clear what the term "severe uncertainty" means (according to info-gap decision theory) they argue that this is not their "definition" of "severe uncertainty". In fact, some may even tell you that they do not use info-gap decision theory for the treatment of "severe uncertainty", but rather as a "sensitivity analysis" tool.

And, some info-gap scholars go so far as to contend that Sniedovich's definition of "severe uncertainty" is too restrictive, hence not really useful. For example consider this:

Sniedovich's definition of severe uncertainty is too narrow to be useful.
Burgman, M. (2008) Shakespeare, Wald and decision making under uncertainty.
Decision Point, 23, p. 10.
http://www.aeda.edu.au/docs/Newsletters/DPoint_23.pdf

It is important, therefore, that I make it crystal clear what Sniedovich's position is on this issue, because this has immediate implications for the REPORT's claims regarding the "on going debate" that Sniedovich is supposedly involved in.

Some Facts about Sniedovich
  1. In his many articles on info-gap decision theory, Sniedovich uses the term severe uncertainty to describe the uncertainty stipulated by info-gap decision theory (Ben-Haim 2001, 2006, 2010).

  2. Sniedovich's criticism of info-gap decision theory is based on the meaning that info-gap decision theory itself (Ben-Haim 2001, 2006, 2010) ascribes to the term severe uncertainty.

  3. In fact, Sniedovich does not have his own definition of severe uncertainty. Therefore, Burgman's (2008) proposition that Sniedovich's supposed definition of severe uncertainty has any bearing whatsoever on the treatment of severe uncertainty in info-gap decision theory is without any merit.

  4. Sniedovich refers all complaints about the meaning attributed to the term severe uncertainty in Ben-Haim (2001, 2006, 2010) to Ben-Haim (2001, 2006, 2010).

  5. Sniedovich's assessment/evaluation of info-gap decision theory is based on this theory being a decision theory, not a sensitivity analysis tool.

And after this somewhat lengthy interlude, I can turn to a more formal discussion of the question under consideration in this section:

How severe is the uncertainty that info-gap decision theory is capable of handling?

The situation is really quite simple. This is so because info-gap's uncertainty model itself is very ... simple. This model consists of two elements that are assumed to be known to the analyst at the outset:

This, obviously, means that info-gap decision theory is not designed to deal with Unknown Unknowns. To repeat: the basic assumption is that the analyst knows the values of U and û at the outset.

Note, however, that these two objects, on their own, do not indicate how "severe" the uncertainty in the true value of u is.

It is therefore important to take note that, according to Ben-Haim (2001, 2006, 2010), the severity of the uncertainty that info-gap decision theory deals with, or more accurately, is designed to deal with, is manifested in the following three properties, or working assumptions:

The third property requires further clarification.

This property implies that there are no grounds whatsoever to assume that the true value of u is more/less likely to be in the neighborhood of a given point in U, rather than in the neighborhood of any another point in U. Thus, there are no grounds whatsoever to assume that the true value of u is more/less likely to be in the neighborhood of the point estimate û rather than in the neighborhood of another point in U.

For example, suppose that the uncertainty space U is the real line, namely U=(-∞,∞), and that the point estimate is û = 0. Then, the third property implies, for instance, that there are no grounds to assume that the true value of u is more/less likely to be in the neighborhood of û = 0, rather than say in the neighborhood of u = 2054, or in the neighborhood of say u = -167.

Queries about this feature of info-gap decision theory, ought to be addressed to Ben-Haim (2001, 2006, 2011), not to Sniedovich.

All this goes to show that, stating formally, explaining and illustrating, how "severe uncertainty" is perceived by info-gap decision theory is simplicity itself. Still, to impress on all readers that info-gap's working assumptions about the severity of the uncertainty give the concept severe uncertainty a clear-cut meaning, I restate them using a slightly different color scheme:

As for the vastness of the uncertainty space U.

Take note that according to Ben-Haim (2001, 2006), most of the commonly encountered info-gap models are unbounded.

It is surprising, therefore, that this is not made clear in the REPORT, indeed that the term "severe uncertainty" is treated as nebulous, hence as open to interpretation, especially in the context of info-gap decision theory.

As a final note on the zigzags in the discourse on "severe uncertainty" in info-gap circles, I want to point out the following. The narrative on the "severe uncertainty" predicated by info-gap decision theory (Ben-Haim 2001, 2006, 2010) --- an uncertainty manifested in the three properties listed above --- has in fact lead applied ecologists to proclaim info-gap decision theory as capable of handling ... truly severe uncertainties. This is illustrated, for instance, in the following figure:

This is a reproduction of a figure by Halpern et al. (2006) showing the severity of the uncertainty that can be handled by various methods/approaches.

The figure speaks for itself.

I should stress that this figure is indeed a most accurate reflection of info-gap decision theory's position vis-a-vis other theories/methods/approaches, if you keep in mind what the theory (Ben-Haim 2001, 2006, 2010) promises its users.

One of the main goals of my campaign to contain the spread of info-gap decision theory in Australia was to enlighten users of info-gap decision theory how groundless these promises are.

And if you read between the lines of the REPORT, you'll reach the same conclusion.

Remark:
For the record, I should point out that info-gap decision theory also assumes that the uncertainty model specified by the pair (U,û) has an additional property. I did not touch on this property above because it has no bearing on the issue of the severity of the uncertainty.

Informally, this property entails that a "distance" structure is imposed on the uncertainty space U. In particular, the assumption is that a recipe is available for measuring the "distance" between points in U and the estimate û. So let,

dist(u,û):= distance of u∈U from û

where, formally, dist is a real-valued function on U2 such that dist(u,û) > 0 for all u ≠ û and dist(û,û) = 0.

This "distance" structure induces a neighborhood structure on the uncertainty space U. That is, using this "distance" structure, we can define

U(α,û):=neighborhood of radius α≥0 around û
={u∈U: dist(u,û)≤ α} , α≥0

Thus, U(α,û) denotes the subset of the uncertainty space U whose elements are within distance α from û. It follows then that, by definition, the neighborhoods are nested, namely α' < α" implies that U(α',û) ⊆ U(α'',û).

For the record, I should point out that info-gap decision theory does not refer to the "distance" function explicitly. It merely assumes the existence of a (nested) neighborhood structure around the point estimate û.

I should also point out that the concept "neighborhood" is a widely used concept in applied mathematics, engineering, economics, physics, and so on. Furthermore, that this concept is a key ingredient in the definitions of models of local robustness/stability, such as the famous Radius of Stability model:

ρ(q,û):= max {ρ≥0: constraints(q,u),∀u∈U(ρ,û)} , q∈Q

where constraints(q,u) denotes the list of constraints that defines the regions of stability of system/decision q, and U(ρ,û) denotes a neighborhood of radius ρ around û.

This reminder is not meant to question the reader's familiarity with the elementary mathematical concept "neighborhood". I call the reader's attention to this fact because some info-gap scholars apparently believe that this concept is a revolutionary innovation introduced by info-gap decision theory.

So for the record, it is important to make it clear that this is not so!

The picture on the right is taken from the article Neighborhood in WIKIPEDIA. It shows a neighborhood around a point p in a set V.

I also call the reader's attention to the fact that the neighborhood structure in info-gap decision theory is subject to the likelihood-free property. This means that the neighborhood structure does not quantify the likelihood of the elements of U.

In other words, the nesting property has no implications whatsoever as to the likelihood of elements of U. Thus, the fact that dist(u',û) < dist(u",û) implies nothing about the likelihood of u' and u" being the true value of u.

And one more thing.

Even prominent info-gap scholars sometimes confuse the "distance" structure induced by the neighborhoods U(α,û), α≥0, around the estimate û, with a (non-existing) likelihood structure on the uncertainty space U (see for example Review 6 and Review 15).


The Maximin connection

Since the REPORT makes do with a terse comment on the (in fact non-existing) "debate" about the info-gap --- Maximin connection, it is important to remind readers of the basic facts pertaining to this matter.

Info-gap decision theory is proclaimed to be a distinct new theory that is radically different from all current theories of decision under uncertainty (e.g. Ben-Haim 2001, 2006).

But to see how groundless this contention is, take a look at the Official Debunker.

Indeed, not only is this not the case, info-gap's robustness model is in fact a simple instance of the most well-known robustness model in all the disciplines dealing with robustness. That is, the purportedly new and radically different info-gap robustness model is no more than a simple instance of Wald's famous Maximin model (circa 1940).

As a matter of fact, the complete story is somewhat more involved and more ... embarrassing. Info-gap's robustness model is an instance of, or rather a re-invention of, a model of local robustness that is known universally as Radius of Stability (circa 1960) --- itself an instance of Wald's Maximin model. Of course, the Radius of Stability model has been used for decades, in a wide range of disciplines (e.g. numerical analysis, control theory, operations research, economics, applied mathematics) for the modeling and analysis of small perturbations in the nominal value of a parameter of interest.

This means that the claim that info-gap decision theory is new and radically different is not only groundless, it is a gross misrepresentation of the state of the art.

Take note that as a model of local robustness, the generic Radius of Stability model asks the following simple but extremely important practical and methodological question:

What is the size of the largest perturbation (in the worst-case sense) in the nominal value of a parameter, that does not destabilize a given system?

The "worst-case sense" clause refers to the "direction" of the perturbation (see explanation).

The point is then that info-gap's robustness model, hence info-gap decision theory, is a re-invented square wheel.

UncyclopediA

The object of this metaphor is to put across the prescribed misapplication of the Radius of Stability model by info-gap decision theory, which instructs using a model of local robustness to determine global robustness.

The unsavory consequences of this profoundly flawed prescription for identifying decisions that are robust to severe uncertainty are illustrated through the No Man's Land metaphor.

No Man's Land

By definition, info-gap's robustness model is a model of local robustness. That is, it measures the largest safe (worst case) deviation from the estimate. This means that the model seeks to identify the robustness of the problem concerned, in the neighborhood of the estimate. The robustness of decision q∈Q is defined formally in info-gap decision theory as follows:

α(q,û):= max {α≥0: rc ≤ r(q,u),∀u∈U(α,û)} , q∈Q

where rc ≤ r(q,u) is a performance constraint that decision q is required to satisfy for values of u in U.

By definition then:

info-gap robustness:
The robustness of decision q, denoted α(q,û), is the largest value of α such that the constraint rc ≤ r(q,u) is satisfied for all u∈U(α,û).

For the record, α(q,û) is the radius of stability of decision q at û as determined by this simple Radius of Stability model.

Thus, according to info-gap decision theory, an optimal decision is a decision whose robustness (radius of stability) is the largest.

The inherently local nature of info-gap's robustness model is manifested in its prescription to measure the performance of decision q with regard to the constraint rc ≤ r(q,u) only in the neighborhood of the estimate û.

But, as the underlying assumption is that the estimate is poor and can be substantially wrong, it is eminently clear that it is imperative to assess how different values of û affect the results of the analysis. Because, as one might well imagine, changing the value of the estimate changes the info-gap robustness of the decisions and accordingly the optimal decision.

So, the trouble is that by solving a problem for a number of values of û, one generates a number of optimal decisions, and the question then arising is: which of the optimal decisions is the "best"?

And to complicate matters even more, there is no apparent way out of this difficulty. Because, given that info-gap's uncertainty model is a likelihood-free model, assigning different "likelihoods" or "plausibilities" to different point estimates is not a proposition.

And yet, in spite of the vitally important ramifications that these issues have for info-gap's methodology, the theory does not even begin to address them.

I raised all these matters a long time ago on this website and in numerous articles and presentations. It is gratifying, therefore, that they are now discussed so clearly in the REPORT.

Still ...

Because one of the most serious flaws in info-gap's treatment of severe uncertainty is not discussed in the REPORT, it behooves me to raise it here. As might be expected, this serious flaw is also a function of info-gap's prescription to focus the entire robustness analysis only on a (poor) estimate and its immediate neighborhood.

And to do this, let us examine the following two positions on dealing with severe uncertainty:

Fundamental difficulty in
Decision-Making Under Severe Uncertainty Info-gap decision theory
  • The true value of the parameter of interest is subject to severe uncertainty.

  • Therefore, our point estimate of the true value can be substantially wrong.

  • Therefore, optimal decisions determined only on grounds of the given point estimate of the true value of the parameter, can equally be substantially wrong.

  • So, how do we deal with the difficulty that our point estimate can be substantially wrong?
  • The true value of the parameter of interest is subject to severe uncertainty.

  • Therefore, our point estimate of the true value can be substantially wrong.

  • Nevertheless, we determine the optimal decision solely on grounds of a given point estimate of the true value of the parameter and ... its neighborhood.

  • This is how we deal with the difficulty posed by our point estimate being substantially wrong!

This blunt juxtaposition brings out why info-gap decision theory fails to address the fundamental difficulty encountered in decision-making under severe uncertainty.

By restricting its local robustness analysis to the point estimate and its neighborhood, info-gap's robustness analysis totally ignores the performance of decisions outside this neighborhood. To see that this is so keep in mind that the severity of the uncertainty is manifested in a vast (indeed even unbounded) uncertainty space. So, if the neighborhood around the estimate, on which the analysis is conducted is small, namely α(q,û) is small, there are no grounds to assume that the robustness analysis will generate decisions that are robust over the uncertainty space U, even if such a decisions exist.

More generally, the difficulty is that a decision that is robust locally, namely in the neighborhood of the estimate, is not necessarily robust globally, namely over the uncertainty space as a whole, and vice versa. And a decision that is fragile in the neighborhood of the estimate is not necessarily fragile over the uncertainty space as a whole, and vice versa.

As I indicate above, it is unfortunate that the REPORT does not discuss this most obvious, most serious flaw in info-gap's robustness analysis, which I termed an "invariance" to the performance of decisions outside their largest "safe" neighborhood (Sniedovich 2007, 2010, 2011). I also describe this flaw by means of the "No Man's Land" metaphor (Sniedovich 2010, 2011). This metaphor brings out clearly and vividly why info-gap decision theory is a voodoo decision theory par excellence.

It goes without saying that one would not have expected the REPORT to use the term "voodoo decision theory" to describe the ramifications of the "invariance property". Still, it could have discussed this issue using the perfectly innocuous (politically correct) term "invariance property".

And for all those who are not familiar with my illustrations of info-gap decision theory's prescription for dealing with severe uncertainty of the type that it claims to address, consider the following picture:

No Man's LandûNo Man's Land
-∞ <-------------- Complete region of uncertainty under consideration --------------> ∞

where

 û   denotes the estimate of the parameter of interest.
    denotes the complete region of uncertainty under consideration.
    represents the region of uncertainty that effectivelly affects the results
  generated by info-gap's robustness analysis.
No Man's Land   represents that vast part of the complete region of uncertainty that has no
  impact whatsoever on the results generated by info-gap's robustness model.

What this picture makes vivid is that info-gap decision theory's central proposition to use a local Radius of Stability type analysis as a means for determining the robustness of situations that are subject to severe uncertainty effectively means that it prescribes the following:

And this, as is my want to emphasize, is in violation of universally accepted maxims such as:

All that's left to say then is that it beggars belief that risk analysts take such a theory seriously!

And there is more.

Voodoo decision-making

I take this opportunity to explain, once again, my contention that info-gap decision theory prescribes voodoo decision making. In a nutshell, what I mean by this is that info-gap decision theory prescribes using a model of local robustness, namely a Radius of Stability type model, in the management of a severe uncertainty that is characterized by a vast (unbounded) uncertainty space, a poor estimate, and a likelihood-free quantified uncertainty. And to appreciate my point, consider the following argument.

Over the past fifty years, an extremely large number of models has been developed, under the overarching title Radius of Stability, in various areas of expertise. The express objective of these models is the modeling and analysis of problems of local stability/robustness where the idea is to determine robustness against small perturbations in a nominal value of a parameter.

In other words, the goal of such models is to identify the largest perturbation (in the worst-case "direction") in a parameter of interest that a system can cope with without being destabilized. Thus, a system's Radius of Stability is the size of the smallest perturbation that can destabilize the system. Equivalently, it is the radius of the largest neighborhood around the nominal value of the parameter that is contained in the system's regions of stability. This concept is illustrated in the following picture, where û denotes the nominal value of the parameter and U denotes the set of all possible/plausible values of the parameter of interest.

The radius of stability of the system is the radius of the thick (blue) circle. Any larger circle centered at û will contain values of the parameter that are unstable.

What this picture makes vivid is that the concept "Radius of Stability" is a concept of local stability/robustness meaning that it is not designed to be used as a measure of the global stability/robustness of a system. In other words, the picture makes clear that the neighborhood represented by the radius of stability can hardly be an indication of the size/shape of the system's region of stability represented by the shaded area!

But, in 2001, info-gap decision theory burst unto the scene proposing to the world the following:

I remind readers that the proofs showing that these claims are without any foundation are detailed in Official Debunker of info-gap decision theory.

Here, I want to comment on the position taken by certain info-gap enthusiasts, who, although now accept my basic criticism of info-gap decision theory, continue to insist that the flaws afflicting info-gap decision theory are not that serious to warrant the title voodoo decision theory par excellence.

So let me explain, once again.

Consider the following two hypothetical assessments of info-gap decision theory:

Assessment A Assessment B
Info-gap's robustness model is a Radius of Stability model, hence it is a model of local robustness, hence methodologically speaking, it is unsuitable for the treatment of severe uncertainty. Nevertheless, I use it in this project because the estimate I have is pretty good and the uncertainty space is relatively small. Info-gap decision theory is particularly well suited for situations where the uncertainty space is unbounded, the estimate is poor and the quantification of the uncertainty is likelihood-free. It thus provides a reliable tool for the management of surprises, rare events, shocks, catastrophes, and so on.

Here is my response to these assessments:

Moshe's response to Assessment A Moshe's response to Assessment B
No worries!

But, take note that according to Ben-Haim's (2007) compilation of FAQs about info-gap decision theory:

1. Info-gap theory is useful precisely in those situations where our best models and data are highly uncertain, especially when the horizon of uncertainty is unknown. In contrast, if we have good understanding of the system then we don’t need info-gap theory, and can use probability theory or even completely deterministic models. It is when we face severe Knightian uncertainty that we need info-gap theory.
Ben-Haim (2007, p. 2)

In other words, if the uncertainty is not severe, why use info-gap decision theory?

 No worries!

But note that this amounts to voodoo decision-making.

The point I want to emphasize in Assessment A is twofold. First, I want to expose the profound inconsistencies in the positions and rhetoric of info-gap adherents. Keep in mind that info-gap's robustness model (Ben-Haim 20001, 2006, 2010) was not designed to serve as a means for sensitivity analysis. Its express objective (Ben-Haim 20001, 2006, 2010) is robust decision under severe uncertainty. Second, I want to emphasize that in spite of this fact, there in not a single reference in (Ben-Haim 20001, 2006, 2010) to the field of Robust Optimization. It is important to point out, therefore, that this thriving area of expertise offers a vast knowledge-base that bears directly on the issues that info-gap decision theory claims to be concerned with namely: robust decision-making in the face of severe uncertainty.

In fact, from the perspective of Robust Optimization, info-gap's robustness model is no more than an elementary robust optimization model.

Regarding Assessment B

This is essentially the assessment of info-gap decision theory given in the info-gap literature (e.g. Ben-Haim 2001, 2006, 2010). To illustrate, consider the opening paragraph in the Preface to Info-Gap Economics: An Operational Introduction (Ben-Haim 2010, p. 10):

The management of surprises is central to the "economic problem", and info-gap theory is a response to this challenge. This book is about how to formulate and evaluate economic decisions under severe uncertainty. The book demonstrates, through numerous examples, the info-gap methodology for reliably managing uncertainty in economic policy analysis and decision making.

Read my short review of this book.

The impression given by this and similar statements in the info-gap literature is that info-gap decision theory offers precisely the methodology for cases where the uncertainty is truly severe.

Such statement are underscored by assertions (Ben-Haim (2001, 2006) that (emphasis added):

Most of the commonly encountered info-gap models are unbounded

To reiterate then, this is my response in a nutshell:

Moshe's response

A theory deploying a Radius of Stability model to tackle the difficult task posed by a severe uncertainty of the type that is most commonly encountered in info-gap models(see fine print), is a voodoo decision theory par excellence.
----------------------
Fine Print: The point estimate is poor, the uncertainty space is unbounded and the quantification of uncertainty is likelihood-free.

Take note that the fine print is crucial here.

The fundamental flaw

The discussion in the REPORT (page 91-92) on the difficulty arising as a result of the true value of the parameter of interest being subject to severe uncertainty makes a valid argument. The difficulty, as the REPORT points out, is that different values of the estimate may yield different optimal decision. This means that a mechanism is required for determining which optimal decision generated by varying the value of the estimate, is best overall.

The REPORT also points out, correctly, that info-gap decision theory does not provide such a mechanism so that it " ... may not be able to unambiguously identify the best course of action. ..."

My position on this "difficulty" is this:

As I explain above in connection with Assessment B, info-gap decision theory's inability to deal properly with the issues arising from the estimate being poor, is not a technical difficulty that one can so to speak live with! One cannot plead "attenuating circumstances" in this case, arguing: "oh well, so info-gap decision theory is not perfect, and like other methods it has its faults. Surely, other theories also have similar faults".

In the case of info-gap decision theory, such a plea is inadmissible because this difficulty is due to a fundamental flaw in its methodology --- a flaw that undermines precisely that which this theory claims to provide: a reliable means for addressing the challenging task of dealing with severe uncertainty.

The reason that this flaw is fundamental is that it cannot be corrected or mitigated. It is endemic in what info-gap decision theory prescribes doing: apply a simple model of local robustness such as the Radius of Stability model, in the search for a robust decision under conditions of severe uncertainty. Differently put, the flaw is fundamental because it prescribes applying a model that by definition lacks the capabilities that are required of a model to identify robust decisions under conditions of severe uncertainty. In a word, it prescribes using a model of local robustness to perform a job that only a model of global robustness can carry out.

I want to note that I fully understand and appreciate that an official CSIRO report may have to avoid using confronting (harsh?) language. But the fact of the matter is that such language is unavoidable if one is to make clear the hard facts about this theory. One has to use the right arguments so as to be able to call a spade a spade.

Thus, the trouble in info-gap decision theory is not that it " ... may not be able to unambiguously identify the best course of action ..." due to the need to consider multiple point estimates, hence a number of (local) optimal decisions. Rather, the fundamental flaw in info-gap decision theory is that it fails to distinguish between local robustness and robustness to severe uncertainty of the type it attempts to address.

In more "diplomatic" language. The flaw in info-gap decision theory is that it is "oblivious" to the very difficulties posed by the severe uncertainty that it seeks to manage. It erroneously takes robustness against severe uncertainty as equivalent to, or as one that can be reliably approximated by, robustness against small perturbations in the value of a poor point estimate of the parameter of interest. And the irony is that the purportedly new, revolutionary, model proposed by the theory for this purpose is a re-invention the well-known and well-established the Radius of Stability model (circa 1960), itself a simple instance of Wald's famous Maximin model (circa 1940).

This is why I claim that info-gap decision theory grossly misrepresents the state of the art.

Other issues

A number of other issues that are discussed in the REPORT require attention. I particularly want to call attention to a number of statements in this paragraph:

Info-Gap’s most important property is its definition of the best decision as that which is most immune to the uncertainty in the decision maker’s model of the world, represented by the reward function. This is an important counterpoint to optimality-based approaches that identify the best decision as that which maximises the reward function. The word “optimal” implies that the decision could be no better but this is conditional on the reward function (model) and may not be true for variations of this function. Info-gap theory places the uncertainty in the reward function at the center of the decision-making process and encourages decision makers to maximise immunity to uncertainty rather than maximising reward. This is a very sensible approach, and one that recognises that there is no such thing as an "optimal” decision in a non-deterministic decision-making process.
Hayes (2011, p. 89, emphasis added)

At this stage I only point out that the conclusion, shown in boldface, is clearly in error. Because, as attested by the first sentence in this paragraph, according to info-gap decision theory itself, an optimal decision (= best decision) is one that maximizes a certain objective function subject to certain constraints. It is important to take note then that info-gap's generic decision model is an optimization model par excellence. It has the following form:

α(r*):=   max   max {α≥0: r* ≤ R(q,u),∀u∈U(α,û)}
q∈Q
=  max {α≥0: q∈Q, r* ≤ R(q,u),∀u∈U(α,û)}

observing that there are two decision valriables in this optimization model, namely q∈Q and α≥0. Formally, the objective function f=f(q,α) is a real valued function of these two variables such that f(q,α)≡ α. The robustness constraint on these two decision variables is r* ≤ R(q,u),∀u∈U(α,û).

This is a simple instance of the generic robust optimization model:

z*:=   max   g(x) subject to constraint(x,u),∀u∈N(x)
  x∈X

Here N(x) denotes a set that may depend on the decision variable x, and constraints(x,u) denotes a list of robustness constraints on the decision variable x and the uncertainty parameter u.

In short, info-gap's decision model is a simple optimization model, so that contrary to the above conclusion, there is definitely such a thing as `... an "optimal” decision in a non-deterministic decision-making process ...'

I shall elaborate on this and other issues in due course. For the time being it is important to emphasize that there definitely is such a thing as an "optimal" decision in a non-deterministic decision-making process. There is a huge literature on this topic (see for instance stochastic optimization, Stochastic_programming, and robust optimization.

For the benefit of some readers, here is the opening paragraph in the entry Stochastic Programmming in WIKIPIDIA (July 15, 2011):

Stochastic programming is a framework for modeling optimization problems that involve uncertainty. Whereas deterministic optimization problems are formulated with known parameters, real world problems almost invariably include some unknown parameters. When the parameters are known only within certain bounds, one approach to tackling such problems is called robust optimization. Here the goal is to find a solution which is feasible for all such data and optimal in some sense. Stochastic programming models are similar in style but take advantage of the fact that probability distributions governing the data are known or can be estimated. The goal here is to find some policy that is feasible for all (or almost all) the possible data instances and maximizes the expectation of some function of the decisions and the random variables. More generally, such models are formulated, solved analytically or numerically, and analyzed in order to provide useful information to a decision-maker.

I suspect that the author of the REPORT meant to say something entirely different on this subject, and I trust that this paragraph will eventually be corrected. This is crucial because .... there are other errors in it. For instance, consider the assertion:

This is an important counterpoint to optimality-based approaches that identify the best decision as that which maximises the reward function.

The fact of the matter is, of course, that info-gap's robustness model and its decision model are no more and no less than elementary ... robust optimization models. This means that the info-gap methodology does precisely that which robust optimization prescribes doing. The inclusion of constraints in the formulation of the optimization problem in question does not make info-gap's optimization model any different from other standard robust optimization models. It does precisely that which standard robust optimization models do as a matter of course. So, far from offering an "... important counterpoint ...", all that info-gap decision theory can offer are extremely simple (unsophisticated) robustness and decision models that fall under the umbrella of... robust optimization.

The reason that serious misconceptions about the issues touched on in this paragraph abound in the info-gap literature, is in the first place due to the misleading statements on them in (e.g Ben-Haim 2001, 2006, 2010). It is also due to the lack of all references in the info-gap literature (e.g Ben-Haim 2001, 2006, 2010) to the field of robust optimization. What is the wonder then that readers of info-gap publications and info-gap users have no clue about info-gap's relation to optimization theory?!

Stay tuned.

Summary and Conclusions

Hayes's (2011) CSIRO report is definitely a step in the right direction. I hope that it will be followed by more in-depth re-assessments of info-gap decision theory and the misconceptions that it continues to advocate.

The REPORT indicates, albeit not sufficiently clearly, that info-gap decision theory is unsuitable for the treatment of a severe uncertainty of the type that it is supposed to deal with, and that it deals with this severe uncertainty by ... ignoring the severity of the uncertainty.

The REPORT does not indicate that the main texts on the theory (Ben-Haim 2001, 2006, 2010) give a thoroughly distorted picture of the state of the art, due to their lack of reference to the field of Robust Optimization.

Indeed, the REPORT upholds a myth that continues to be propounded in the info-gap literature. Namely, that info-gap decision theory offers a methodology that is radically different from those offered by optimization theory. This myth argues that info-gap's robustness model seeks to "satisfice a constraint" rather than "optimize reward" and what is more, that under severe uncertainty, "satisficing" has a clear advantage over "optimizing". The fact of the matter is of course that info-gap's robustness model is simply an elementary robust optimization model of the Maximin type. This means that info-gap's robustness model does precisely that which optimization theory has been doing for ages: it maximizes a function subject to constraints!

The REPORT should have pointed out to info-gap users who claim to be using the theory for the purposes of a local sensitivity analysis that info-gap's robustness model was not designed to serve this purpose. But what is more, that info-gap's robustness model is a simple Radius of Stability model (circa 1960). Also, the REPORT should have advised info-gap users, seeking to perform a local sensitivity analysis, that Radius of Stability models have been used for decades, in many disciplines, precisely for the purpose of modeling and analyzing small perturbations in the nominal value of the parameter. This means that by clinging to info-gap decision theory, info-gap users simply cut themselves off a huge body of knowledge and expertise (models, algorithms, software packages) that has accrued over the past fifty years!

As for info-gap's handling of the Radius of Stability model, I want to emphasize again that info-gap decision theory in fact prescribes a misapplication of the Radius of Stability model. That is to say, info-gap decision theory instructs using a Radius of Stability type model (info-gap's robustness model) to identify decisions that are robust to severe uncertainty. Namely, it (mis)applies this model to identify global robustness. Thus, to the best of my knowledge, info-gap decision theory is the only theory in the trade to advocate the absurd proposition that the local Radius of Stability model (circa 1960) is a reliable tool for the management of a severe uncertainty that is characterized by a poor estimate, a vast uncertainty space, and a likelihood-free quantification of uncertainty.

And this leads me to renew my call for a reassessment of info-Gap decision theory.

Second Call for the Reassessment of Info-Gap Decision Theory

I remind info-gap scholars in Australia of my 2008 Call for The Reassessment of The Use and Promotion of Info-Gap Decision Theory in Australia. See also my short article on this matter in Decision Point (Issue 24, 2008).

Given the Info-gap Experience in Australia (2003-2011), it is important to identify and elucidate the fundamental flaws afflicting this theory, and to make clear why these flaws render it utterly unsuitable for the treatment of a severe uncertainty of the type that it stipulates. It is equally important to understand how such a flawed theory managed to attract such a number of followers, and why it managed to pass muster in the peer-review process of so many journals, especially in applied ecology and conservation biology.

Hence,

A Second Call for The Reassessment of The Use and Promotion of Info-Gap Decision Theory in Australia

is now in order!

Stay tuned.

Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.


Last modified: