A week of links

Links this week:

  1. I worry that most smart people have not learned that a list of dozens of studies, several meta-analyses, hundreds of experts, and expert surveys showing almost all academics support your thesis – can still be bullshit.” Awesome.
  2. I have only just realised that Gary Klein blogs at Psychology Today. A relatively recent post – The Insight Test.
  3. Bad statistics – same sex marriage edition.
  4. How to doctor a cost-benefit analysis.
  5. Replicating epigenetic claims.

Complexity versus chaos

Another clip from David Colander and Roland Kupers’s Complexity and the Art of Public Policy: Solving Society’s Problems from the Bottom Up – a nice description of how two often confused terms, complexity and chaos, differ and interrelate:

Chaos theory is a field of applied mathematics whose roots date back to the nineteenth century, to French mathematician Henri Poincaré. Poincaré was a prolific scientist and philosopher who contributed to an extraordinary range of disciplines; among his many accomplishments is Poincaré’s conjecture that deals with a famous problem in physics first formulated by Newton in the eighteenth century: the three body problem. The goal is to calculate the trajectories of three bodies, planets for example, which interact through gravity. Although the problem is seemingly simple, it turns out that the paths of the bodies are extraordinarily difficult to calculate and highly sensitive to the initial conditions.

One of the contributions of chaos theory is demonstrating that many dynamical systems are highly sensitive to initial conditions. The behavior is sometimes referred to as the butterfly effect. This refers to the idea that a butterfly flapping its wings in Brazil might precipitate a tornado in Texas. This evocative—if unrealistic—image conveys the notion that small differences in the initial conditions can lead to a wide range of outcomes.

Sensitivity to initial conditions has a number of implications for thinking about policy in such systems. For one, such an effect makes forecasting difficult, if not impossible, as you can’t link cause and effect. For another it means that it will be very hard to backward engineer the system—understanding it precisely from its attributes because only a set of precise attributes would actually lead to the result. How much time is spent on debating the cause of a social situation, when the answer might be that it simply is, for all practical purposes, unknowable? These systems are still deterministic in the sense that they can be in principle specified by a set of equations, but one cannot rely on solving those equations to understand what the system will do. This is known as deterministic chaos, but is mostly just called chaos.

While chaos theory is not complexity theory, it is closely related. It was in chaos theory where some of the analytic tools used in complexity science were first explored. Chaos theory is concerned with the special case of complex systems, where the emergent state of the system has no order whatsoever—and is literally chaotic. Imagine birds on the power line being disrupted by a loud noise and fluttering off in all directions. You can think of a system as being in these three different kinds of states, linear, complex, or chaotic—sitting on the line, flying in formation, or scrambling in all directions.

Like chaos theory, complexity theory is about nonlinear dynamical systems, but instead of looking at nonlinear systems that become chaotic, it focuses on a subset of nonlinear systems that somehow transition spontaneously into an ordered state. So order comes out of what should be chaos. The complexity vision is that these systems represent many of the ordered states that we observe—they have no controller and are describable not by mechanical metaphors but rather by evolutionary metaphors. This vision is central to complexity science and complexity policy.

I’ll post a full review next week.

More praise of mathematics

Following my post last week on the need for more complicated models in economics, a new paper in PLOS Biology argues for the importance of mathematical models in showing ‘proof of concept’ (HT: Santa Fe Institute News). The authors write:

Proof-of-concept models, used in many fields, test the validity of verbal chains of logic by laying out the specific assumptions mathematically. The results that follow from these assumptions emerge through the principles of mathematics, which reduces the possibility of logical errors at this step of the process. The appropriateness of the assumptions is critical, but once they are established, the mathematical analysis provides a precise mapping to their consequences.

They point to lack of trust many people have in mathematical models, but argue that once the theoretician fulfils their duty of making the robustness of the assumptions transparent, readers should take the results seriously.

Much of the doubt about the applicability of models may stem from a mistrust of the effects of logistical assumptions. It is the responsibility of the theoretician to make his or her knowledge of the robustness of these assumptions transparent to the reader; it may not always be obvious which assumptions are critical versus logistical, and whether the effects of the latter are known. It is likewise the responsibility of the empirically-minded reader to approach models with the same open mind that he or she would an experiment in an artificial setting, rather than immediately dismiss them because of the presence of logistical assumptions.

Several examples are provided in the paper, but my favourite example of models as ‘proof of concept’ relates to the handicap principle. I have posted about this model before (that time in the context of economists solving the problem 17 years before the biologists figured it out), so I will use some of my previous words.

[In 1975], Amotz Zahavi had a paper published titled Mate selection – a selection for a handicap. This paper spelt out Zahavi’s handicap principle, which described how honest signals of quality between animals could evolve. The signals are honest because they impose a handicap on the signaller that only a high quality signaller can bear.

The handicap principle was not accepted at first. Richard Dawkins wrote in an early edition of The Selfish Gene:

I do not believe this theory, although I am not quite so confident in my scepticism as I was when I first heard it. I pointed out then that the logical conclusion to it should be the evolution of males with only one leg and only one eye. Zahavi, who comes from Israel, instantly retorted: ‘Some of our best generals have only one eye!’ Nevertheless, the problem remains that the handicap theory seems to contain a basic contradiction. If the handicap is a genuine one-and it is of the essence of the theory that it has to be a genuine one-then the handicap itself will penalize the offspring just as surely as it may attract females. It is, in any case, important that the handicap must not be passed on to daughters.

John Maynard Smith published papers (such as this) suggesting that no model could be found in which the handicap principle could hold (although he did not rule out someone else finding one).

Finally, in 1990, Alan Grafen published two papers in which he established the population genetic and game theoretic foundations to the handicap principle. Mathematically, it could work. It convinced people such as Dawkins that the handicap principle could be right. … While Grafen’s papers are quite technical, the following diagram by Rufus Johnstone provides a simple illustration of how it works – and how similar it is to the work of Michal Spence. If two different quality individuals face differential costs and the same benefits (or differential benefits and the same costs), they will signal at different levels, making their signal a reliable indicator of their quality. The high-quality individual maximises costs relative to benefits at shigh, while the low-quality individual maximises their benefits relative to costs at slow.

Johnstone (2005)

I like this example for two reasons. First, a mathematical model effectively settled a dispute in biology. Most biologists now agree the evolution of handicaps as signals is plausible – it is now a question of how prevalent. But second, once the complicated model was developed, a quick intuitive mathematical explanation that is relatively easy to convert back into English followed.

My year

In the day job, for most of this year I was seconded onto the Australian Government’s Financial System Inquiry. The Inquiry was established to provide a broad review of the Australian financial system, looking at system stability, competition, consumer protection, technological change and whether the system was serving the needs of users.

The Inquiry’s final report is now out and available here. It has received a lot of press here – I think my favourite article so far is this one (if you hit the paywall, google “David Murray has gone rogue” and try that link).

Among other things, there are recommendations to increase bank capitalisation, introduce new obligations on financial product issuer and distributors, and to hold a review into the ownership and use of customers’ financial data. But given my role in the Inquiry and the stage the Government is at – it is now seeking public comment – it’s not really appropriate for me to say which recommendations I support.

Possibly the most interesting recommendations are in the retirement income space. Australia has a compulsory superannuation system, where (currently) 9.5 per cent of our income is required to go into retirement savings. But after a lifetime of being forced to save, once we reach what is called the “preservation age”, you can take the money out. You are free to blow it on a holiday, sportcars or pension means-test exempt house, and then receive the pension.

To try to change this behaviour, the Inquiry recommended introduction of a default retirement product, which will have some mix of income flow and longevity insurance (so your money doesn’t run out before you die). It will be an interesting exercise to design a system where that default will be an successful anchor. It will require a lot of tax, pension and other social policy settings to stop people from ignoring that default and taking their lump of cash in another way.

The other big event of the year was the arrival of twin boys. We think they are identical – four months of confusing who is who is the basis for that – but DNA tests are on the way to confirm. And I’ll be keeping one locked in the cupboard for the next five years to prove to the genetic determinists that environment does matter.

We need more complicated mathematical models in economics

I am half way through David Colander and Roland Kupers’s book Complexity and the Art of Public Policy: Solving Society’s Problems from the Bottom Up. Overall, it’s a good book, although the authors are somewhat slow to get to the point and there are plenty of lines that perplex or annoy (Arnold Kling seemed to have a similar reaction).

I’ll review later, but one interesting line in the book is that under a complexity approach, you may need more complicated mathematical models than used in neoclassical economics. This is because the purpose of the models under a complexity approach is different. They write:

A person is walking home late one night and notices an economist searching under a lamppost for his keys. The person stops to help. After searching a while without luck he asks the economist where he lost his keys. The economist points far off into the dark abyss. The person asks, incredulously, “Then why the heck are you searching here?” To which the economist responds—“This is where the light is.”

Critics of economists like this joke because it nicely captures economic theorists’ tendency to be, what critics consider, overly mathematical and technical in their research. Superficially, searching where the light is (letting available analytic technology guide one’s technical research) is clearly a stupid strategy; the obvious place to search is where you lost the keys.

Telling old jokes doesn’t do much, and in this case the joke was a setup for a different punch line. That punch line is that the critic’s lesson taken from the joke is the wrong lesson if the economy is complex. For a complex system, which the social system is, a “searching where the light is” strategy makes good sense. Since the subject matter of social science is highly complex—arguably far more complex than the subject matter of most natural sciences—it is as if the social science policy keys are lost in the equivalent of almost total darkness. The problem is that you have no idea where in the darkness you lost them, so it would be pretty stupid to just go out searching in the dark. The chances of getting totally lost are almost 100 percent. In such a situation, where else but in the light can you reasonably search in a scientific way?

What is stupid, however, is if the scientist thinks he or she is going to find the keys under the lamppost.

The fact that decisions in complex systems are so uncertain and difficult to make does not mean that one should avoid dealing with them mathematically and scientifically. Quite the contrary; it allows for much more complicated mathematical models since the models are used for a different purpose. Returning to our economist joke in the first chapter, they aim not to precisely describe the real world, but to understand the topography of the landscape under the light. The mathematical models are trying to map different types of topography, which may be helpful when searching for the policy keys, but they do not represent the full search for the keys.

The policy answers can be found only by those searching in the dark, which involves dealing with the full complexity of the system. The fact that one is using the models primarily for guidance, rather than for prescriptions, frees one from forcing the models to have direct policy relevance, which, as we will discuss, is a major reason for the problems with existing economic models. Instead one can use higher-level mathematics that is up to the task. In technical terms, instead of using static equilibrium models that can be analytically solved, one is free to use nonlinear, dynamic models that are beyond analytic solution, but upon which computational tools can shed light. As we will discuss in later chapters, the mathematics of complex evolving systems is really hard and still developing. That is why in the past economists and other social scientists have avoided them. It’s also why their policy advice has not been especially useful when the solution required a comprehensive understanding of our complex evolving socioeconomic system.

[T]he criticism coming from complexity scientists was different from that of most heterodox economists. The usual heterodox criticism of standard economics was that it was too mathematical. This was not the criticism here. Complexity scientists were arguing that economics was not mathematical enough—not only was it not mathematical enough, it was using the wrong mathematics. They agreed that if it was to be science, it had to be “under analytical control.” But they were arguing that by using the right mathematics, highly complex systems containing high levels of agent interdependence could come under analytic or computational scrutiny. Complexity scientists argued that economists needed to start exploring nonlinear dynamic models, path-dependent models by using the mathematics and tools of complexity science

They also give a word of caution as to where the science is at:

Every period has its excesses: the current hype about the usefulness of formal models in complexity science holds echoes of the overconfidence in models that one saw from the 1930s onward. The time when models will provide complete answers to social policy questions, if such a time ever will exist, is still far in the future. Complexity models, like all models, are very useful and necessary, but they are not sufficient.

A week of links

Links this week:

  1. The case against early cancer detection. The charts on mammogram and PSA testing effectiveness are just as Gerd Gigerenzer would have us present the statistics.
  2. The case for business experimentation.
  3. Airline inequality.
  4. While arguments continue about the predictive power of genetic testing, entrepreneurs are already using it. HT: Steve Hsu
  5. However, there are still plenty of average ‘gene for’ studies being produced. HT: Tim Frayling

The unrealistic assumptions of biology

Biologists are usually among the first to tell me that economists rely on unrealistic assumptions about human decision making. They laugh at the idea that people are rational optimisers who care only about maximising consumption.

Some of the points are undoubtedly correct. Humans do not care primarily about consumption. They seek mates or other objectives related to their fitness. And of course, humans do not solve complex optimisation problems with constraints in their heads.

But, as most economists will tell you, the assumptions of rationality and consumption maximisation are mechanisms to derive general predictions about behaviour. And the funny thing is, biologists often do the same. Biologists tend to treat their subjects as optimisers.

That places biologists in a similar position to economists. Biologists may be able to predict or explain behaviour, but often they have not actually explained how their subjects make decisions. If they were to attempt to predict how their subjects would behave in a changed environment – which is the type of predictive task many economists attempt to do – they would likely fail as their understanding of the decision making process is limited.

In Rationality for Mortals: How People Cope with Uncertainty, Gerd Gigerenzer has a great chapter considering how biologists treat decision making, and in particular, to what extent biologists consider that animals use simple decision-making tools such as heuristics (I’ve written two other posts on parts of the book here and here). Gigerenzer provides a few examples where biologists have examined heuristics, but much of the chapter asks whether biologists are missing something with their typical approach.

As a start, Gigerenzer notes that biologists are seeking to make predictions rather than accurate descriptions of decision making. However, Gigerenzer questions whether this “gambit” is successful.

Behavioral ecologists do believe that animals are using simple rules of thumb that achieve only an approximation of the optimal policy, but most often rules of thumb are not their interest. Nevertheless, it could be that the limitations of such rules of thumb would often constrain behavior enough to interfere with the fit with predictions. The optimality modeler’s gambit is that evolved rules of thumb can mimic optimal behavior well enough not to disrupt the fit by much, so that they can be left as a black box. It turns out that the power of natural selection is such that the gambit usually works to the level of accuracy that satisfies behavioral ecologists. Given that their models are often deliberately schematic, behavioral ecologists are usually satisfied that they understand the selective value of a behavior if they successfully predict merely the rough qualitative form of the policy or of the resultant patterns of behavior.

You could write the same paragraph about economists, minus the statement about natural selection. That said, if you were to give the people in an economic model objectives shaped by evolution, even that statement might hold.

But Gigerenzer has another issue with the optimisation approach in biology. As from most analysis of human decision making, “missing from biology is the idea that simple heuristics may be superior to more complex methods, not just a necessary evil because of the simplicity of animal nervous systems.” Gigerenzer writes:

There are a number of situations where the optimal solution to a real-world problem cannot be determined. One problem is computational intractability, such as the notorious traveling salesman problem (Lawler et al., 1985). Another problem is if there are multiple criteria to optimize and we do not know the appropriate way to convert them into a common currency (such as fitness). Thirdly, in many real-world problems it is impossible to put probabilities on the various possible outcomes or even to recognize what all those outcomes might be. Think about optimizing the choice of a partner who will bear you many children; it is uncertain what partners are available, whether each one would be faithful, how long each will live, etc. This is true about many animal decisions too, of course, and biologists do not imagine their animals even attempting such optimality calculations.

Instead the behavioral ecologist’s solution is to find optima in deliberately simplified model environments. We note that this introduces much scope for misunderstanding, inconsistency, and loose thinking over whether “optimal policy” refers to a claim of optimality in the real world or just in a model. Calculating the optima even in the simplified model environments may still be beyond the capabilities of an animal, but the hope is that the optimal policy that emerges from the calculations may be generated instead, to a lesser level of accuracy, by a rule that is simple enough for an animal to follow. The animal might be hardwired with such a rule following its evolution through natural selection, or the animal might learn it through trial and error. There remains an interesting logical gap in the procedure: There is no guarantee that optimal solutions to simplified model environments will be good solutions to the original complex environments. The biologist might reply that often this does turn out to be the case; otherwise natural selection would not have allowed the good fit between the predictions and observations. Success with this approach undoubtedly depends on the modeler’s skill in simplifying the environment in a way that fairly represents the information available to the animal.

Again, Gigerenzer could equally be writing about economics. I think we should be thankful, however, that biologists don’t take their results and develop policy prescriptions on how to get the animals to behave in ways we believe they should.

One interesting question Gigerenzer asks is whether humans and animals use similar heuristics. Consideration of this question might uncover evidence of the parallel evolution of heuristics in other lineages facing similar environmental structures, or even indicate a common evolutionary history. This could form part of the evidence as to whether these human heuristics are evolved adaptations.

But are animals more likely to use heuristics than humans? Gigerenzer suggests the answer is not clear:

It is tempting to propose that since other animals have simpler brains than humans they are more likely to use simple heuristics. But a contrary argument is that humans are much more generalist than most animals and that animals may be able to devote more cognitive resources to tasks of particular importance. For instance, the memory capabilities of small food-storing birds seem astounding by the standards of how we expect ourselves to perform at the same task. Some better-examined biological examples suggest unexpected complexity. For instance, pigeons seem able to use a surprising diversity of methods to navigate, especially considering that they are not long-distance migrants. The greater specialism of other animals may also mean that the environments they deal with are more predictable and thus that the robustness of simple heuristics may not be such as advantage.

Another interesting question is whether animals are also predisposed to the “biases” of humans. Is it possible that “animals in their natural environments do not commit various fallacies because they do not need to generalize their rules of thumb to novel circumstances.” The equivalent for humans is mismatch theory, which proposes that a lot of modern behaviour (and likely the “biases” we exhibit) is due to a mismatch between the environment in which our decision making tools evolved and the environments we exercise them in today.

Finally, last year I wrote about why economics is not more “evolutionary”. Part of the answer there reflects a similar pattern to the above – biologists aren’t that evolutionary either.

The power of heuristics

Gerd Gigerenzer is a strong advocate of the idea that simple heuristics can make us smart. We don’t need complex models of the world to make good decisions.

The classic example is the gaze heuristic. Rather than solving a complex equation to catch a ball, which requires us to know the ball’s speed and trajectory and the effect of the wind, a catcher can simply run to keep the ball at a constant angle in the air, leading them to the point where it will land.

Gigerenzer’s faith in heuristics is often taken to be based on the idea that people have limited processing capacity and are unable to solve the complex optimisation problems that would be needed in the absence of these rules. However, as Gigerenzer points out in Rationality for Mortals: How People Cope with Uncertainty, this is perhaps the weakest argument for heuristics:

[W]e will start off by mentioning the weakest reason. With simple heuristics we can be more confident that our brains are capable of performing the necessary calculations. The weakness of this argument is that it is hard to judge what complexity of calculation or memory a brain might achieve. At the lower levels of processing, some human capabilities apparently involve calculations that seem surprisingly difficult (e.g., Bayesian estimation in a sensorimotor context: Körding & Wolpert, 2004). So if we can perform these calculations at that level in the hierarchy (abilities), why should we not be able to evolve similar complex strategies to replace simple heuristics?

Rather, the advantage of heuristics lies in their low information requirements, their speed and, importantly, their accuracy:

One answer is that simple heuristics often need access to less information (i.e. they are frugal) and can thus make a decision faster, at least if information search is external. Another answer – and a more important argument for simple heuristics – is the high accuracy they exhibit in our simulations. This accuracy may be because of, not just in spite of, their simplicity. In particular, because they have few parameters they avoid overfitting data in a learning sample and, consequently, generalize better across other samples. The extra parameters of more complex models often fit the noise rather than the signal. Of course, we are not saying that all simple heuristics are good; only some simple heuristics will perform well in any given environment.

As the last sentence indicates, Gigerenzer is careful not to make any claims that heuristics generally outperform. A statement that a heuristic is “good” is ill-conceived without considering the environment in which it will be used. This is the major departure of Gigerenzer’s ecological rationality from the standard approach in the behavioural sciences, where the failure of a heuristic to perform in an environment is taken as evidence of bias or irrationality.

Once you have noted what heuristic is being used in what environment, you can have more predictive power than in a well-solved optimisation model. For example. an optimisation model to catch a ball will simply predict that the catcher will be at the place and time where the ball lands. Once you understand that they use the gaze heuristic to catch the ball, you can also predict the path that they will take to get to the ball – including that they won’t simply run in a straight line to catch it. If a baseball or cricket coach took the optimisation model too seriously, they would tell the catcher that they are running inefficiently by not going straight to where it will land. Instructions telling them to run is a straight line will likely make their performance worse.

A week of links

Links this week:

  1. Where is the literature on behavioural political economy?
  2. Bashing a paper that claims people search for meaning as they approach a new decade (i.e. at 29, 39, 49 etc.)
  3. Vernon Smith on Adam Smith.
  4. A bucketload of Charles Darwin’s papers are now available online.