Manzi on the abortion-crime hypothesis

My recent reading of David Colander and Roland Kupers’s Complexity and the Art of Public Policy prompted me to re-read James Manzi’s Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society. I see the two books as riffs on a similar theme.

I’ll post a review of Uncontrolled later this week, but in the meantime, Manzi provides an interesting take on the Donohue-Levitt abortion-crime hypothesis. Their hypothesis is that abortion reduces crime as unwanted children are more likely to become criminals. As the legalisation of abortion increased access to abortion and decreased the number of unwanted children, decreases in crime through the 1990s and 2000s could be due to this legalisation.

Donohue and Levitt’s initial paper triggered a raft of responses, including one demonstrating an analytical error, which, once corrected for, resulted in the abortion-crime link disappearing. Donohue and Levitt then redid the work, and showed by recasting a few assumptions, the error could be corrected for and the link re-established. As Manzi states:

The revealing observation is not that there was an analytical error in the paper (which almost certainly happens far more often than we like to think), but that once it was found and corrected, it was feasible to rejigger the regression analysis to get back to the original directional result through various defensible tweaks to assumptions. If one could rule out either the original assumptions or these new assumptions as unreasonable, that would be better news for the technique. Instead we have a recipe for irresolvable debate.

Manzi also points out that Levitt, in his book Freakonomics (with Stephen Dubner), indirectly identified one of the reasons why Donohue and Levitt’s  claim is so tenuous:

In Freakonomics, Levitt and Dubner write that Roe [the Supreme Court decision in Roe v Wade establishing a right to abortion] is “like the proverbial butterfly that flaps its wings on one continent and eventually creates a hurricane on another.” But this simile cuts both ways. It is presumably meant to evoke the “butterfly effect”: meteorologist Edward Lorenz’s famous description of a global climate system with such a dense web of interconnected pathways of causation that long-term weather forecasting is a fool’s errand. The actual event that inspired this observation was that, one day in 1961, Lorenz entered .506 instead of .506127 for one parameter in a climate-forecasting model and discovered that it produced a wildly different long-term weather forecast. This is, of course, directly analogous to what we see in the abortion-crime debate and Bartels’s model for income inequality: tiny changes in assumptions yield vastly different results. It is a telltale sign that human society is far too complicated to yield to the analytical tools that nonexperimental social science brings to bear. The questions addressed by social science typically have none of the characteristics that made causal attribution in the smoking–lung cancer case practical.

A week of links

Links this week:

  1. Does public policy promote obesity? This month’s Cato Unbound on whether public policy can stop obesity could be interesting when the discussion begins, but the response essays so far have generally talked past each other.
  2. Three links via Tyler Cowen. New cars fake their engine noise. People turn down high-cost low-value treatments when they can pocket part of the savingsThe right won the economics debate; left and right are just haggling over details.
  3. The Dunning-Kruger Peak of Advertising.
  4. Chickens prefer beautiful humans. So much for the subjectivity of beauty.
  5. Default retirement savings in Illinois.

Grade inflation and the Dunning-Kruger effect

The famous Dunning-Kruger effect, in the words of Dunning and Kruger, is a bias where:

People tend to hold overly favorable views of their abilities in many social and intellectual domains

in part, because:

[P]eople who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it.

There have been plenty of critiques and explanations over the years, including an article by Marian Krajc and Andreas Ortmann who argue the overestimation of ability is partly a signal extraction problem. In environments where people are not provided with feedback on their relative standing, they are will tend to make larger estimation errors.

Krajc and Ortmann point out that the Dunning-Kruger study, as is typical, was done using psychology undergraduates at Cornell. This sample is already a self-selected pool that excludes those unable to gain admission. And once in University, the feedback they receive on their performance is not as useful as it could be. Krajc and Ortmann write [references largely excluded]:

In addition, it is well-known from studies of grade inflation that grades at the undergraduate level have – with the notable exception of the natural sciences – become less and less differentiating over the years: more and more students are awarded top grades. For example, between 1965 and 2000 the number of A’s awarded to Cornell students has more than doubled in percentage while the percentage of grades in the B, C, D and F ranges has consequently dropped (in 1965, 17.5% of grades were A’s, while in 2000, 40% were A’s). These data strongly suggest that Cornell University experiences the same phenomenon of (differential) grade inflation that Harvard experiences and the schools discussed in Sabot and Wakeman-Linn (1991). The dramatic grade inflation documented for the humanities and social-sciences devalues grades as meaningful signals specifically in cohorts of students that are newly constituted and typically draw on the top of high-school classes. Inflated grades complicate the inference problem of student subjects that, quite likely, were students in their first year or in their first semester.

Grade inflation is robbing people of feedback they could use to understand their level of competence.

*A post by Eaon Pritchard on the “Dunning-Kruger peak” reminded me that I was sitting on this passage.

The benefits of cognitive limits

Cleaning up some notes recently, I was reminded of another interesting piece from Gerd Gigerenzer’s Rationality for Mortals (other clips from late last year are here, here and here):

Is perfect memory desirable, without error? The answer seems to be no. The “sins” of our memory seem to be good errors, that is, by-products (“spandrels”) of a system adapted to the demands of our environments. In this view, forgetting prevents the sheer mass of details stored in an unlimited memory from critically slowing down and inhibiting the retrieval of the few important experiences. Too much memory would impair the mind’s ability to abstract, to infer, and to learn. Moreover, the nature of memory is not simply storing and retrieving. Memory actively “makes up” memories—that is, it makes inferences and reconstructs the past from the present. This is in contrast to perception, which also makes uncertain inferences but reconstructs the present from the past. Memory needs to be functional, not veridical. To build a system that does not forget will not result in human intelligence.

Cognitive limitations both constrain and enable adaptive behavior. There is a point where more information and more cognitive processing can actually do harm, as illustrated in the case of perfect memory. Built-in limitations can in fact be beneficial, enabling new functions that would be absent without them (Hertwig & Todd, 2003). …

Newport (1990) argued that the very constraints of the developing brain of small children enable them to learn their first language fluently. Late language learners, in contrast, tend to experience difficulties when attempting to learn the full range of semantic mappings with their mature mental capacities. In a test of this argument, Elman (1993) tried to get a large neural network with extensive memory to learn the grammatical relationships in a set of several thousand sentences, yet the network faltered. Instead of taking the obvious step of adding more memory to solve the problem, Elman restricted its memory, making the network forget after every three or four words—to mimic the memory restrictions of young children who learn their first language. The network with the restricted memory could not possibly make sense of the long complicated sentences, but its restrictions forced it to focus on the short simple sentences, which it did learn correctly, mastering the small set of grammatical relationships in this subset. Elman then increased the network’s effective memory to five or six words, and so on. By starting small, the network ultimately learned the entire corpus of sentences, which the full network with full memory had never been able to do alone.

Gigerenzer also makes the case that most visual illusions are “good errors” necessary in an intelligent animal. Assumptions used to create the illusions, such as “light tends to come from above”, inform what we “see”.

Perceptual illusions are good errors, a necessary consequence of a highly intelligent “betting” machine (Gregory, 1974). Therefore, a perceptual system that does not make any errors would not be an intelligent system. It would report only what the eye can “see.” That would be both too little and too much. Too little because perception must go beyond the information given, since it has to abstract and generalize. Too much because a “veridical” system would overwhelm the mind with a vast amount of irrelevant details. Perceptual errors, therefore, are a necessary part, or by-product, of an intelligent system. They exemplify a second source of good errors: Visual illusions result from “bets” that are virtually incorrigible, whereas the “bets” in trial- and-error learning are made in order to be corrected eventually. Both kinds of gambles are indispensable and complementary tools of an intelligent mind.

The case of visual illusions illustrates the general proposition that every intelligent system makes good errors; otherwise it would not be intelligent. The reason is that the outside world is uncertain, and the system has to make intelligent inferences based on assumed ecological structures. Going beyond the information given by making inferences will produce systematic errors. Not risking errors would destroy intelligence.

In other parts of his work Gigerenzer builds the case that many of the “biases” identified by Kahneman and friends fall into the “good errors” camp.

I have one last piece from Rationality for Mortals that I will post about in coming weeks – Gigerenzer’s attack on the “null ritual” in statistics.

A week of links

Links this week:

  1. Skip your annual physical.
  2. The phrase “Statistical significance is not the same as practical significance” is leading us astray.
  3. The ineffectiveness of food and soft drink taxes (although not all calories are the same). The latest extension of the nanny state – banning junk food from playgrounds. And a new book worth looking at – Government Paternalism: Nanny State or Helpful Friend? (HT: Diane Coyle)
  4. Marijuana in Colorado – no surprise that the most grandiose claims of both sides haven’t come to fruition.
  5. Even more on lead and crime.

That chart doesn’t match your headline – fertility edition

Under the heading “Japan’s birth rate problem is way worse than anyone imagined“, Ana Swanson at The Washington Post’s Wonkblog shows the following chart:

Japan fertility rate

So, the birth rate problem is worse than forecast in 1976, 1986, 1992 and 1997. However, the birth rate is higher than was forecast in 2002 and 2006 – so has surprised on the upside. It’s only “worse than anyone imagined” if you’ve had your head in the sand for the last 10 or so years. As Noah Smith asks, didn’t any of the people tweeting the graph (it appeared at least half a dozen times in my feed) look at it?

That said, the chart demonstrates the lack of robust conceptual models that might be used to forecast fertility. As another example, the below figure comes from Lee and Tuljapurkar’s Population Forecasting for Fiscal Planning: Issues and Innovations and shows US Census Bureau forecasts through to 1996. As for the Japan forecasts, the tendency is to assume a slight reversion toward replacement fertility followed by constant fertility.

US Census forecasts

The Bureau of the Census produced high and low estimates (as in the figure below), but these don’t make the forecasting look any better. For many forecasts, the fertility rate was outside the range within 3 years. In 1972, fertility fell outside the range before the forecast was even published.

US Census High-Low forecasts

Over the last ten years, fertility “surprises” on the upside are typical in developed countries. Japan is not an outlier. Below are three consecutive projections from the Australian Government’s Intergenerational Report. IGR1 was published in 2002, IGR2 in 2007 and IGR 2010 in 2010 (obviously). As you can see, they’ve been chasing an upward trend in fertility. The fertility problem is less severe than once thought. Long-term fertility is assumed to be 1.60 in the 2002 forecast, but 1.90 in 2010. A new IGR is due out this year, so it will be interesting to see where that forecast goes.

IGR_2007 fertility chartIGR_2010 fertility chart

As for building better conceptual models of fertility, I don’t envy anyone attempting that task. But as I argue in a working paper, evolutionary dynamics will tend to drive fertility rates up from recent lows. Is that part of the story behind what we are seeing in Japan and elsewhere?

Bad statistics – cancer edition

Are two-thirds of cancer due to bad luck as many recent headlines have stated? Well, we don’t really know.

The paper that triggered these headlines found that two-thirds of the variation in log of cancer risk can be explained by the number of cell divisions. More cell divisions – more opportunity for “bad luck”.

But, as pointed out by Bob O’Hara and GrrlScientist, an explanation for variation in incidence is not an explanation for the absolute numbers. As they state, although all variation in the depth of the Marianas trench might be due to tides, tides are not the explanation for the depth of the trench itself. There might be some underlying factor X affecting all cancers.

My reason for drawing attention to this misinterpretation is that a similar confusion occurs in discussions of heritability. Heritability is the proportion of variation in phenotype – an organism’s observable traits – due to genes. If heritability of height is 0.8, 80 per cent of variation in height is due to genetic variation. But your height is not “80 per cent due to genes”.

To make this distinction clear, consider the number of fingers on your hand. Heritability of the number of fingers on your hand is close to zero. This is because most variation is due to accidents where people lose a finger or two. But does this mean that the number of fingers on your hand is almost entirely due to environmental factors? No, it’s almost entirely genetic – those five fingers are an expression of your genes. There is an underlying factor X – our genes – that are responsible for the major pattern.

Turning back to the cancer paper, as PZ Myer points out, there may be no underlying factor X affecting cancer and the two-thirds figure could be correct. Extrapolating one chart hints that might be the case. But that’s not what the paper states.

As an endnote, a recent study pointed out that most errors in scientific reporting start in the research centre press release. This case looks like no exception. From the initial John Hopkins press release (underlining mine):

Scientists from the Johns Hopkins Kimmel Cancer Center have created a statistical model that measures the proportion of cancer incidence, across many tissue types, caused mainly by random mutations that occur when stem cells divide. By their measure, two-thirds of adult cancer incidence across tissues can be explained primarily by “bad luck,” when these random mutations occur in genes that can drive cancer growth, while the remaining third are due to environmental factors and inherited genes.

And from their updated press release:

Scientists from the Johns Hopkins Kimmel Cancer Center have created a statistical model that measures the proportion of cancer risk, across many tissue types, caused mainly by random mutations that occur when stem cells divide. By their measure, two-thirds of the variation in adult cancer risk across tissues can be explained primarily by “bad luck,” when these random mutations occur in genes that can drive cancer growth, while the remaining third are due to environmental factors and inherited genes.

Good on them for updating, but it would have been nice if they had clarified why their first release was problematic.

A week of links

Links this week:

  1. Arnold Kling’s review of Complexity and the Art of Public Policy.
  2. Are some diets mass murder? HT: Eric Crampton
  3. Social conservatism correlates with lower cognitive ability test scores, but economic conservatism correlates with higher scores.”
  4. More on lead and crime.
  5. A risk averse culture. HT: Eric Crampton
  6. Welfare conditional on birth control.
  7. We may regret the eclipse of a world where 6,000 different languages were spoken as opposed to just 600, but there is a silver lining in the fact that ever more people will be able to communicate in one language that they use alongside their native one.” HT: Steve Stewart Williams
  8. If you want to feel older, read this. HT: Rory Sutherland

The blogs I read

Although RSS seems to be on the way out, I’ve found myself explaining feed readers to a few people recently. They asked for some suggestions of blogs to follow, so below are some from my reading list.

I try not to live in a bubble, but you can see a libertarian bent to these recommendations. My full reading list (as at 4 January 2015) is here – unzip and upload it into your favourite feed reader – and is a bit broader than the below might suggest.

Statistical Modeling, Causal Inference, and Social Science: My favourite blog. Regularly skewers statistical papers of all types. I’ve learnt more about the practical use of statistics from Andrew Gelman than I have in any statistics or econometrics class.

Offsetting Behaviour: Eric Crampton’s regular dismantling of those who want to protect us from ourselves is always worth reading.

Gene Expression: Still the best evolutionary biology and genetics blog.

Bleeding Heart Libertarians: The blog at which I feel most at home politically.

Econlog: I have only Bryan Caplan’s posts in my feed, although Caplan is possibly the most infuriating thinker I regularly read.

Askblog: Arnold Kling’s post-Econlog blog is always a source of sharp comment on interesting material.

Marginal Revolution: One of the most popular economics sites, but possibly the best aggregator of interesting content.

Econtalk: Not a blog but a podcast. Russ Roberts has an impressive guest list and is rarely dull. There is a massive back catalogue worth working through.

Club Troppo: A centrist Australian political blog. I don’t have any Australian “libertarian” or “free market” blogs in my feed, as they are generally horrible – conservative at best (rare), corporatist at worst, with posts closer to trolling than informative and comment sections that make the eyes bleed.

Information Processing: Stephen Hsu provides plenty of material at the cutting edge of research into genetics and intelligence.

Santa Fe Institute News: The best feed of complexity related stories and ideas.

Matt Ridley’s Blog: Hit and miss (a bit like The Rational Optimist), but more than enough good material.

The Enlightened Economist: A constant source of additions to my book reading list.

Self evident but unexplored – how genetic effects vary over time

A new paper in PNAS reports on how the effect of a variant of a gene called FTO varies over time. Previous research has shown that people with two copies of a particular FTO variant are on average three kilograms heavier than those with none. But this was not always the case. I’ll let Carl Zimmer provide the background:

In 1948, researchers enlisted over 5,000 people in Framingham, Mass., and began to follow their health. In 1971, the so-called Framingham Heart Study also recruited many of the children of original subjects, and in 2002, the grandchildren joined in. In addition to such data as body mass index, the researchers have been gathering information on the genes of their subjects.

The scientists compared Framingham subjects with the risky variant of FTO to those with the healthy variant. Over all, the scientists confirmed the longstanding finding that people with the risky FTO variant got heavier.

People born before the early 1940s were not at additional risk of putting on weight if they had the risky variant of FTO. Only subjects born in later years had a greater risk. And the more recently they were born, the scientists found, the greater the gene’s effect.

Some change in the way people lived in the late 20th century may have transformed FTO into a gene with a big impact on the risk of obesity, the researchers theorized.

This result is unsurprising. It is standard knowledge that genetic effects can vary with environment, and when it comes to factors likely to influence obesity such as diet, there have been massive changes to the environment over time. As the authors of the PNAS paper note:

This idea, that genetic effects could vary by geographic or temporal context is somewhat self-evident, yet has been relatively unexplored and raises the question of whether some association results and genetic risk estimates may be less stable than we might hope.

It is great that the authors have provided a particular example of this change. And also useful, the study provides another response to the claim genetics is not relevant to increases in obesity because there has been limited genetic change since levels of obesity took off. The high heritability of obesity has always pointed to the relevance of genetics, but this paper strengthens the case.

In his NY Times piece, Carl Zimmer quotes study co-author Nicholas Christakis on whether the changing role of genes may be a more general phenomenon:

Dr. Nicholas A. Christakis, a sociologist and physician at Yale University and a co-author of the new study, suggested that the influence of many other genes on health had waxed and waned over the past century. Reconstructing this history could drastically influence the way doctors predict disease risk. What might look like a safe version of a gene today could someday become a risk factor.

“The thing we think is fixed may not be fixed at all,” said Dr. Christakis.

I have written before about another example of the changing effect of genes over time – the effect of genes on fertility.

Before the demographic transition when fertility rates plunged in the world’s developed countries, the heritability of fertility was around zero. This is unsurprising as any genetic variation in fitness is quickly eliminated by natural selection.

But when you look at the heritability of fertility after the demographic transition, things have changed. The heritability of fertility as derived from twin studies is around 0.2 to 0.4. That is, 20 to 40 per cent of the variation in fertility is due to genetic variation. People with different genes have responded to changes in environment in different ways.

The non-zero heritability of fertility has some interesting implications for long-term fertility. My working paper outlines the research on the heritability of fertility in discussing these long-term implications. I have posted on the working paper here.