Failure to respond as a measure of conscientiousness and IQ

A few weeks ago, Bryan Caplan pointed out this great working paper by David Hedengren and Thomas Stratmann:

The Dog that Didn’t Bark: What Item Non-Response Shows about Cognitive and Non-Cognitive Ability 

What survey respondents choose not to answer (item non-response) provides a useful task based measure of cognitive ability (e.g., IQ) and non-cognitive ability (e.g., Conscientiousness). Using the German Socio-Economic Panel (SOEP) and the National Longitudinal Survey of Youth 1997 (NLSY97), we find consistent correlation between item non-response and traditional measures of IQ and Conscientiousness. We also find that item non-response is more strongly correlated with earnings in the SOEP than traditional measures of either IQ or Conscientiousness. We also use the Survey of Income and Program Participation (SIPP) Gold Standard, which has no explicit measure of either cognitive or non-cognitive ability, to show that item non-response predicts earnings from self-reported and administrative sources. Consistent with previous work showing that Conscientiousness and IQ are positively associated with longevity, we document that item non-response is associated with decreased mortality risk. Our findings suggest that item non-response provides an important measure of cognitive and non-cognitive ability that is contained on every survey.

The triumph of this paper is the mass of survey data to which this technique can be applied. As the authors state in the opening paragraph:

Studying the importance of non-cognitive skills, such as conscientiousness, perseverance, and motivation, has been hamstrung by the fact that many popular data sets in economics do not contain information on an individual’s personality traits. However, surveys contain a valuable but neglected piece of data: what respondents do not say. Respondents skip, refuse to answer, or claim ignorance on at least a few questions in virtually all surveys. When a respondent forgets to fill in answers to some questions on the survey form, or refuses to provide an answer to the interviewer, we gain important information about the respondent.

4 comments

  1. hmmm. You are taking a lot at face value here, Jason. Without bothering to read the whole study, let us mull over the issues in the abstract, shall we?

    They find that people who dont answer many questions have less earnings. And how does the Gsoep measure earnings? Why, via a whole battery of questions of course! So people who answer less questions turn out to have less total earnings. Surprise, surprise.

    Then they find that people who dont answer questions are less likely to die. And how do you think the Gsoep knows that they have died? Well, they often only know this from getting answers from others as to the question whether they are still alive. So people who answer fewer questions are less likely to have others give out information on them? Again, surprise, surprise.

    It is the mortality effect that sounds particularly suspicious and should make you much more sceptical of this paper than you seem to be. The authors are right that IQ is related to longevity but in the opposite direction of what they find themselves: the smarter are usually found to live longer, so saying that the non-respondents are the dumbos who live longer doesnt quite fit, now does it?

    Apart from all that, thanks for the write-up of the PhD conference. Very kind.

    1. The authors test their results using non-survey income and death data:

      Having established support for the hypothesis that item response is a function of Conscientiousness and IQ, we find the item response rate derived from the Survey of Income and Program Participation (SIPP), which has no explicit measures of cognitive or non-cognitive ability. We find that higher item response is associated with higher self-reported earnings. In order to confirm that this relationship is not a mechanical artifact of the data collection process, we use the SIPP Gold Standard File, that links SIPP records to Social Security Administration earnings data, to obtain out-of-sample estimates of how item response correlates with earnings. We find that item response is an even stronger predictor of administrative earnings than it is of self-reported earnings.

      Motivated by the finding that Conscientiousness and IQ are positively correlated with longevity (Savelyev, 2010), we also analyze how item non-response effects mortality. For this we take advantage of the SIPP Gold Standard’s administrative death records. We find that individuals with higher response rates are less likely to die. This again suggests that item response is capturing the same characteristics captured in traditional measures of IQ and Conscientiousness.

      On the direction of the IQ-longevity correlation, their abstract is poorly worded, as the last sentence of the above quote suggests.

      1. Fair enough. The abstract indeed reads to say that more item non response would lead to lower morbidity.
        Still, the authors are relying on what they call a gold-standard data file. And how do you think social security knows about income? Even in Germany taxes involve a large set of questions people have to fill in, so the potential for a spurious relation remains. They should focus on an item of earnings that is easier to manually verify, such as perhaps job title.
        With the reversed longevity result though it does start to sound more plausible!

Comments are closed.