Lehrer on measurement

Jonah Lehrer has expanded his recent focus on measurement and grit (on which I recently posted) in an article on the usefulness of the Wonderlic test, a quasi-IQ test, in predicting quarterback performance. Lehrer cites a paper by David Berri and Rob Simmons which suggests that some metrics, including the Wonderlic test, are influencing draft position even though they are not predictive of performance. Lehrer writes:

While they found that Wonderlic scores play a large role in determining when QBs are selected in the draft — the only equally important variables are height and the 40-yard dash — the metric proved all but useless in predicting performance. The only correlation the researchers could find suggested that higher Wonderlic scores actually led to slightly worse QB performance, at least during rookie years.

Unlike Lehrer’s piece on which I previously posted, I’m sympathetic to the argument that this suggests that some people are acting on some not particularly useful measurements. However, I’m not as convinced when Lehrer (again) moves into the idea that the missing element is grit. Lehrer closes the article with the following:

So where is all this heading? How will grit become a bigger part of the scouting equation? The first step is to finally acknowledge that maximal tests aren’t effective. “I really see the Wonderlic as a reading test,” says former NFL executive Michael Lombardi, now with the NFL Network. “Until we get a better test, teams are just going to have to evaluate players the old-fashioned way, by watching them play in actual games. It takes good instincts to be a QB. Maybe it takes good instincts to find one, too.”

Hasselbeck suggests that teams pay more attention to the fundamentals of college quarterbacks, since their passing mechanics are often a window into how much grit they possess. “You know these guys have been coached for years,” he says. “So if you see a QB with flawed fundamentals, you gotta wonder what’s wrong. Is he coachable? Will he work to improve? Because that’s important. You can teach a kid to throw the ball, but only if he wants to learn.”

After all, deliberate practice makes perfect.

This first paragraph is almost the opposite of Michael Lewis’s Moneyball, where he suggested that there was too much faith in instinct and not enough in measured performance. Having years of college performance at hand, I’d be sceptical if some measures of “grit”, assuming it was the important missing variable, do not already exist.

On that note, the second paragraph suggests an opportunity. Maybe we should see some predictions in the lead-up to the next few drafts, where some of these grit loving experts could assess “passing mechanics” as a measure of grit, state who the teams should draft where and see if their performance measure is a better indicator of future success than actual draft position. That is what was impressive about Moneyball. Rather than being a story about someone complaining that teams should do a difficult task better,  it was a story about someone taking their belief and acting on it (and with the presence of Michael Lewis, putting one season’s draft selections on the record for an assessment of those beliefs).


  1. I get skeptical. It reeks of the ‘looking for the one magic property’ that somehow should explain differences, when there most likely are multiple properties within individuals, between individuals, in the situation, luck, the mathew effect, what have you.

    The post Lehrer had on grit was a summary on a paper about spelling bees, where evidently the kids that were obsessively willing to sit with flash cards on their own learning spelling did the best. A bit of a … peculiar situation. And, in that he claimed that supposedly grit could be measured by asking if one ever did things like that (obsessively and doggedly pursue something that was kind of dull, just because … well, I don’t know).

    Perhaps that is grit (I have only seen it in his blog post – I’m in psychology and I scan a lot of crap, but have not seen this. Psychology is full of individual property du jour, though, because you kind of need them. People are different. On multiple dimensions.)

    It sounded a bit… maybe obsessive. Or perhaps high on conscientiousness. Or possibly being a two marshmallow person (to bring up another favorite). Which has been measured.

    What I get a little bit tired of is this ‘it can’t possibly be intelligence as measured by IQ’ flavor. Intelligence does matter. Intelligence is well measured (for being a psychological property – compared to other psychological properties). It is clearly not the only thing that matters. There clearly are difficulties with measuring (as there always are).

    I don’t think ‘grit’ will be the salvation. It strikes me as a possible factor (even outside spelling bees) but far from the only factor.

  2. Actually Ase the original research by Duckworth looked at an individuals “grit” in no fewer than 6 studies ranging from the Spelling Bee to performance and retention of West Point cadets among others. To simply dismiss it as a search for “one magic property” is a disservice and doesn’t fairly represent the research at hand. Doesn’t show much grit, especially for someone “in psychology.” More to the point, Duckworth’s research, and Leher’s article, don’t say that intelligence doesn’t matter, or that grit is the only measure of merit either…they argue that it’s what individuals DO with intelligence that matters. The original research might be worth a read before castigating it.

Comments welcome

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s