Studies Are Usually Bunk, Study Shows

From the WSJ:

The world is inundated with the manipulation of flighty studies to prove some larger point about mankind in the name of behavioral science. Pop psychologists have churned out mountains of books proving some intuitive point that turns out to be wrong. It’s “sciencey,” with a whiff of (false) authenticity.

Malcolm Gladwell is the master. In his 2008 book, “Outlier,” he argues that studies show no one is born better than anyone else. Instead success comes to those who put in 10,000 hours of practice. That does sound right, but maybe Steph Curry shoots hoops for 10,000 hours because he is better than everyone at basketball in the first place. Meanwhile I watch 10,000 hours of TV. Facing criticism, Mr. Gladwell somewhat recanted: “In cognitively demanding fields, there are no naturals.” News alert: Professional sports are cognitively demanding.

When reading people like Mr. Gladwell, you’re probably thinking many of the studies’ conclusions sound right but don’t really reflect your own experience. That’s probably because you’re not a hung-over grad student. Andrew Ferguson of the Weekly Standard studied behavioral economic studies and discovered many are done by grad students observing their peers doing trivial tasks. Then researchers draw hard conclusions from this. Rather than a study of human nature, behavioral science is, Mr. Ferguson observes, “the study of college kids in psych labs.”

Many of the studies quoted in newspaper articles and pop-psychology books are one-offs anyway. In August 2015, the Center for Open Science published a study in which 270 researchers spent four years trying to reproduce 100 leading psychology experiments. They successfully replicated only 39. Yes, I see the irony of a study debunking a study, but add to this a Nature survey of 1,576 scientists published last year. “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments,” the survey report concludes. “And more than half have failed to reproduce their own experiments.”