This is a story about what happened when a published peer-reviewed paper was discovered to be wrong. It is a great illustration of science being a process about the pursuit of truth.
For some “Science” is perhaps viewed as a body of knowledge. For example in some educational systems it is a topic where you learn facts. It has in that context been sliced up into specific streams that have names such as “Physics”, “Biology”, or “Chemistry”. The idea is that you learn some facts and then pass an exam to demonstrate that you understand those facts and can repeat them.
The tragedy here is that you can get A in everything and end up understanding nothing at all. You have simply acquired information. The key to it all is that it is a process, a methodology, and that can be easily missed.
Now here is a story about the process playing out.
Making an Honest Mistake
Back in 2015 there was a study by neuroscientist Jean Decety of the University of Chicago. It appeared within the peer-reviewed journal “Current Biology”, and revealed something very surprising – “Religion makes kids less compassionate and more intolerant“. I wrote about it at the time.
It was rather shocking to some because this conflicted with many previous studies. Psychologist Azim Shariff who had a professional interest in this was perplexed. It conflicted with all the previous studies on this and so he reached out to Jean Decety asking for all the data. Jean happily provided it.
Azim then carefully reviewed it all and made a discovery … Jean had made a mistake. The Boston Globe lays it all out in detail …
Azim Shariff, a professor of social psychology, was perplexed by Decety’s analysis, which seemed to fly in the face of considerable evidence that religious people tend to be more likely to exhibit altruistic, charitable, and pro-social behavior. In his 2006 book “Who Really Cares,” the public policy scholar Arthur Brooks, citing years of social-science data, showed that religious practice correlated with higher rates of volunteering, donating blood, aiding the homeless, and giving money to secular charities.
To better understand how Decety’s team had come to such a strikingly different result, Shariff asked to see their data. When he re-crunched the numbers, Shariff discovered a major blunder: The six countries in the study had been coded by number — 1 for the United States, 2 for Canada, etc. — and in calculating the global results, the researchers had inadvertently treated those country codes as mathematical variables. Needless to say, that significantly skewed the study’s results. When Shariff re-analyzed the data without the coding error, the surprising findings vanished.
In other words, the surprising conclusion was wrong and being religious really did correlate with higher rates of altruism. This is not a reflection of the religious belief being true, it simply reflects how people that are part of a religious community behave.
What happened next
This is not a posting about that specific study, nor is the intention to dig into the topic. Instead it is here to illustrate the scientific methodology in action.
The corrected data was published within the same journal in 2016.
There was still a problem here because even after 2016 various media outlets were picking up and using the published 2015 study, but remember it contains a statistical error and is wrong …
The Independent cited it in a 2017 article on the virtues of an atheist upbringing, for example. Buzzworthy ran a story on it (“Could Religion Actually Make Children Less Generous?”) just two months ago.
To deal with this, here is what happened next.
The authors of the original study reached out to the journal and asked them to Retract it and to also publish a note explaining that they had made a statistical mistake. If you go to the study link now you will find this …
What do we learn here?
The key lesson is this – there are no villains here. Instead it is a story about people following a methodology, the scientific one, to find out what is really true.
When faced with their mistake, the authors did not double down but instead demonstrated a degree of integrity and honesty.
When Decety’s paper was published, Shariff didn’t denounce it — he asked to see the underlying data. Decety, unlike some researchers, promptly made the information available. After discovering the coding error that undermined the paper’s validity, Shariff, rather than run to the popular media for publicity, submitted a sober, technical correction to the science journal where the study had appeared. And though it took a while, the original researchers conceded that their much-commented-on paper could not stand, and asked that it be retracted.
In other words, no rancorous ranting, no backbiting, no doubling down, no accusations or demands for somebody to be fired. Not even some twitter mockery. Instead we find laudable integrity and professional honesty.
This is a story of science working well. It is a saga of an honest mistake being confronted and a better understanding of what is not wrong emerging. When science works well like this then we are all better off.
If you are interested in believing as many true things as possible then you might wonder who the heroes of this story are. The answer to that is everybody involved.
Science doesn’t purvey absolute truth. Science is a mechanism. It’s a way of trying to improve your knowledge of nature. It’s a system for testing your thoughts against the universe and seeing whether they match. And this works, not just for the ordinary aspects of science, but for all of life. I should think people would want to know that what they know is truly what the universe is like, or at least as close as they can get to it.’
Isaac Asimov, interview, Bill Moyers’ World of Ideas, October 21, 1988
Further Reading
Boston Globe (4th Oct 2019) – To err is human, but to admit error is sublime