Peer review: Trial by Twitter

Interesting article in nature a couple of days ago … first the core of the article, then some additional thoughts from me ..

Blogs and tweets are ripping papers apart within days of publication, leaving researchers unsure how to react.

“Scientists discover keys to long life,” proclaimed The Wall Street Journal headline on 1 July last year. “Who will live to be 100? Genetic test might tell,” said National Public Radio a day later.

These and hundreds of similarly enthusiastic headlines were touting a paper in Science in which researchers claimed to have identified a set of genes that could predict human longevity with 77% accuracy — a finding with potentially huge implications for medicine, health policy and the economy.

But even as the popular media was trumpeting the finding, other researchers were taking to the web to criticize the paper’s methodology. “We expect that most of the results of this study will not have the same longevity as its participants,” sniped a blog posted by researchers at the personal genomics company 23andMe, based in Mountain View, California.

Critics were particularly perturbed by the genome-wide association study (GWAS) that the authors had used to identify their longevity genes: the centenarians and the controls in the study had been tested with different kinds of DNA chips, which potentially skewed the results.

“Basically anybody that does a lot of GWAS knows this [pitfall], which is why we all said it so fast,” says David Goldstein, director of Duke University’s Center for Human Genome Variation, who voiced his concerns to a Newsweek blogger the day the study appeared.

This critical onslaught was striking — but not exceptional. Papers are increasingly being taken apart in blogs, on Twitter and on other social media within hours rather than years, and in public, rather than at small conferences or in private conversation. In December, for example, many scientists blogged immediate criticisms of another widely publicized paper — this one heralding bacteria that the authors claimed use arsenic rather than phosphorus in their DNA backbone.

In essence … the world has changed …

Now what is more interesting is their reference to the second example, the now infamous NASA alien arsenic story. That’s an example of what happens when you attempt to play footsie with the media. So what happened there? I guess we all know.

A paper that could at best be described as preliminary research was published in science. And thats OK, it was quite obviously preliminary, there was still much to be done and further studies were called for to eliminate other explanations. Unfortunately, NASA were funding this and somebody there decided to get a bit of PR out of it by having a press conference with the lead researcher and also gave it an alien spin. Sigh! … what can I say except… bad idea … no really, a very bad idea. The lead researcher might indeed be a smart cookie, but she was quite obviously not media aware. Then when folks start to ask questions, its an even dumber move to simply go dark … well you know the rest.

Peer-review by media … just don’t go there. Between blogging and twitter, you will be ripped the shreds.

Anyway, back to the nature article. Their point relates beyond this to the manner in which papers are published, and don’t get touted in the media spotlight, are now being put under a lot more public scrutiny … some see this as a good thing (and I agree)

To many researchers, such rapid response is all to the good, because it weeds out sloppy work faster. “When some of these things sit around in the scientific literature for a long time, they can do damage: they can influence what people work on, they can influence whole fields,” says Goldstein. This was avoided in the case of the longevity-gene paper, he says. One week after its publication, the authors released a statement saying, in part, “We have been made aware that there is a technical error in the lab test used … [and] are now closely re-examining the analysis.”

So where will this take us, comments sections after each article in peer-reviewed journals? Perhaps not, but it will at least cause folks who propose to publish to pause and consider if its solid and will hold up under the barrage of such an onslaught … now that must be good.

You can read the full article here. They also have interesting things to say about the various new ways of capturing, organizing and measuring all these scattered inputs are panning out.

Leave a Reply

%d bloggers like this: