Don’t tell anyone, but in our vast test kitchens and cavernous research facilities under Yucca Mountain, we often face the dilemma of conflicting conclusions reached by different research teams — and then, in the interest of objectivity, we’re forced to flip a coin.
We use a tamper-proof 1899 Morgan silver dollar.
Since you can’t very well subject peerless researchers to peer review, we’ve never had to disclose our coinflip methodology to a peer-reviewed journal; but you can imagine our relief when we learned that everybody else’s research is comparably flawed, whether or not they admit it.
I’m not referring to the stock market recommendations of sell-side analysts on Wall Street or the studies sponsored by tobacco companies or drug companies, all of which are Easter egg hunts for new arguments to support foregone conclusions – as are the climate change denying studies covertly funded by coal and oil companies and the ubiquitous Koch brothers.
No, I mean real research by good scientists in pursuit of genuine discoveries – but who also have careers to build, tenure to seek, grants to obtain, and passionate beliefs in their own theories.
Get thee behind me, Satan, so you can push a little harder.
In Tuesday’s Science Times, George Johnson recalls a 2005 paper published by Dr. John Ioannidis – a scientist whose research subject is research itself. It was titled, “Why Most Published Research Findings Are False.”
Dr. Ioannidis noted the human tendency to see what we want to see, magnified by competition and a shrinking pool of grant money, along with some error-prone experimental designs – a context within which it’s easy to fool yourself, even with the best of conscious intentions.
Other scientists wondered if Dr. Ioannidis’ findings might have been skewed by his own biases.
But then he published a definitive report on the most highly regarded research papers of the previous decade (e.g. on the effects of a daily aspirin on cardiac patients, the risks of hormone replacement therapy for older women), finding that in most cases the reported results were later altered or contradicted.
Since then other researchers have come forward with kindred views.
One recalled that while working at Amgen he and his colleagues tried to replicate the findings of 53 landmark cancer papers. In 47 of the 53 cases they were unable to do so, even when they got the original researchers to collaborate.
Now that the alarm has been sounded, various journals and institutes are looking for ways to reform the process. But as Johnson points out in the Times, the scientific literature “has roughly doubled in size every 10 or 15 years since the days of Isaac Newton.”
He says the National Library of Medicine’s PubMed database alone containes 23 million citations.
How can we help? We’re thinking of sending them our 1899 silver dollar.