I have been reading a book by Mark Bowen, "Censoring Science", with the subtitle "Inside the political attack on Dr. James Hansen and the truth of global warming." I have to confess I have not read much of it, and that is partly because what I see in there is simply too disturbing to an innocent person such as myself, to read in big chunks. But while I have been randomly trying to make headway in the book, several related stories have caught my eye.The first is a study in a peer reviewed Journal, with the hypothesis that remote prayer can help infertile couples conceive. According to the study, fertility rates are boosted 100%.
http://www.nytimes.com/2001/10/02/health/vital-signs-fertility-a-study-links-prayer-and-pregnancy.html
http://www.ebm-first.com/prayer/columbia-prayer-study-controversy.html
The second "The Truth Wears Off: Is there something wrong with the scientific method" by Jonah Lehrer, in the New Yorker Magazine, December 13, 2010. This article refers to something called "The Decline Effect". Which means that when a new scientific study is made it's effect decreases with time. To give a simple example (and this is made up, so don't look for it): If a scientific study on ESP finds that there is a person who can predict the next card off a deck 50% of the time, (compared to the average of 2%), then in future studies on the same subject, that figure of 50% will most likely decrease, and possibly drop down all the way down to 2%, thus eventually contradicting the results of the first study.
http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=1
It has been proposed that despite the efforts to eliminate bias in peer reviewed science, the bias still exists. But it drops with time, and not only that, but the bias reverses itself with time.
Why is that so? I think the Decline Effect is fairly easy to understand. Scientific studies that produce no result are rarely submitted to peer reviewed journals. For example, take the remote prayer study. If, say 100 independent studies had been made, and only one showed a positive effect, the one with the positive effect would show up in a Journal of Medicine, the other 99 would not even be submitted. That is the first level of bias. Then, if the study is kind of uninteresting, very few other scientists will attempt further tests on it. The peer review system does not compel other scientists to immediately duplicate any experiment to confirm or deny its hypothesis. So basically we have another level of bias, based on how interesting the experiment is to other scientists. Then, if an experiment leads to a general consensus, the bias swings the other way. That is, if remote prayer was accepted by the scientific community as more or less proven, more scientists would be encouraged to submit experiments proving that it was not true.
Many of these Decline Effects have to do with built in biases in the peer reviewed system. But I would imagine that the more important a study, or the more it makes sense, then the more scientists are attracted to it. And statistically, the more scientists that are interested, the better the peer reviewed system works. That is, you would have more studies submitted. Then you can better eliminate statistical outliers and mistakes. And with more research and debate, the better the understanding of the subject becomes.
We have some noticeable random glitches in the scientific peer reviewed process, when it comes to studies of low interest. I guess we can expect a "Decline Effect" with scientifically repulsive or uninteresting topics such as remote prayer. But with topics of extreme high interest such as global warming and evolution, the mass of research, and related experiments being done should help eliminate statistical aberrations in the system. Especially when those aberrations are directly related to the interest level.
Considering the general assault on science being conducted by the anti-global warming people and by Bible Literalists, all shoddy peer reviewed science is liable to be taken as a weakness of the entire structure of science. Any system, no matter how good, has weaknesses. And as it is exposed to attacks, those weaknesses will become more critical.
Recently we have seen leaked documents from global warming scientists, who seem to be expressing their frustration that science is being gradually undermined by outside forces. We are also starting to see pseudo-scientists finding ways to state that their experiments have been peer reviewed, even though they have not. No, it's not outright lying, it's simply twisting words around, half-truths, and looking for weak links in a complicated system.
For example, a person self-publishes a scientific experiment, then sends their book to an eminent scientist (albeit in another field) and asks for that scientist's opinion, then getting the opinion that "it looks interesting", calls it "peer reviewed". That is not peer review. But is it a lie?
The Peer Review system will probably need to set up a legal defence department one day to protect their good name against infringements. And scientists need to rigorously eliminate statistical anomalies like ESP experiments and Remote Prayer studies.
Just like Disney Corporation protects their Mickey Mouse character, and the Hell's Angels protect their death head logo, I think scientists need to protect their "Peer Review" brand. The world is changing, and science needs to keep up.
Another article about what's wrong with the peer review process and wondering how it can be improved.
http://whitehead.mit.edu/news/paradigm/spring_2008/peer_review.html
Picture: http://www.cartoonstock.com/directory/p/peer_review.asp
Any human endeavor is bound to be imperfect. Prejudice and emotion (what scientists refer to as 'subjectivity') are virtually impossible to eliminate in any field.
ReplyDeleteThe practice of science is no different. Historically we have often seen reluctance of the part of those holding 'generally accepted' opinions to concede when evidence of a new paradigm is first presented. Even as respected a figure as Albert Einstein was reluctant to accept the 'fundamental dice-game' of Quantum Theory.
However, despite its flaws, doing science does tend to move us ever closer to the truth. The problems with funding mechanisms and publication processes will persist, to a greater or lesser degree. And the nefarious use of pseudoscience for political and avaricious will also persist.
And, as Robert Weinberg puts it in that Paradigm Magazine article you link, '[Science is] like democracy - it's flawed, but it's better than the alternative.'
Some may recall the controversies swirling around a year ago about the 'safety' of the H1N1 vaccine.
ReplyDeleteA large number of North Americans were convinced that vaccines could cause autism in their children. This concern reached the level of hysteria in the U.S. and worked against general immunization programs.
Much of the 'scientific evidence' for that fear was based on a 1998 'The Lancet' (peer-reviewed, naturally) study. The Lancet, of course, is a leading British medical journal.
The British Medical Journal now reveals that the data in that original study was faked - by a physician taking kickbacks from lawyers attempting to sue vaccine manufacturers.
www.washingtonpost.com/wp-dyn/content/article/2011/01/05/AR2011010507052.html
The 'bad news' is that this article was not retracted for a decade. The 'good news' is that the fraud was eventually exposed.
Science may be an imperfect process but, in the end, we do move closer to the 'truth.'
I had to repost the previous comment as it was deleted by Google before I could complete my response.
ReplyDeleteMy response to the vaccine comments:
Any scientist taking money from an interested party must be required to disclose it, and prove it has not affected their impartiality. In the Global Warming debate, so-called scientists who deny global warming have been found to have contributions made indirectly from Heritage Foundation etc. and they deny knowledge once they are accused. Also, there is the issue of a cash reward offered for papers denying global warming. (I can't find that one right now, but at the time it seemed that nobody thought it was a big deal.)
Of course this must work both ways. Hopefully it is not too late, because when or if most peer-review scientists get their paycheck from interested parties, it will be impossible to stop.
Enlightenment?
ReplyDeleteTheories, Blind men-Elephant And Peer-Review
A. Theory, the first of several definitions:
http://www.merriam-webster.com/dictionary/theory
The analysis of a set of facts in their relation to one another.
B. Enlightenment, a philosophical movement of the 18th century, characterized by belief in the power of human reason and by innovations in political, religious, and educational doctrine.
http://dictionary.reference.com/browse/enlightenment
C. The tale re the blind men and elephant:
http://en.wikipedia.org/wiki/Blind_men_and_an_elephant
D. Peer-review:
http://www.merriam-webster.com/dictionary/peer%20review
A process by which something proposed (as for research or publication) is evaluated by a group of experts in the appropriate field.
E. AAAS peer-review:
http://universe-life.com/2011/01/25/hope-for-2011-science/
http://universe-life.com/2011/06/11/for-a-scientism-culture/
F. What kind of enlightenment can re-evolve in a counter-enlightenment culture, in which scientific theories postulated by “blind men” are selectively “certified” or “rejected” by an omnipotent blind-men trade-union-association?
Dov Henis (comments from 22nd century)
http://universe-life.com/2011/12/13/21st-century-science-whence-and-whither/