Something Rotten at the Core of Science?

by David F. Horrobin

http://news.bmn.com/hmsbeagle/95/viewpts/op_ed
Reprinted with permission from Trends in Pharmacological Sciences, Vol. 22,
No. 2, February 2001

Posted February 2, 2001 · Issue 95



Abstract
A recent U.S. Supreme Court decision and an analysis of the peer review
system substantiate complaints about this fundamental aspect of scientific
research. Far from filtering out junk science, peer review may be blocking
the flow of innovation and corrupting public support of science.



The U.S. Supreme Court has recently been wrestling with the issues of the
acceptability and reliability of scientific evidence. In its judgement in
the case of Daubert v. Merrell Dow, the court attempted to set guidelines
for U.S. judges to follow when listening to scientific experts. Whether or
not findings had been published in a peer-reviewed journal provided one
important criterion. But in a key caveat, the court emphasized that peer
review might sometimes be flawed, and that therefore this criterion was not
unequivocal evidence of validity or otherwise. A recent analysis of peer
review adds to this controversy by identifying an alarming lack of
correlation between reviewers' recommendations.

The Supreme Court questioned the authority of peer review.
Many scientists and lawyers are unhappy about the admission by the top legal
authority in the United States that peer review might in some circumstances
be flawed [1]. David Goodstein, writing in the Guide to the Federal Rules of
Evidence - one of whose functions is to interpret the judgement in the case
of Daubert - states that "Peer review is one of the sacred pillars of the
scientific edifice" [2]. In public, at least, almost all scientists would
agree. Those who disagree are almost always dismissed in pejorative terms
such as "maverick," "failure," and "driven by bitterness."

Peer review is central to the organization of modern science. The
peer-review process for submitted manuscripts is a crucial determinant of
what sees the light of day in a particular journal. Fortunately, it is less
effective in blocking publication completely; there are so many journals
that most even modestly competent studies will be published provided that
the authors are determined enough. The publication might not be in a
prestigious journal, but at least it will get into print. However, peer
review is also the process that controls access to funding, and here the
situation becomes much more serious. There might often be only two or three
realistic sources of funding for a project, and the networks of reviewers
for these sources are often interacting and interlocking. Failure to pass
the peer-review process might well mean that a project is never funded.
Science bases its presumed authority in the world on the reliability and
objectivity of the evidence that is produced. If the pronouncements of
science are to be greeted with public confidence - and there is plenty of
evidence to suggest that such confidence is low and eroding - it should be
able to demonstrate that peer review, "one of the sacred pillars of the
scientific edifice," is a process that has been validated objectively as a
reliable process for putting a stamp of approval on work that has been done.
Peer review should also have been validated as a reliable method for making
appropriate choices as to what work should be done. Yet when one looks for
that evidence it is simply not there.

Why not apply scientific methods to the peer review process?
For 30 years or so, I and others have been pointing out the fallibility of
peer review and have been calling for much more openness and objective
evaluation of its procedures [3-5]. For the most part, the scientific
establishment, its journals, and its grant-giving bodies have resisted such
open evaluation. They fail to understand that if a process that is as
central to the scientific endeavor as peer review has no validated
experimental base, and if it consistently refuses open scrutiny, it is not
surprising that the public is increasingly skeptical about the agenda and
the conclusions of science.

Largely because of this antagonism to openness and evaluation, there is a
great lack of good evidence either way concerning the objectivity and
validity of peer review. What evidence there is does not give confidence but
is open to many criticisms. Now, Peter Rothwell and Christopher Martyn have
thrown a bombshell [6]. Their conclusions are measured and cautious, but
there is little doubt that they have provided solid evidence of something
truly rotten at the core of science.

Forget the reviewers. Just flip a coin.

Rothwell and Martyn performed a detailed evaluation of the reviews of papers
submitted to two neuroscience journals. Each journal normally sent papers
out to two reviewers. Reviews of abstracts and oral presentations sent to
two neuroscience meetings were also evaluated. One meeting sent its
abstracts to 16 reviewers and the other to 14 reviewers, which provides a
good opportunity for statistical evaluation. Rothwell and Martyn analyzed
the correlations among reviewers' recommendations by analysis of variance.
Their report should be read in full; however, the conclusions are alarmingly
clear. For one journal, the relationships among the reviewers' opinions were
no better than that obtained by chance. For the other journal, the
relationship was only fractionally better. For the meeting abstracts, the
content of the abstract accounted for only about 10 to 20 percent of the
variance in opinion of referees, and other factors accounted for 80 to 90
percent of the variance.

These appalling figures will not be surprising to critics of peer review,
but they give solid substance to what these critics have been saying. The
core system by which the scientific community allots prestige (in terms of
oral presentations at major meetings and publication in major journals) and
funding is a non-validated charade whose processes generate results little
better than does chance. Given the fact that most reviewers are likely to be
mainstream and broadly supportive of the existing organization of the
scientific enterprise, it would not be surprising if the likelihood of
support for truly innovative research was considerably less than that
provided by chance.

Objective evaluation of grant proposals is a high priority.
Scientists frequently become very angry about the public's rejection of the
conclusions of the scientific process. However, the Rothwell and Martyn
findings, coming on top of so much other evidence, suggest that the public
might be right in groping its way to a conclusion that there is something
rotten in the state of science. Public support can only erode further if
science does not put its house in order and begin a real attempt to develop
validated processes for the distribution of publication rights, credit for
completed work, and funds for new work. Funding is the most important issue
that most urgently requires opening up to rigorous research and objective
evaluation.

What relevance does this have for pharmacology and pharmaceuticals? Despite
enormous amounts of hype and optimistic puffery, pharmaceutical research is
actually failing [7]. The annual number of new chemical entities submitted
for approval is steadily falling in spite of the enthusiasm for techniques
such as combinatorial chemistry, high-throughput screening, and
pharmacogenomics. The drive to merge pharmaceutical companies is driven by
failure, and not by success.

The peer review process may be stifling innovation.

Could the peer-review processes in both academia and industry have destroyed
rather than promoted innovation? In my own field of psychopharmacology,
could it be that peer review has ensured that in depression and
schizophrenia, we are still largely pursuing themes that were initiated in
the 1950s? Could peer review explain the fact that in both diseases the
efficacy of modern drugs is no better than those compounds developed in
1950? Even in terms of side-effects, where the differences between old and
new drugs are much hyped, modern research has failed substantially. Is it
really a success that 27 of every 100 patients taking the selective 5-HT
reuptake inhibitors stop treatment within six weeks compared with the 30 of
every 100 who take a 1950s tricyclic antidepressant compound? The
Rothwell-Martyn bombshell is a wake-up call to the cozy establishments who
run science. If science is to have any credibility - and also if it is to be
successful - the peer-review process must be put on a much sounder and
properly validated basis or scrapped altogether.

David F. Horrobin, a longtime critic of anonymous peer review. heads Laxdale
Ltd., which develops novel treatments for psychiatric disorders. In 1972 he
founded Medical Hypotheses, the only journal fully devoted to discussion of
ideas in medicine.

Andrzej Krauze is an illustrator, poster maker, cartoonist, and painter who
illustrates regularly for HMS Beagle, The Guardian, The Sunday Telegraph,
Bookseller, and New Statesman.







Tell us what you think.




Endlinks
International Congress on Biomedical Peer Review and Scientific
Publication - articles and abstracts from the third congress, held in 1997.
The fourth congress will be held in September 2001.
Peer-Review Practices at EPA - a section of the 2000 NAS report
Strengthening Science at the U.S. Environmental Protection Agency:
Research-Management and Peer-Review Practices, which discusses the strengths
and limitations of the process.
Can Peer Review Help Resolve Natural Resource Conflicts? - suggests that a
modified form of peer review could be useful in policy-related decisions.
Evidence and Expert Testimony - includes many online references for
scientific evidence.
Peer Review Articles - an annotated bibliography covering scientific peer
review and its relevance to judicial proceedings.
Related HMS Beagle Articles:
Top Ten Reasons Against Peer Review and Top Ten Reasons For Peer Review -
arguments both humorous and serious.
Anatomy of a Rejection - strategies for improving the outcome of the peer
review process.

--------------------------------------------------------