For Peer review week 2019 Sense about Science have asked for a series of blogposts, to begin a Research ‘TO DO’list. What can researchers, universities, journals, funders and governments do to improve the quality of peer review and research? This is just the start, we need your ideas too.
Imagine a world where society has the high-quality evidence it needs to make informed
decisions about crime, health and education. What action can researchers, universities, funders and governments take to get us there?
Síle Lane, head of international campaigns and policy at Sense about Science which runs the AllTrials campaign for clinical trial transparency @Senseaboutsci #AllTrials
When evidence goes unpublished the evidence
base is incomplete – it has gaps. Furthermore, because it is often so-called
‘negative’ results (results that show there was little or no effect from an
intervention or even harm, for example) that more often go unpublished, the
evidence that exists is biased. There are many reasons why results go
unpublished. It happens when researchers and editors believe that research that
didn’t find an ‘interesting’ result is of no interest to the community or no
use for a researcher’s career. It happens where publication of results in a
journal isn’t the normal practice. It happens when research groups run out of
time or funding to spend on a particular project and need to move onto something
else. It can also happen, of course, when researchers want to keep evidence
hidden for commercial or security reasons.
When results from research aren’t published it means the same
research can get repeated unnecessarily. This wastes funds and time and in the
case of clinical research, it can lead to patient harm. It means opportunities
to explore new areas and to progress the evidence base are missed. Failure to
publish results from clinical trials puts subsequent research participants at risk
unnecessarily. When research is not peer reviewed and is not opened up to
scrutiny by the research community, we miss opportunities to improve research. Decisions
made on the basis of the evidence base risk being wrong. Systematic reviews and
meta-analyses – the gold standard of evidence – are at risk of being incomplete
and biased, meaning that guidelines and decisions based on them may be wrong.
The most studied area of publication bias is in
clinical trials. Clinical trials are the large tests often involving hundreds or
thousands of people, to check whether a medicine works and is safe. The problem
of missing evidence in clinical trials has been known about for decades and
hundreds of research papers have been published investigating publication bias
in different areas of medicine These pieces of research have routinely shown
that around half of the body of clinical trials carried out on medicines we use
today have never reported results.
Modern clinical trial tracker tools which display live information on which
clinical trials have reported results and which have not, show the situation
has improved recently but still show that today at least 30% of US and EU clinical trials that are due to have reported results have
not. Furthermore, research has shown that clinical trials that show a ‘positive’
result (ie that the medicine is effective) are twice as likely to be published
than those that show no effect,
and are published more rapidly.
Selective publication of results from clinical
trials puts patients at risk and means money is wasted.
A clinical trial of the heart drug
lorcainide carried out in 1980 identified that more of the people in the trial who were given
lorcainide died than people in the trial taking the placebo. These results were
not published until 1993. In the more than a decade when the evidence wasn’t
available patients continued to be prescribed drugs in the same class as
lorcainide. It has been estimated that 100,000 of these patients in the US
alone died unnecessarily.
In response to two separate
outbreaks of influenza in 2006-7 and 2012-13, the UK’s NHS spent £424 million
on the drug Tamiflu. They decided to do this because guidelines based on
evidence from clinical trials supplied by Tamiflu’s manufacturer, Roche,
suggested that the evidence showed that Tamiflu reduced complications from the
flu and helped people recover from the flu quicker. However, researchers at
Cochrane found out that 60% of the research these guidelines were based on was
unpublished, and some of the published research contradicted reports from the
trials submitted to regulators. It turned out that no one had seen the complete
evidence. When all the evidence was finally brought together it showed that the
evidence was so flawed such that it was impossible to judge whether Tamiflu was
any more effective than placebo.
How to fix the problem of missing evidence
Government TO DO list:
- Set up publicly accessible registers for research.
These already exist for clinical research. In the last decade
the proportion of clinical trials that are registered before they begin has
risen sharply. Now, it is rare that a clinical trial that occurs in the main
clinical trial centres does not get publicly registered in at least one place.
This means that researchers, patient groups, investors in research and research
regulators can scrutinise which clinical trials have reported results and which
have not, and chase down results from unpublished trials.
The World Health Organisation, working with the clinical
research community, maintains the Trial Registration Data Set, an internationally agreed 24-item list of the
pieces of information that must appear
in a register in order for a given trial to be considered fully registered.
- Instruct ethics committees, research governance
boards and public research funders to make registration of research before it
begins and reporting of the results, whatever the result, a condition of
approval to get funding for or run research.
The UK’s research regulator, the Health Research Authority,
committed in 2104 to making registration of a clinical trial within 6 months of
its start date a condition for granting approval to run the trial. The HRA is
exploring levers to ensure results from UK clinical trials are published.
- Where appropriate, bring in legislation to mandate
registration of research and reporting of results.
In 2014 the European Union has agreed a new Clinical
which mandates that every clinical trial carried out in the EU or EEA must be
registered before it begins and report results onto the publicly accessible
register within a year of the end of the trial (within 6 months for paediatric
trials). There will be sanctions including financial penalties for breaching
the law. This will be the law in Europe from 2020.
- Monitor adherence to registration and reporting
The US law, the
FDA Amendments Act 2007, mandates that certain US clinical trials must be registered om
the federal register before they begin and report results there within a year
of the end of the trial, and gives the FDA power to fine research sponsors who
breach these rules. Despite the fact that hundreds of applicable trials are
gong unreported the FDA has never issued any fines and adherence to this law is
Publisher/journal TO DO list:
- Make the decision about accepting a
research paper independently of whether the results are ‘positive’ or ‘negative’.
- Editors can start to encourage researchers to
register their research project before it begins by announcing it as a
requirement to have a research paper accepted in the journal.
In 2013 the members of the International
Committee of Medical Journal Editors committed to only publishing papers from
clinical trials that had been proactively registered on a publicly accessible
register in their journals. This led to a sharp increase in the proportion of
clinical trials that were registered.
Funder TO DO list:
- Make registration of research and reporting of the
results, whatever the result, a condition of new grants
- Mandate that researchers must have reported
results for all previous research to be considered for new funds
Since 2016 the US federal funder of health
research, the National Institutes of Health, has had the power to withhold remaining or future grant funds from a grantee for failure to
submit clinical trial registration and results information.
In 2017 twenty-one more of the world’s largest
non-commercial clinical research funders signed up to a World Health Organisation initiative, committing them to bring in a new
funding policy to mandate registration of all funded clinical trials and
reporting of results within a year of their end and that says that a
researcher’s past record of registering clinical trials and reporting results
would be taken into consideration when assessing an application for new funds.
- Ensure that grantees understand that the costs, if
any, of registering research and reporting results can be covered by grant
Universities TO DO list:
- Universities, hospitals and research centres should appoint a named person to be responsible for research reporting. This person should put in place internal processes to ensure research carried out by employees of the institute does not go unpublished.
- Ensure researchers are aware that non-publication of research results is increasingly viewed as research misconduct by governments, institutes and professional societies therefore will be viewed poorly by promotion committees/in tenure decisions.
We extend our thanks to David Tovey @DavidTovey for his work
editing these blogposts.