Publication bias: Why null results are not necessarily ‘dull’ results

The publication of research findings is crucial to scientific progress. It is one of the primary ways scientists share their research with their peers and contemporaries. Indeed, it enables scientists to share their knowledge with, and inspire, future generations by preserving and keeping a record of their work that lights the way for future discoveries. 

Nonetheless:

 “To light a candle is to cast a shadow”—Ursula Le Guin, A Wizard of Earthsea.

While the publication of research ‘lights a candle’ by illuminating discoveries and allowing researchers to learn of, and build on, each other’s work, publication bias potentially casts a shadow over a trail of valuable hidden discoveries and research.

Publication bias typically describes the phenomenon whereby positive results which support a hypothesis are more likely to be published than negative or null results, thus some discoveries remain hidden from view.

This has led to the potential distortion of scientific literature, from the view that bilingualism confers certain cognitive advantages, to the exaggeration of neurological sex differences.

Publication bias, isn't just a matter of academic rigour nor some niche pretentious academic concern, it has real consequences and null results do matter.

Misleading estimates of drug efficacy due to publication bias in antidepressant drug trials could cause doctors to undertake distorted risk-benefit analyses when prescribing drugs for patients.

Moreover, publication bias can misdirect research efforts resulting in wasted time, effort, resources, and funding which ultimately slow scienfitic progress as the overstated success of drug candidates in animal trials could lead to ineffective drug candidates being brought to clinical trial.

Furthermore, publication bias can skew the results of meta-analyses when it is not accounted for—thus compromising the effectiveness of a powerful scientific tool.

This is significant; meta-analyses are designed to account for variations and errors between studies by systematically synthesising and statistically analysing their combined findings. 

Clearly, null results are important in science.

Publication bias can arise from a researcher’s decision not to publish negative results due to a combination of personal, financial, and professional pressures to publish ‘exciting’ results.

For example, the results may be contrary to existing research or the researcher’s beliefs; they may deem the results to be ‘null and dull’ and unlikely to attract further funding; and negative results may even be omitted to save page space in a journal and better highlight positive results from a study.

Some may even re-analyse their negative results or aggregate outcomes into a new endpoint to extract positive results for publication. 

Researchers may also fail to submit a study showing negative results due to their perception that journal editors and peer-reviewers are biased against such results.

Publication bias can also arise from decisions of journal editors and peer reviewers to reject studies with null results because: “negative results have never made riveting reading”; the study may be similar to the peer-reviewer’s own work; they may be biased against research in different fields; or because studies with null results tend to be scrutinised more than positive ones.

Organisational and business conflicts of interest can also produce publication bias. 

For example, a business may not publish negative results because they could endanger profits by diminishing the perceived effectiveness of a product or test.

Science is a rigorous and dynamic discipline that is attempting to implement strategies to mitigate publication bias.

This includes methods to discern and assess the extent of publication bias; statistical methods to reduce its impact in meta-analyses;10 the pre-registering of experiment design;1 and certain journals requesting and publishing null results.

So next time you peruse a scientific journal, spare a thought for the null results that never made it onto the page and the importance of these hidden discoveries.