Skip to main content
Images of a brain

International effort tackles reproduction of original science results

28 August 2015
Research finds fewer than half of replications produce the same findings as the original study

A massive international collaboration of researchers replicated 100 studies in prominent psychology journals to test the replicability of their results.

Poor reproducibility has been suggested anecdotally, and has been argued from a theoretical standpoint, but now we have solid evidence that our research practices don’t always generate reproducible findings.
Dr Patrick Goodbourn

The most comprehensive investigation ever made into the reproducibility of results in a field of science has found that, regardless of the analytic method or criteria used, fewer than half of the replications produced the same findings as the original study.

The massive international collaboration involved 270 researchers, who published their findings in Science today. They conducted replications of 100 studies published in three prominent psychology journals.

“Our findings suggest that, as a field, we have some work ahead of us. But by identifying some key predictors of reproducibility, and by suggesting ways to increase it, the project marks an important milestone in improving our research practices,” said Dr Patrick Goodbourn, a researcher in the School of Psychology at the University of Sydney.

The University of Sydney and the Australian National University are the only two Australian institutions to have taken part in the study, which involved 125 institutions in 17 countries.

Launched in 2012, the Reproducibility Project: Psychology, co-ordinated by the Centre for Open Science in Virginia, USA, tests scientists’ reliance on the reproducibility of results to gain confidence in ideas and evidence. 

Reproducibility means that the results recur when the same data are analysed again, or when new data are collected using the same methods.

Fewer than half of the original findings tested were successfully replicated. This held true across multiple different criteria of success.

However, the team is careful to point out that a failure to replicate doesn’t always mean that the original finding is false. Some replications will simply have failed to detect the original result by chance. And even though most replication teams worked with the original authors to use the same materials and methods, small differences in when, where, or how the replication was carried out might influence the outcome.

The report’s authors also recognise that a problem for psychology and other fields is that incentives for scientists are not always aligned with reproducibility.

“What is good for science and what is good for scientists are not always the same thing. In the present culture, scientists’ key incentive is earning publications of their research, particularly in prestigious outlets,” said Ljiljana Lazarević, a team member from the University of Belgrade.

Research with new, surprising findings is more likely to be published than research examining when, why, or how existing findings can be reproduced.

“Poor reproducibility has been suggested anecdotally, and has been argued from a theoretical standpoint, but now we have solid evidence that our research practices don’t always generate reproducible findings,” said Dr Goodbourn. 

Suggestions for addressing the issue include improving access to all aspects of the original research and the pre-registration of research designs. In keeping with the goals of openness and reproducibility, every replication project undertaken in this study posted its methods, data and computer code on a public website.

Many organisations are already working on the issue of reproducibility, including the Reproducibility Project: Cancer Biology.

“The great thing about science when it is operating properly is that it can reflect on its own practices and correct its mistakes. This project is an example of science doing what it does best,” said Dr Goodbourn.

Verity Leatherdale

Manager, Faculty Media and PR
  • Level 2 Services Building G12

Related articles

17 June 2016

Satellite testing launches

Three small Australian cube-sized satellites will be launched from the International Space Station to research new regions that could impact technology used on Earth, with the University of Sydney leading development of one of the CubeSats, undergoing testing in Canberra this week. 

23 June 2016

New insights into the causes of sudden cardiac death in the young revealed

Sudden cardiac death claims the lives of 2-3 young Australians every week. 

23 June 2016

Researchers help Qantas fly cleaner and cheaper

Unveiled this week, the new flight planning system is the result of a world-first, four-year project conducted at the University’s Australian Centre for Field Robotics (ACFR). The team comprising four aeronautical research fellows, 3 PhD candidates and 10 software engineers worked on designing new system models.

21 June 2016

University welcomes new Sydney School of Entrepreneurship

The University of Sydney has welcomed the NSW Government's $25 million pledge to create the Sydney School of Entrepreneurship as a new collaborative venture in the higher education sector. 

15 June 2016

Robotics for farms of the future

With Australia’s farming future looking increasingly reliant on automated technologies, such as low-cost robotics, the industry’s vision should include more support for agri-technology start-ups, retraining growers and agronomists for a digital age, and introducing rural kids to hands-on robotics.   

02 June 2016

Why $1 a litre milk won’t take Australia from the mining to the dining boom

Despite demand from overseas markets such as China, Australia’s milk wars illustrate the difficulty of taking a low-cost commodity approach to competition. Farmers need to embrace digital disruption, promoting our renowned clean, green produce by using tools that are increasingly available to detail the journey from behind the farm gate to the consumer.

16 June 2016

MadMaker tours rural NSW

MadMaker is the brainchild of electrical and information technologies engineer Dr Abelardo Pardo, a specialist in technology-based learning. It was designed to inspire students who may not have previously considered a future career in engineering, technology or science-related fields.

13 December 2016

Confidence boost linked to weight loss in smartphone trial

Confidence is key when motivating young people to change their diet and exercise habits, new research from the University of Sydney shows.

13 December 2016

New diamond harder than ring bling

An international team has created a harder-than-diamond Lonsdaleite diamond – usually found at the site of meteoric impacts. Unlike cubic diamonds, the hexagonal creation is more likely to be used in manufacturing.

13 December 2016

Sydney alum off to Stanford as Monash scholar

University of Sydney alumnus Dr Martin Seneviratne has been named the 2017 Roden Cutler NSW John Monash Scholar. The award will see Dr Seneviratne head to Stanford University to continue his ground-breaking work into clinical informatics.