A new approach to understanding research relevance
5 March 2013
"Science is broken; let's fix it," says the University of Sydney's Associate Professor Alex Holcombe, who is part of a major new effort to improve the reliability of psychological research.
The new approach encourages psychologists to publish replication studies of previously published results, helping make this a part of ongoing scientific practice.
"Reproducing results using the same methods as the original experiment but with different participants is critical to science, but we are currently facing a crisis," says Associate Professor Holcombe.
"If someone makes an important research discovery we need to replicate the results and publish the findings to have confidence in the original research.
"Addressing this issue will provide benefits to the scientific community and the general public as it will make research more transparent and accountable."
The 'replicability crisis' exists because few scientists are interested in publishing their attempts to reproduce research findings, despite its importance.
Professor Holcombe, from the University's School of Psychology, has helped devise a new initiative to solve the problem. He will be one of two editors of registered replication reports to be coordinated by the journal Perspectives on Psychological Science, published by peak professional society, the Association for Psychological Science.
"The crisis in published replications of psychological and other scientific research is a result of scientists receiving greater encouragement, both from their institutions and from journals, to produce new findings, instead of attempting to reproduce others' results."
"So there is typically an uphill battle to get a replication study published as well as the fear you may antagonise other researchers if your results contradict theirs."
There are several innovative aspects to the journal's approach. One is that, where possible, the original research authors will be involved so that the replicating authors can get their support and ensure their method is as close as possible to the original study.
"Another difference is that researchers will commit in advance to how they will analyse their results instead of being free to analyse their findings until they find a significant but possibly spurious result," Professor Holcombe said.
"The researchers will be assured that their results will be published even if they do not obtain a statistically significant result."
Before publication the method will be made public and laboratories will be invited to join the replication attempt, then all the results will be posted.
"Most importantly, the results will not be presented as a 'successful replication' or 'failed replication'. It is very rare that a set of results is definitive by itself - that is the whole point of what we are undertaking."
"Instead we will analyse and summarise to what extent all the replications, considered together, reproduce the original findings or differ from them. This is essential information for understanding those original findings and their applications."
An example of the reproduction problem is American research done in the 1990s that described 'verbal overshadowing', where immediately verbally describing an object appears to make recall harder, not easier. The finding continues to be highly influential but the original researcher, Jonathan Schooler, has reported on the difficulty of reproducing his results, undermining his own faith in the original findings.
Professor Holcombe's editorial role is the culmination of 10 years of ongoing effort to improve the transparency and accountability of psychological research.
"It started with an unsuccessful attempt to persuade journals to post comments on published papers to allow flaws and missed connections to come to light earlier."
"So I am excited, 10 years on, to be part of this initiative because I think it can overcome the main obstacles to replication studies being undertaken and published."
|Follow University of Sydney Media on Twitter|
Media enquiries: Verity Leatherdale, 02 9351 4312, 0403 067 342, firstname.lastname@example.org