In a world where the number of scientific publications is growing dramatically and competition over publication is intense, it is more important than ever to ask which research findings we can in fact rely on. Research itself is in need of scrutiny.

Dreber Almenberg is a professor of economics at Stockholm School of Economics, and is researching how the research process can be improved to increase the credibility of scientific findings. The field is called metascience, and has seen an upswing in recent years.

“Since I began researching in this field more than ten years ago there has been a steadily increasing awareness of these issues.”

One way of testing whether findings are sound is to carry out a replication project, i.e. perform the study again. Dreber Almenberg has redone many experiments in the fields of psychology and economics and reached a striking conclusion. Only about half of the findings could be replicated. This means that the other half consists of false positives, i.e. results where the researchers have thought there was an effect but nothing actually happens. There may be several reasons for this.

“Researchers enjoy breaking new ground, and in attempting to do so, may fool themselves. There is also pressure to be published in reputable journals. We can also see a lack of statistical training. Deliberate falsification of research findings is rare, however.”

Focusing on methods instead of results

There are ways of reducing the number of false positives. One effective method is to introduce preregistered analysis plans. Before the researcher begins gathering data they must clearly define how the analysis will be performed, and what tests will be carried out.

“Most researchers are ethical people who want to conduct high-quality research. But it’s easy to fall in love with your hypothesis, and this may influence the way data is interpreted. The preregistration approach reduces the risk of the researcher choosing findings that best support a hypothesis even though most tests do not support it,” Dreber Almenberg explains.

Some academic journals have already chosen this publication format. Instead of waiting for the completed findings, they rigorously scrutinize the research plan. If the plan is given the thumbs up, the journal agrees to publish the study whatever its findings.

“For many studies, this is the right focus – evaluating the issues addressed by the research and its design instead of its findings.”

We’re constantly learning new ways of improving our methods, which is what makes research so exciting.

Variation in research design

The choice of research design is an important factor. Dreber Almenberg is currently studying how outcomes can differ so much even when researchers use the same material.

Her study is based on data from SOEP (Socio-Economic Panel), where a representative cross-section of German households contributes information on topics such as attitudes, labor market situation, and family composition.

The researchers are asked to study the effect on decision making and attitude to equality of having daughters as compared with sons. Earlier research based on responses from members of the U.S. Congress has shown that fathers with daughters may have a different view of gender roles, which may in turn influence their political decisions.

There are many ways of analyzing these data, and depending on which researcher you ask, you will likely get different results.”

Another project involves examining how findings from historical studies hold up when new data are added or when analytical methods change. This may be relevant when an experiment cannot be replicated, for instance when researchers are attempting to ascertain how major historical events impact the contemporary era.

“We can’t reunite Germany again to test the effects of reunification, but we can see whether the findings of different studies on German reunification hold when we add more recent data. Or look farther back in time – there are studies on whether or not societies used plows, relating their historical plough use to present-day gender norms. We add contemporary data and see whether the findings remain.”

Better research through collaboration

One successful strategy may also be closer collaboration between researchers. Larger team science studies tend to be more statistically robust and offer greater scope to generalize their findings.

“Instead of a small specific study that has something interesting to say but whose findings are not very robust, a larger study involving more researchers can say something much more reliable.”

We live in a time when the concept of truth is often debated. This makes our research all the more important, as Dreber Almenberg explains:

“There is a risk that the public will also start to disbelieve research, with an accompanying debate over ‘fake science.’ But our findings suggest there is nothing wrong with the scientific method. A certain proportion of false positives is unavoidable, but a proportion as high as we have seen can be remedied. The process of eliminating false positives from scientific literature can be speeded up. It’s vital that people and organizations can make decisions based on credible research results.”

Text Nils Johan Tjärnlund
Translation Maxwell Arding
Photo Magnus Bergström