“Published Scientific Literature Is Systematically Biased Toward Positive Findings,” Says Researcher Michal Smetana

Your research often relies on experimental methods. In practice, what does it mean to design an experiment so that someone else can replicate it?

The key is to provide detailed information within the publication about the experimental design, the exact procedures used to recruit participants, and the methods applied to analyse the collected data. In the case of the survey experiments I conduct, this also includes sharing the complete questionnaire in all of its experimental variations. It is equally important to specify the criteria used to exclude participants from the sample. Today, it should also be standard practice to make both the dataset and the replication code available, allowing others to reproduce the statistical analysis exactly as it was performed by the original authors.

How often do you rely on datasets or code published by other researchers when starting a new research project? And when you do find such data, what condition are they usually in?

Not very often. Most of my research is based on original experimental designs and the testing of new hypotheses. That said, I recently launched a project with colleagues from Charles University and the University of Oxford in which we are replicating a large number of well-known experiments on attitudes towards nuclear weapons that were originally conducted before the outbreak of the war in Ukraine.

As for the quality of the underlying materials, datasets, and replication code, the experience has been quite disappointing. Even though these studies were published by prominent authors in leading journals, we repeatedly encountered unavailable datasets, incomplete questionnaires, problematic coding, missing information, the omission of important results, and reporting errors. In some cases, we were able to resolve these issues through direct communication with the authors. In others, however, the shortcomings were significant enough that we could not include the studies in our replication project. I suspect the situation is even more challenging for studies published in less rigorous journals.

Do you think that greater data availability has changed the way research is conducted today in international relations and security studies?

I would say that, as a field, we are still adopting these practices rather slowly. There has definitely been noticeable progress in experimental political science over the past few years, but in many other subfields the shift towards greater openness and data sharing is not yet as evident.

Open science is built on sharing knowledge and international collaboration. Do you think the current geopolitical tensions—such as rivalry between major powers or growing concerns about research security—are affecting how openly researchers share their data?

Certainly, but I think this applies more to the natural sciences and technical disciplines. In my own field, I don't really see this pressure affecting data sharing. My approach is to make the full dataset and replication code openly available as soon as the article is published.

Openly available data do more than enable new research—they also make it possible to revisit published studies and verify their findings. This has been a major focus of your work for many years. What exactly does replicating a study involve?

The primary goal is to repeat an existing study to verify that the effect identified in the original research is real and that other researchers can observe the same result by following the original methodology. While discussions often focus on fabricated or selectively reported findings, many studies fail to replicate simply because the original authors made unintentional mistakes or because the reported effect was a false positive that occurred by chance. This problem is further amplified by what is known as publication bias—the tendency for journals to publish positive findings while leaving null results unpublished. As a consequence, the published scientific literature is systematically biased toward positive findings. This is one of the main reasons why replication studies are so important and why practices such as preregistering an analysis plan before data collection have become increasingly common.

That said, the main reason my team and I conduct replication studies is to examine how people's attitudes change over time in response to major international events. For example, following Russia's invasion of Ukraine in February 2022, many argued that Russia's nuclear threats had significantly weakened the international norm against the use of nuclear weapons—the so-called nuclear taboo. By replicating experiments conducted before the invasion, we are investigating whether changes in individual attitudes actually support this claim.

As part of the project, we first established transparent criteria for selecting studies to include in our replication pool. We then collected the original study materials, datasets, and replication code. We are now replicating each selected study according to the original methodology and statistically testing whether the observed changes correspond to our preregistered hypotheses.

To what extent is open access to research data a prerequisite for independently verifying studies like these? Would replication research even be possible without open science?

In principle, replication is possible whenever you have access to the necessary materials. In other words, if the original author shares them with me privately, I can still conduct the replication even if they are not publicly available. In practice, however, this is often problematic. Authors are not always willing to share their data and replication code, for a variety of reasons. Openly sharing research data therefore plays a crucial role in improving the transparency and credibility of science as a whole.

What has surprised you the most when replicating published studies? Are there any recurring issues—whether methodological errors, differences in analytical approaches, or something else—that you keep encountering across different research projects?

Differences in analytical approaches are not the main issue. The real challenge lies in the lack of consistent standards for what should be shared and how. Many authors formally meet a journal's requirements by uploading their data and replication code, but the package often lacks a codebook or any meaningful documentation explaining what the variables in the dataset actually represent. In other cases, key parts of the questionnaire are missing, making it impossible to replicate the study exactly as it was originally conducted. We have even encountered cases where the uploaded dataset revealed the existence of additional experimental groups that were never mentioned in the published article. That is a highly problematic research practice.

When replication studies show that some findings are less robust than originally thought, this can sometimes be perceived negatively—even as criticism of the original authors. But isn't the ability to continuously verify and reassess scientific findings one of the greatest strengths of science?

For me, absolutely. In quantitative research, replication and the verification of results should be fundamental pillars of the scientific process, and they certainly should not be viewed as a form of attack on the original authors.

That said, I would also like to emphasize that scientific research encompasses approaches based on very different epistemological assumptions from those underlying the experimental research I conduct. Many interpretive approaches in the social sciences and humanities are not built on formal hypothesis testing, quantitative data collection, or direct replicability. This does not make them any less valuable than the type of research I do. They are simply grounded in a different way of understanding the world, and applying the same expectations of “replicability” to them makes little sense given their distinct ontological and epistemological foundations.

What does the current wave of replication studies tell us about the state of modern science—or, more specifically, about the field in which you work?

On the one hand, it reveals a great deal about the shortcomings of long-standing incentives and practices within scientific research. In some cases, these have led to the publication of studies based on fabricated, manipulated, or selectively reported data, or to insufficient verification of whether reported effects are genuine rather than the result of chance.

On the other hand, there is also a positive side to this story. The scientific community itself has developed self-correcting mechanisms to address these issues. The fact that we have been able to identify and define the replication crisis—and are actively developing tools and practices to reduce these problems in the future—is, in itself, a very encouraging sign.

Generative AI has the potential to significantly accelerate many aspects of scientific research. Do you think its continued development will also increase the need for independent verification of results and replication studies?

Yes. While artificial intelligence offers researchers many valuable tools, it also brings a number of significant challenges. Many journal editors are already reporting a growing influx of AI-generated manuscripts that fail to meet scientific standards while placing additional strain on an already overburdened peer-review system. In this respect, we are currently in a highly dynamic period. The research ecosystem has not yet fully adapted to this technological shift, and the scientific community is still working to establish new shared rules and principles for how AI should be used in research.