Conducting my Master thesis research in a transparent and reproducible way

Open Research objectives /practices

The study materials (i.e., the full questionnaire and all stimuli materials) are publicly available on the OSF. Moreover, a detailed public pre-registration ensured the entire research (process, methods, and expectations) was conducted and is presented in a transparent fashion. This also increases reproducibility.

Introduction

I heard of Open Science when doing my B.Sc. in Germany and was so convinced by many aspects of it that I committed to always pre-register and share studies that I conduct as the main person responsible. So naturally, I did this with my master thesis research. This means I made a public project on the OSF that includes detailed pre-registration, all materials (questionnaires, stimuli) that I used, and later added specifications of the deviations from my original approach (see below) as I went on with the research.

Motivation

Open Science practices generally make research more credible and useful for the public and the scientific community. Other researchers can use my materials, comment on them, and see exactly what I did. Researchers or the public might discover the study on the OSF, which means that Open Science can increase the number of people for whom this research might be relevant.

Finally, I also benefit from it because I force myself to rigorously plan my studies in much more detail, starting with distinguishing confirmatory from exploratory hypotheses, over the research design, to data analysis. This gives me more confidence in the credibility of my work.

Lessons learned

The OSF and all the other Open Science resources available make it incredibly easy to get started with doing research more transparently and reproducibly, thus they are my biggest supporting factors.

Three noteworthy challenges were:

First, sometimes (always) things do not go according to plan: I planned my study as a SONA study, but had to move it to MTurk due to the approaching end of the academic year after my pre-registration. This made several adjustments necessary, which I described and justified on the OSF (i.e., attention checks, data screening, and sampling approach).

Second, pre-registering data analysis scripts is really difficult (for me). I would really like to pre-register my analysis script beforehand to see if my analysis plan works as intended, but this means I have to simulate data and write a script that includes all possible forks in my data analysis plan up front – and this is hard! I want to take on this challenge for my next projects, because computational reproducibility is important as well.

Lastly, regulations for openly publishing data on the OSF are pretty complex. Working through the very important data management rules and protecting participant’s privacy is of course key, therefore I am still learning what it possible in terms of sharing data on repositories such as the OSF.

URLs, references and further information

Project page on OSF:

• https://osf.io/2ewuv/

Pre-registration on OSF:

• https://osf.io/2ewuv/registrations