Could you be more specific? Preregistering analysis methods to examine the hypotheses for Transitions in Depression (TRANS-ID) Recovery

• This case study is one of the three winning submissions for the Open Research Award, which will be celebrated on 22 October 2020.

My experience with open research practices centres on the task of translating the ‘grand’ research questions and hypotheses from a grant proposal, to the ‘specific’, testable questions for an analysis plan. A large portion of my PhD trajectory has been dedicated to collecting data that was designed to answer a particular research question: Can we find early warning signals before symptom improvements in depression? Briefly: we collected intensive longitudinal data of mood and depressive symptoms from people seeking psychological treatment for depression, with the expectation that they were more likely to show large, discontinuous improvements. The hypothesis to be tested is that certain statistical patterns emerge in the daily measurements (e.g., variance and autocorrelation rise) that may signal that a symptom shift is imminent.

Although the research question and hypotheses were broadly specified when the project was designed, many particular decisions still had to be made about how we would define and analyse crucial components of the study. Particularly challenging was the fact that, although my project was based on an earlier study, that was only an n=1 case study of a patient that relapsed, rather than remitted from depression. This meant that I could not simply repeat the steps from that study, and the methodological details of my analysis still had to be adapted to work for a larger sample (42 repeated case studies), as well as for a different clinical process (depressive symptom improvement will look different over time than a depressive relapse). In short, the analysis choices I had to make could only be guided by the existing work and background theory to a certain extent.

Because our study is a somewhat complex hybrid between exploratory and confirmatory work, the preregistration was an important opportunity for us to show that we have taken care in devising our analysis steps. In the context of a preregistration, we could write down the hypotheses that will be tested with the method that we believe, a priori, to be theoretically most likely to work. Additionally, we have planned robustness checks and extensive descriptions of the data to provide further information about our sample, and the preregistration is a way to demonstrate that the additional tests and explorations are pre-designed and not a result of post hoc reasoning in response to the results. Essentially, the preregistration offered us a way to timestamp what we believed, to the best of our theoretical knowledge, to be an optimal analysis plan before seeing the data, while also serving as a testament of our good intent to give a thorough description of our data.

In the past few years (while data was being collected), we spent a lot of time making carefully weighed considerations and analysis choices, so it was not much extra work to write them down and publish them online. On the project OSF page, I have provided 1) a detailed description of the study methods, 2) a brief methods piece describing an adapted version of the Reliable Change Index for use when change can happen over multiple time points, 3a) the original (Dutch) experience sampling questionnaires and 3b) an English translation including the measured constructs, and finally, 4) a preregistration of our analysis plan. Numbers 1 and 2 are also available on ResearchGate and, to my great joy, the detailed study methods (1) have already been cited in a recent review. In conclusion, my experience with open research practices is only positive, it has brought extra clarity and thoroughness to my research work; a practice I hope to continue in the future!

References and further information:

• The full project in OSF
• The preregistration