1  The Problem of Analytic Multiplicity

A published result usually reports one estimate from one analysis. Behind that single number sits a branching set of decisions: how the outcome and exposure were coded, which covariates entered the model, how missing values were handled, which observations met inclusion criteria, and what functional form linked predictors to outcome. Each decision has several defensible answers, and different combinations can yield different estimates. The distribution of results across all defensible analyses is a property of the study, and describing it is the aim of this book.

1.1 Same data, different answers

When many independent teams analyse the same data to answer the same question, their estimates spread out. In a coordinated study of whether football referees give more red cards to darker-skinned players, twenty-nine teams reached conclusions ranging from no association to a strong one, all from a single dataset (Silberzahn et al. 2018). A later effort on immigration and preferences for social policy found comparable dispersion across seventy-three teams (Breznau et al. 2022). The variation came from analytic choices, each one reasonable on its own.

Observational epidemiology faces the same exposure. The vibration-of-effects framework fit thousands of models that differed only in their adjustment sets and showed that both the effect size and its p-value can swing widely across those models, sometimes reversing sign (Patel et al. 2015). Applying the framework to established cohorts confirmed that many reported associations are fragile to defensible changes in specification (Klau et al. 2021).

1.2 The response

Two complementary methods make the decision space visible. Multiverse analysis enumerates the defensible specifications and reports the full set of estimates (Steegen et al. 2016). Vibration-of-effects analysis turns that lens on one target association and quantifies how much it moves as the model changes (Patel et al. 2015). Specification-curve analysis orders the estimates and tests them as a set (Simonsohn et al. 2020).

This book implements those methods in R through the companion package forkflow. The workflow is five verbs: enumerate the specifications, fit a model across the whole grid, summarise the resulting distribution, visualise it, and report it reproducibly. The chapters that follow build each verb in turn, close with a worked survival-analysis example, and end with practical guidance.

Breznau, Nate et al. 2022. “Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty.” Proceedings of the National Academy of Sciences 119 (44).

Klau, Simon, Sabine Hoffmann, Chirag J. Patel, John P. A. Ioannidis, and Anne-Laure Boulesteix. 2021. “Examining the Robustness of Observational Associations to Model, Measurement and Sampling Uncertainty with the Vibration of Effects Framework.” International Journal of Epidemiology 50 (1): 266–78.

Patel, Chirag J., Belinda Burford, and John P. A. Ioannidis. 2015. “Assessment of Vibration of Effects Due to Model Specification Can Demonstrate the Instability of Observational Associations.” Journal of Clinical Epidemiology 68 (9): 1046–58.

Silberzahn, Raphael et al. 2018. “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.” Advances in Methods and Practices in Psychological Science 1 (3): 337–56.

Simonsohn, Uri, Joseph P. Simmons, and Leif D. Nelson. 2020. “Specification Curve Analysis.” Nature Human Behaviour 4: 1208–14.

Steegen, Sara, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. 2016. “Increasing Transparency Through a Multiverse Analysis.” Perspectives on Psychological Science 11 (5): 702–12.