Multi-Analyst Reanalysis of IPD on Esketamine and Suicidality / SHARE-CTD datathon 2

Abstract

Multi-Analyst Reanalysis of IPD on Esketamine and Suicidality (Team 1, 2, 3) SHARE-CTD study group Lay summary This study explores if and how different analytical choices may affect the results of meta-analyses using individual participant data (IPD) from clinical trials. Three independent research teams will separately analyse data on esketamine’s effect on suicidal thoughts in depression, shared through the Yale Open Data Access (YODA) project. By comparing their methods, results, and interpretations, the study will test whether analytic flexibility changes scientific conclusions. The project promotes transparency, reproducibility, and more trustworthy clinical research. Share CTD is funded by the European Union under the grant agreement Horizon-MSCA.2022-DN 101120360. Scientific abstract Background: The reproducibility of meta-analyses can be affected by analytical flexibility, especially in individual participant data (IPD) meta-analyses, which require complex data management and methodological decisions. The effect of such flexibility on outcomes remains underexplored. Objective: To assess the reproducibility of independent IPD meta-analyses evaluating esketamine’s efficacy in reducing suicidality among patients with major depressive disorder. Study Design: Three independent teams from the SHARE-CTD network will conduct parallel IPD meta-analyses using a common dataset, working in isolation until a collaborative datathon, where results and methods will be compared. Participants: IPD from all esketamine clinical trials available via the YODA project, encompassing patients with major depressive disorder. Primary and Secondary Outcome Measure(s): Primary: Change in suicidality, assessed using standardised scales or adverse event reporting. Secondary: Variability in analytical decisions, observed results, and conclusions across teams. Statistical Analysis: Teams will define and preregister their own analysis plans. Reproducibility will be evaluated by comparing methodological choices, quantitative results (effect sizes, p-values, confidence intervals), and interpretations. Based on the forking path diagram, all possible combinations of analytic decisions will be run in a vibration of effects (VoE) exercise.   SHARE-CTD study group SHARE-CTD (Sharing and Re-using clinical trial data to maximise impact) is a doctoral network funded by the European Union that involves 11 principal investigators and 16 academic and non-academic partners. This project was motivated by the growing movement toward open science and by the need to define good practices for the preparation and use of clinical trial data. Its aim is to train a new generation of biomedical researchers with a deep understanding of the processes, values, and benefits of clinical trial data sharing. To gain a comprehensive understanding, biomedical researchers must be trained in areas such as data science, trial regulation, meta-research, as well as ethical, legal, and social issues. Acronyms EBM: Evidence Based Medicine IPD: Individual participant data OSF: Open Science Framework PICOS: Patients Interventions Comparators Outcomes Study design SHARE-CTD: Sharing and Re-using clinical trial data to maximise impact VoE: Vibration of effects YODA Project: Yale University Open Data Access Project   Background Meta-analyses are often placed at the top of the evidence-based medicine (EBM) hierarchy. Yet, they are published in large numbers, frequently overlapping or redundant, and sometimes producing contradictory results [1]. Such inconsistencies are concerning, given that meta-analyses are expected to be exhaustive and reproducible syntheses of the literature. However, they are retrospective exercises that involve many degrees of freedom, making them prone to vibration of effects (VoE) [2]. This phenomenon has been observed using multiverse analyses of 1/ head-to-head meta-analyses [3], 2/ indirect comparisons [4], and—though to a lesser extent—3/ individual participant data (IPD) meta-analyses [5]. However, multiverse meta-analyses rely on many meta-analyses to evaluate result variability, often taking an overly agnostic approach, with sometimes very arbitrary analytical choices. Less is known about how specific analytical scenarios and decisions—especially in data management, statistical modelling, and interpretation—can shape findings within a single research question. IPD meta-analyses are a particularly relevant context for exploring this phenomenon, as they involve numerous analytical choices—not only in defining the research question, selecting inclusion criteria, and analyzing outcomes, but also during data management. Because IPD must be harmonized across studies, and outcomes are sometimes reconstructed when they were not originally collected for the specific purpose of the meta-analysis, these additional steps introduce further degrees of freedom that may influence results. To explore this question, we selected the case study of esketamine’s effect on suicidal ideation, drawing on our prior experience with this topic [6]. This clinically significant outcome exemplifies the many degrees of freedom involved in IPD meta-analyses making it an ideal example to investigate how analytical decisions can influence results. The SHARE-CTD datathons provide a unique opportunity to investigate these issues in a many-analyst framework [7], helping to better understand the role of analytical flexibility in IPD meta-analysis. Objective To explore the reproducibility of independent analyses conducted by three teams performing an individual participant data (IPD) meta-analysis assessing the efficacy of esketamine in reducing suicidality among patients with major depressive disorder. Clinical trial data will be sourced via the YODA project. Reproducibility of those analyses will be assessed along three key dimensions, as defined by Fanelli et al. [8]: • Methods reproducibility (consistency in analytic procedures), • Results reproducibility (consistency in quantitative outcomes), and • Inferential reproducibility (consistency in interpretation and conclusions). Structure and Recruitment of Analysis Teams • Composition: Three independent teams from the SHARE-CTD network, each consisting of 3–4 analysts (including at least one with biostatistical expertise), and one senior researcher with domain expertise in psychiatry or psychopharmacology (IAC, FN, MP). • Rationale: IPD meta-analyses demand diverse competencies spanning data management, statistical methodology, and clinical knowledge. Senior researchers will serve as independent advisors—providing methodological and clinical input—but will not engage in data handling or analysis. • Recruitment: All analysts will be SHARE-CTD doctoral candidates. Teams must commit to working independently until the datathon in February 2026. Roles, Responsibilities, and Process a. Research Question "Does esketamine reduce suicidality in patients with depression?" This research question is intentionally broad to allow for variation in analytic strategies (see Box1). It addresses a clinically relevant issue, given esketamine’s U.S. approval for patients depression and suicidal ideation. The dataset offers substantial analytical flexibility, with potential variability in: - Population: e.g., inclusion of patients with or without suicidal ideation at baseline; baseline depression; history of suicide - Intervention: different dosing regimens - Outcome: suicidality assessed through specific scales, standardisation to a Z-score or adverse event reporting - Comparator: placebo vs. active comparators (e.g., quetiapine) - Study design: double-blind, single-blind, or open-label trials Box 1: source of analytical variability (not exhaustive) Furthermore, members of our consortium (IAC and FN) possesses prior experience with this dataset [6]. Notably, the question pertaining to suicidality necessitated adaptations and modifications to the original protocol, underscoring the practical significance of the present study. Despite these degrees of freedom, feasibility is enhanced by the homogeneity of the dataset: all studies originate from a single manufacturer and drug development program. b. Code of Conduct • All participants will sign a YODA-compliant data use agreement, including provisions for confidentiality, data security, and ethical compliance. • Each team will preregister its statistical analysis plan (SAP) on the Open Science Framework (OSF) under embargo (to preserve independence) prior to accessing the data or initiating any analysis. c. Reporting Requirements • Teams will prepare a comprehensive research report detailing their SAP (including deviations), analytic decisions, results, commented code, and final conclusions. • All code and methodological documentation will be made openly accessible upon publication. d. Inclusion of Analyses • All analyses submitted by the three teams will be included in the final synthesis, regardless of their methodology or findings. e. Authorship and Attribution • Results of the datathon will be published in a research article. • Analysts’ names will be publicly linked to their team’s work, with individual contributions denoted by initials. • All co-analysts will be included as co-authors of the final publication, provided they also meet the other criteria (approval, accountability) for being listed as an author, with contributions described using the CRediT taxonomy. f. Update Rights • Teams may update their analyses before the February 2026 datathon, provided all changes are transparently documented. • During the datathon, analyses will be finalised, presented, compared, and discussed with other teams and SHARE-CTD principal investigators. No further modifications will be permitted after this meeting. g. Timeline • Anticipated data access and start of analyses: December 2025 • Datathon and synthesis workshop: February 2026 Data and Materials • Datasets: IPD from all esketamine trials available via the YODA project, accompanied by study protocols, variable dictionaries, and relevant metadata. All available studies are listed here: https://yoda.yale.edu/trials-by-generic-name/ Identifier Enrollment NCT04599855 477 NCT04338321 676 NCT03434041 252 NCT02497287 802 NCT02493868 719 NCT01998958 108 NCT02918318 202 NCT01627782 68 NCT01640080 30 NCT03097133 230 NCT02133001 68 NCT03039192 226 NCT02422186 139 NCT02418585 236 NCT02417064 346 Table 1: available studies • Training and Access: All teams must complete the YODA training and sign the data use agreement before receiving access. • Analytic Task: Each team will define and justify their PICOS criteria and analytic methods in their preregistered SAP. Independence of Analyses • Before the Datathon: Teams must work in isolation, with no inter-team communication regarding data analysis until the February 2026 datathon (9-13 january). • During the Datathon: Teams will present their results, compare methods, identify discrepancies, and collaboratively assess methodological variability. Sharing and Processing Results Each team will provide the lead team with: • Final results • Cleaned and annotated code (or step-by-step descriptions for GUI-based software) • A narrative linking findings to conclusions By the end of the datathon, the lead team will archive the (aggregated) data and results with restricted access. All materials will be publicly released afterward as part of post-datathon activities. Meta-Research and Synthesis Strategy Submitted analyses will be compared along three dimensions of reproducibility: • Methods Reproducibility: o All methodological choices will be qualitatively analyzed and described, for instance using a “forking path diagram.” This will include:  Defined PICOS (Population, Intervention, Comparator, Outcomes, Study design);  Statistical analysis plans and methods;  Tools and software used;  Any additional options that may be identified. • Results Reproducibility: o Consistency across analyses will be assessed using the following approach. Each analysis will be summarized in terms of (i) its conclusion (positive or negative), (ii) p-value, (iii) effect size (with outcome details), and (iv) any deviations from the initial protocol regarding the primary outcome. (v) uncertainty estimate (e.g., Confidence Interval). o Based on the forking path diagram, all possible combinations of analytic decisions will be run in a vibration of effects (VoE) exercise. A volcano plot will display effect sizes on the x-axis and p-values on the y-axis. We will quantify the VoE by calculating:  The range of p-values (RP);  The difference between the 99th and 1st percentile of the –log(p-value);  The range of effect sizes;  The Janus effect (if the 1st and 99th percentiles of the effect size are in opposite directions. • Inferential Reproducibility: o Differences in interpretation and clinical implications will be explored collaboratively during the datathon. As interpretation of a meta-analysis involves clinical judgment and cannot rely solely on quantitative metrics, a structured discussion among co-analysts and SHARE-CTD PIs will take place. This discussion, grounded in both quantitative results and expert clinical perspectives, will assess whether observed differences in results could meaningfully lead to different conclusions or clinical recommendations Transparency and FAIR Principles • All code will be shared in accordance with the FAIR principles (Findable, Accessible, Interoperable, Reusable). • Clear guidance will be provided for researchers seeking to access the data. • Materials will be hosted on open-access repositories (e.g., OSF). • This protocol will be publicly available (no embargo) via the OSF as the master project document. • The study’s conduct and reporting will adhere to the principles outlined in a recent consensus-based framework for multi-analyst studies [9]. • The teams will be asked to provide a description of usage of and prompts for artificial intelligence for this project, in particular drafting and revision of the protocol and statistical analysis plan, statistical code, presentation. References 1. Ioannidis JP: The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q 2016, 94(3):485-514. 2. Vinatier C, Hoffmann S, Patel C, DeVito NJ, Cristea IA, Tierney B, Ioannidis JPA, Naudet F: What is the vibration of effects? BMJ Evid Based Med 2025, 30(1):61-65. 3. El Bahri M, Wang X, Biaggi T, Falissard B, Naudet F, Barry C: A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects. J Clin Epidemiol 2022, 152:140-150. 4. Vinatier C, Palpacuer C, Scanff A, Naudet F: Vibration of effects resulting from treatment selection in mixed-treatment comparisons: a multiverse analysis on network meta-analyses of antidepressants in major depressive disorder. BMJ Evid Based Med 2024, 29(5):324-332. 5. Gouraud H, Wallach JD, Boussageon R, Ross JS, Naudet F: Vibration of effect in more than 16 000 pooled analyses of individual participant data from 12 randomised controlled trials comparing canagliflozin and placebo for type 2 diabetes mellitus: multiverse analysis. BMJ Med 2022, 1(1):e000154. 6. Naudet F, Pellen C, Fodor LA, Gastaldon C, Barbui C, Turner EH, Le Pabic E, Cristea IA: Efficacy and safety of esketamine for “treatment resistant depression”: registered report for a Systematic Review with an Individual Patient Data Meta-analysis of Randomized, Double-Blind, Placebo-Controlled Trials. medRxiv 2025:2025.2007.2017.25331696. 7. Mansmann U, Locher C, Prasser F, Weissgerber T, Sax U, Posch M, Decullier E, Cristea IA, Debray TPA, Held L et al: Implementing clinical trial data sharing requires training a new generation of biomedical researchers. Nature Medicine 2023, 29(2):298-301. 8. Goodman SN, Fanelli D, Ioannidis JP: What does research reproducibility mean? Sci Transl Med 2016, 8(341):341ps312. 9. Aczel B, Szaszi B, Nilsonne G, van den Akker OR, Albers CJ, van Assen MALM, Bastiaansen JA, Benjamin D, Boehm U, Botvinik-Nezer R et al: Consensus-based guidance for conducting and reporting multi-analyst studies. eLife 2021, 10:e72185.

Topics & conditions

Detected automatically by Psychedelic Commons from the title and abstract, not labels from the source. They may be incomplete.

Keywords