Project

Back to overview

Reverse-Bayes Design and Analysis of Replication Studies

English title Reverse-Bayes Design and Analysis of Replication Studies
Applicant Held Leonhard
Number 189295
Funding scheme Project funding (Div. I-III)
Research institution Institut für Epidemiologie, Biostatistik und Prävention Universität Zürich
Institution of higher education University of Zurich - ZH
Main discipline Mathematics
Start/End 01.11.2019 - 31.10.2023
Approved amount 469'141.00
Show all

All Disciplines (2)

Discipline
Mathematics
Methods of Epidemiology and Preventive Medicine

Keywords (6)

Sample Size; P-value; Replication Studies; Reverse-Bayes; Power; Bayes Factor

Lay Summary (German)

Lead
Die Replizierbarkeit von Forschungsergebnissen ist ein Eckpfeiler derwissenschaftlichen Methode. In den empirischen Wissenschaftenexistiert aber kein einheitlicher Standard, um den Erfolg einerReplikationstudie statistisch zu bewerten. Der Reverse-Bayes Methode,bei der der Satz von Bayes rückwärts angewandt wird, stellt einenneuartigen und vielversprechenden Ansatz dar, um Replikationstudienzu planen und auszuwerten.
Lay summary
   
Inhalt und Ziel des Forschungsprojekts

Wir werden die Reverse-Bayes Methode auf die wichtigsten Studientypen
der empirischen Fuschung anwenden. Neben Überlegenheitsstudien gehören
dazu Äquivalenzstudien und Nicht-Unterlegenheitsstudien. Desweiteren
werden wir die Reverse-Bayes Methode so erweitern, dass sie auch für
Replikationsstudien mit geringer Fallzahl verwendet werden kann. Zur
erfolgreichen Planung von Replikationsstudien werden wir Methoden
entwickeln, mit Hilfe derer man die notwendige Fallzahl bestimmen kann
um einen Replikationserfolg mit vorgegebener Power zu gewährleisten.
Dabei werden auch Zwischenauswertungen erlaubt sein, um
Replikationsstudien vorzeitig beenden zu können.

Die Reverse-Bayes Methode wurde bisher auf klassische statistische
Kenngrössen angewendet (Schätzwerte, Konfidenzintervalle und
p-Werte). Alternativ werden wir die Methode mit sogenannten
Bayes-Faktoren kombinieren, die eine direkte Quantifizierung der
statistischen Evidenz erlauben. Auch hier werden wir Verfahren sowohl
zur Planung als auch zur Auswertung entwickeln und mit den
entsprechenden Reverse-Bayes Methoden für klassische Kenngrössen
vergleichen.

Wissenschaftlicher und gesellschaftlicher Kontext des Forschungsprojektes


In unserer Forschung werden wir neue statistische Methoden entwickeln,
um Replikationsstudien besser planen und auszuwerten. Damit
antworten wir auf die steigende Bedeutung von Replikationsstudien in
allen empirischen Wissenschaften und tragen dazu bei, die
Reproduzierbarkeit von Forschungsergebnissen zu verbessern und das
Vertrauen der Öffentlichkeit in die Wissenschaft zu stärken.



Direct link to Lay Summary Last update: 01.10.2019

Responsible applicant and co-applicants

Employees

Project partner

Associated projects

Number Title Start Funding scheme
159715 Objective Bayesian model selection in generalized regression 01.05.2015 Project funding (Div. I-III)

Abstract

Replicability of research findings is crucial to the credibility of all empirical domains of science. As a consequence of the so-called replication crisis, the past years have witnessed increasing interest in large-scale replication projects. Such efforts help assess to what extent claims of new discoveries can be confirmed in independent replication studies whose procedures are as closely matched to the original studies as possible.However, there is no established standard for the statistical evaluation of replication success. Traditional statistical significance of the replication study is often used as a criterion, but significance alone does not take the effect sizes of the original and replication study into account and can easily lead to conclusions opposite to what the evidence warrants. Meta-analytic effect estimates can also be computed, which, however, treat the original and replication study as exchangeable, a questionable assumption as claims of new discoveries are often subject to publication and other biases.Recently a new definition of replication success has been developed based on a reverse-Bayes approach. The original study result is challenged with a sceptical prior such that the corresponding posterior distribution is no longer ’significant’. The goal of the replication study is to persuade the sceptic by showing that his prior is unrealistic. If this is the case, replication success is achieved and the original study result is confirmed. This novel definition of replication success has attractive properties, taking into account both significance and effect size of both the original and replication study. To avoid the dichotomization of replication studies into successful yes/no, a quantitative measure of replication success is suggested, the sceptical p-value.The proposed approach is so far developed only for a normally distributed test statistic based on a univariate point null hypothesis. We will extend these ideas further and apply them to equivalence studies, multivariate outcomes and small samples. We will also consider the reverse-Bayes approach in a Bayesian hypothesis testing framework based on Bayes factors. This leads to the sceptical Bayes factor, the reverse-Bayes analogue of the replication Bayes factor. The properties of this new quantitative measure of replication success will be thoroughly investigated and compared with the sceptical p-value. Statistical power is crucial to assess the reliability of science. Appropriate design of a replication study is key to tackle the replication crisis as many such studies are currently severely under-powered, even by traditional standards. We will develop novel ways to calculate the sample size of a replication study based on the sceptical p-value and the sceptical Bayes factor. The methodology will ensure that the study size of the replication study is capable to achieve replication success in the light of the original study with appropriate power. Furthermore, we will investigate the applicability of sequential designs to replication studies. This will enable researchers to stop a replication study at interim and will thus potentially save considerable amounts of human and financial resources.Science would proceed more efficiently if new statistical methods can be aligned with scientific needs and practice. The research plan described in this proposal aims to achieve exactly this: to develop more rigorous statistical standards tailored to an increasing demand of replication studies conducted in all empirical fields of science.
-