PROBAST

Quality assessment of primary studies is a crucial component of any systematic review process. It is important that reviewers and users of systematic reviews are able to assess the potential impact of sources of bias in the included studies upon the results and conclusions of the review. Systematic review of prediction studies, in particular prediction modelling studies, is a relatively new and evolving area. In comparison to systematic reviews of treatments and diagnostic tests critical appraisal tools and review methods in this area are not yet well developed.

The PROBAST project brought together experts in the field of prediction research, experienced systematic reviewers and methodologists with experience of developing quality assessment tools, with the aim of developing a prediction study risk of bias tool, which will focus on prediction modelling studies.

We used a Delphi process to develop PROBAST. Thirty-nine experts in the fields of prediction research and systematic review methodology participated in the Delphi process. Early rounds of the Delphi process resulted in agreement that PROBAST should address both risk of bias and applicability. Risk of bias addresses the extent to which reported estimates of the predictive performance/accuracy (e.g. coefficients, discrimination, calibration and (re)classification estimates) of the prediction model are potentially biased and applicability refers to the extent to which the reported prediction model matches the review question. A domain-based approach, similar to that implemented in QUADAS-2, with domains being rated as high, low and unclear risk of bias, was also agreed.

The PROBAST structure includes the following four domains: participants; predictors; outcome; analysis. Two journal articles on the tool, one introducing the tool and another one with detailed explanation and elaboration have been published in the Annals of Internal Medicine and are freely available:

• PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies

• PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Further information and examples on how to use PROBAST for the assessment of studies are available on the PROBAST website:

• www.probast.org

Progress in the development of PROBAST was presented at recent Cochrane Colloquia, Guidelines International Conference and at the University of Birmingham symposium, Methods for Evaluating Medical tests and Biomarkers.

• Gary Collins 
• Jos Kleijnen
• Susan Mallett
• Carl Moons
• Hans Reitsma 
• Richard Riley
• Marie Westwood
• Penny Whiting
• Robert Wolff

• Bouwmeester W, Zuithoff NP, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KG. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1-12.
• Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011 Oct 18;155(8):529-36.
• Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013 Feb 19;158(4):280-6.
• Altman DG, McShane LM, Sauerbrei W, Taube SE. Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): explanation and elaboration. PLoS Med. 2012;9(5):e1001216.
• Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015;350:g7594
• Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-W73. doi:10.7326/M14-0698

CHARMS - Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies

TRIPOD – Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis

For more information about PROBAST, please contact Robert Wolff.