An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand.
External validation studies evaluate the performance of one or more prediction models (eg, developed previously using statistical, machine learning, or artificial intelligence approaches) in a different dataset to that used in the model development process.1 2 3 Part 2 in our series describes how to undertake a high quality external validation study,4 including the need to estimate model performance measures such as calibration (agreement between observed and predicted values), discrimination (separation between predicted values in those with and without an outcome event), o
External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model’s predictive performance and clinical usefulness.
A clinical prediction model is used to calculate predictions for an individual conditional on their characteristics. Such predictions might be of a continuous value (eg, blood pressure, fat mass) or the probability of a particular event occurring (eg, disease recurrence), and are often in the context of a particular time point (eg, probability of disease recurrence within the next 12 months). Clinical prediction models are traditionally based on a regression equation but are increasingly derived using artificial intelligence or machine learning methods (eg, random forests, neural networks). Regardless of the
Objective To examine the effect of a (fictional) doctor working during the festive period on population health.
Design Natural experiment.
Setting England, Wales, and the UK.
Main outcome measures Age standardised annual mortality rates in England, Wales, and the UK from 1963, when the BBC first broadcast Doctor Who , a fictional programme with a character called the Doctor who fights villains and intervenes to save others while travelling through space and time. Mortality rates were modelled in a time series analysis accounting for non-linear trends over time, and associations were estimated in relation to a new Doctor Who episode broadcast during the previous festive period, 24 December to 1 January. An interrupted time series analysis modelled the shift in mortality rates from 2005, when festive episodes of Doctor Who could be classed as a yearly Christmas intervention.
Results 31 festive periods from 1963 have featured a new Doctor Who episode, including 14 broadcast on Christm
Some remaining questions on the QCOVID models bmj.com - get the latest breaking news, showbiz & celebrity photos, sport news & rumours, viral videos and top stories from bmj.com Daily Mail and Mail on Sunday newspapers.