At the start of the COVID-19 pandemic, little was known about the disease. A monumental effort was made to understand the evolving data and develop prediction tools that patients, health-care workers, and policy makers could use to optimise care. The unfortunate result was a tidal wave of poorly conceptualised prediction models, often using small convenience samples, incorporating little or no validation, and including no substantive plan for implementation.
1
- Wynants L
- Van Calster B
- Collins GS
- et al.
Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal.
As a result, most of the prediction tools developed were never meaningfully applied in clinical care.
Examples of good practice exist, including two collaborative projects QCOVID (estimating risk of being hospitalised or dying due to catching COVID-19)
2
- Clift AK
- Coupland CAC
- Keogh RH
- et al.
Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study.
and the ISARIC 4C models (estimating risk of dying or deteriorating after hospital admission with COVID-19).
3
- Knight SR
- Ho A
- Pius R
- et al.
Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score.
,
4
- Gupta RK
- Harrison EM
- Ho A
- et al.
Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study.
However, an obvious gap existed in the assessment of symptomatic patients in the community. As the profile of COVID-19 has changed and the focus of care shifts to supporting diagnosis, treatment, and monitoring outside hospitals, the assessment of patients has become increasingly important.
In the Lancet Digital Health, we welcome the study by Ana Espinosa-Gonzalez and colleagues
5
- Espinosa-Gonzalez AB
- Prociuk D
- Fiorentino F
- et al.
Remote COVID-19 assessment in primary care (RECAP) risk prediction tool: derivation and real-world validation studies.
on the derivation and validation of two much-needed risk stratification tools for use in a community setting. The two pragmatic decision aids support the assessment of patients with symptoms of COVID-19, seeking to identify those who will probably require further monitoring (Remote COVID-19 assessment in primary care–General Practice, without peripheral oxygen saturation [RECAP-GP]) and those in whom treatment escalation is warranted (RECAP-oxygen [RECAP-O2]). The models were developed according to a prepublished protocol and used linked primary and hospital health-care records, together with data from the WhatsApp-based patient monitoring platform, Doctaly Assist.
6
- Espinosa-Gonzalez AB
- Neves AL
- Fiorentino F
- et al.
Predicting risk of hospital admission in patients with suspected COVID-19 in a community setting: protocol for development and validation of a multivariate risk prediction tool.
What do these data tell us and how well do the models work? It is important to reflect on what the models actually capture. The patients included in the cohorts had symptoms of COVID-19, but they did not necessarily have COVID-19. This is pragmatic and appropriate because a COVID-19 diagnostic test might not be available at the time of assessment. But as COVID-19 prevalence decreases in the community, how patients are selected to use this tool will significantly affect its performance.
An additional point of reflection is around a concept termed incorporation bias. Espinosa-Gonzalez and colleagues
5
- Espinosa-Gonzalez AB
- Prociuk D
- Fiorentino F
- et al.
Remote COVID-19 assessment in primary care (RECAP) risk prediction tool: derivation and real-world validation studies.
are testing to see if symptoms predict admission, but the same symptoms have probably been used to determine the need for the actual hospital admission. Therefore, the prediction tool can become a self-fulfilling prophecy, and this circularity can artificially increase sensitivity and specificity. The authors mitigate against this by requiring an admission to be at least one night (and by implication require clinical management rather than simply assessment), but the effects of this bias might persist.
The RECAP-GP model performs well in the first external cohort of patients from northwest London, but the discrimination is poorer in the second (COVID Clinical Assessment Service; area under the receiver operator characteristic curve [AUROC] 0·66). A similar result was seen for RECAP-O2 (Doctaly-2; AUROC 0·68). Compared with the derivation and first external validation cohorts, the Doctaly-1 cohort was recruited later in the pandemic and differences in population (younger age with fewer comorbidities), virus variants, and vaccination status might partly explain this.
7
- Sperrin M
- McMillan B
Prediction models for covid-19 outcomes.
Calibration (the performance of the model across the range of risk) is important;
8
- Van Calster B
- McLernon DJ
- van Smeden M
- Wynants L
- Steyerberg EW
Calibration: the Achilles heel of predictive analytics.
although good to see calibration data reported for the development dataset, it would have been useful for the external validation too. Similarly, while good to see model performance presented by age and sex, it is important to ensure that it performs as well across different ethnic groups.
As presented the models might confuse users. The risk of hospital readmission for patients who were breathless after moderate exertion is lower than for those with breathlessness after mild exertion, which is not what we would expect to see (RECAP-GP; similar finding in RECAP-O2). For instance, a 45-year-old man with hypertension and a fever complaining of moderate breathlessness after exertion will be graded as being at amber risk (8·1% risk of hospital admission) while the same patient describing mild breathlessness after exertion will be graded as being at red risk (11·5% risk of hospital admission). This could be explained by the incorporation of non-significant factor levels, but the resulting biological implausibility might reduce face validity.
Applicability in low-income and middle-income countries must also be considered. Continued reduced access to vaccination, varied public health policy implementation and higher death rates
9
- Salyer SJ
- Maeda J
- Sembuche S
- et al.
The first and second waves of the COVID-19 pandemic in Africa: a cross-sectional study.
,
10
Johns Hopkins Coronavirus Resource Center
Mortality Analyses.
suggest research should be relevant and generalisable to such settings. The widespread absence of peripheral oxygen monitors means the RECAP-O2 model is currently unlikely have relevance beyond a select few countries. However, RECAP-GP has the potential for global clinical use and validation in low-income and middle-income countries is an urgent priority.
We declare no competing interests.
References
- 1.
- Wynants L
- Van Calster B
- Collins GS
- et al.
Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal.
BMJ. 2020; 369m1328
- 2.
- Clift AK
- Coupland CAC
- Keogh RH
- et al.
Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study.
BMJ. 2020; 371m3731
- 3.
- Knight SR
- Ho A
- Pius R
- et al.
Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score.
BMJ. 2020; 370m3339
- 4.
- Gupta RK
- Harrison EM
- Ho A
- et al.
Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study.
Lancet Respir Med. 2021; 9: 349-359
- 5.
- Espinosa-Gonzalez AB
- Prociuk D
- Fiorentino F
- et al.
Remote COVID-19 assessment in primary care (RECAP) risk prediction tool: derivation and real-world validation studies.
Lancet Digit Health. 2022; ()
- 6.
- Espinosa-Gonzalez AB
- Neves AL
- Fiorentino F
- et al.
Predicting risk of hospital admission in patients with suspected COVID-19 in a community setting: protocol for development and validation of a multivariate risk prediction tool.
JMIR Res Protoc. 2021; 10e29072
- 7.
- Sperrin M
- McMillan B
Prediction models for covid-19 outcomes.
BMJ. 2020; 371m3777
- 8.
- Van Calster B
- McLernon DJ
- van Smeden M
- Wynants L
- Steyerberg EW
Calibration: the Achilles heel of predictive analytics.
BMC Med. 2019; 17: 230
- 9.
- Salyer SJ
- Maeda J
- Sembuche S
- et al.
The first and second waves of the COVID-19 pandemic in Africa: a cross-sectional study.
Lancet. 2021; 397: 1265-1275
- 10.
- Johns Hopkins Coronavirus Resource Center
Mortality Analyses.
Article Info
Publication History
Identification
DOI: https://doi.org/10.1016/S2589-7500(22)00146-7
Copyright
© 2022 The Author(s). Published by Elsevier Ltd.
User License
Creative Commons Attribution (CC BY 4.0) |
ScienceDirect
Access this article on ScienceDirect
Linked Articles
- Remote COVID-19 Assessment in Primary Care (RECAP) risk prediction tool: derivation and real-world validation studies
-
Both RECAP models are valid tools to assess COVID-19 patients in the community. RECAP-GP can be used initially, without need for observations, to identify patients who require monitoring. If the patient is monitored and SpO2 is available, RECAP-O2 is useful to assess the need for treatment escalation.
- Full-Text
Open Access
-
