Original ArticleThe transfer of clinical prediction models for early trauma care had uncertain effects on mistriage
Introduction
Trauma, defined as physical injuries to a host by outside objects [1], accounts for approximately 458.5 million hospital visits annually across the globe [2], and around four million deaths [3,4]. Each year, 9% of global deaths are the result of trauma, with the leading causes being road traffic accidents, suicide, and homicide. Predictions indicate that the incidence of trauma due to these causes is likely to increase by 2030 [5].
In a typical high-resource setting, the initial management of trauma is performed on the scene by emergency medical services. Patient data and vital signs are transferred to the receiving hospital. This information is then evaluated using a system to determine the level of trauma, prepare adequate resources [6], and dictate whether a full or limited trauma team is activated [7].
Systems that determine the level of trauma during early trauma care can be based on clinical prediction models. Models differ in quality and characteristics but generally perform well at predicting survival [8]. Many models are developed in a single, standardized context, such as a major trauma center, and are then implemented in different contexts [9].
What is not fully understood is how this transfer affects model performance. Previous research has shown that model performance in terms of calibration can be adversely affected [9]. However, that study assessed model transfers between substantially different settings (India and the United States) and did not assess more clinically relevant performance measures, such as misclassification.
In trauma, misclassification is often referred to as mistriage. Triage refers to the classification of trauma severity as minor or major. Mistriage can be subdivided into overtriage, which is the incorrect classification of a patient with minor trauma as one with major trauma, or undertriage, which is the incorrect classification of a patient with major trauma as one with minor trauma. Mistriage can ultimately lead to decreased patient survival and is also detrimental to patient care and the distribution of resources [7].
The effects of model transfers between care contexts within a single health care system, as well as the effect of such transfers on mistriage, have not been studied and represent substantial knowledge gaps. The aim of this study was to assess how transfers of clinical prediction models for early trauma care between different care contexts within a single health system affect mistriage rates.
Section snippets
Design
A registry-based cohort study was conducted using SweTrau data to create clinical prediction models, which were then transferred between different data sets to study the effects of model transfer on mistriage. The study and analysis plans were made publicly available before the research was undertaken [10].
Source of data
Sweden has a nationally encompassing trauma registry, SweTrau. At the time of the study, the register consisted of 55,000 trauma cases, recorded from 52 of Sweden's 55 hospitals [11].
Participants
The
Results
We analyzed data from 26,965 trauma patients (Table 1) after excluding 78 patients with missing date and time of trauma. The total number of missing observations across all variables was 9,984 in the entire study cohort. The data set with the highest percentage of missing observations was the nonmetropolitan data set, with 48% incomplete observations. The variable with the highest number of missing values was RR, with 8,296 missing values, or 31% of the total values for this variable. The
Discussion
This study aimed to assess how transfers of clinical prediction models for early trauma care between different contexts within a single health system affect mistriage rates.
The most notable effect on model performance after model transfer was observed after transferring the single center model. This transfer resulted in an increased mistriage rate of 0.29. Mainly contributing to this was an increase in overtriage. By contrast, the transfer of the multicenter model to the single center
Conclusion
Depending on the care context, model transfer led to either increased or decreased mistriage. Both overtriage and undertriage were affected by model transfer, and the effects on mistriage were in all cases primarily due to changes in overtriage. Data sets with a high number of patients had the lowest mistriage both during validation, and after transfer to the validation sample in the other data set. However, the transfer of these models (while improving mistriage) also led to increased
CRediT authorship contribution statement
Martin Henriksson: Software, Formal analysis, Data curation, Writing - review & editing. Dell D. Saulnier: Conceptualization, Methodology, Formal analysis, Data curation, Writing - review & editing. Johanna Berg: Formal analysis, Data curation, Writing - review & editing. Martin Gerdin Wärnberg: Conceptualization, Methodology, Software, Formal analysis, Data curation, Writing - review & editing.
Acknowledgments
The analyses were performed on resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC. The data were provided by the Swedish trauma registry (SweTrau).
References (29)
- et al.
The clinical usefulness of prognostic prediction models in critical illness
Eur J Intern Med
(2017) - et al.
Traumatic transfers: calibration is adversely affected when prediction models are transferred between trauma care contexts in India and the United States
J Clin Epidemiol
(2016) - et al.
A simulation study of the number of events per variable in logistic regression analysis
J Clin Epidemiol
(1996) - et al.
Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure
J Clin Epidemiol
(2011) - et al.
Substantial effective sample sizes were required for external validation studies of predictive logistic regression models
J Clin Epidemiol
(2005) - et al.
External validation is necessary in prediction research: a clinical example
J Clin Epidemiol
(2003) - et al.
External validity of predictive models: a comparison of logistic regression, classification trees, and neural networks
J Clin Epidemiol
(2003) - et al.
The role of risk prediction models in prevention and management of aki
Semin Nephrol
(2019) - et al.
Trauma
(2017) - et al.
Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the global burden of disease study 2017
Lancet
(2018)
Global, regional, and national age-sex specific mortality for 264 causes of death, 1980-2016: a systematic analysis for the global burden of disease study 2016
Lancet
Injuries and violence: The facts. Report
Advanced Trauma Life Support (ATLS)
A criteria-directed protocol for in-hospital triage of trauma patients
Eur J Emerg Med
Cited by (0)
Trial registration: The protocol for this study was uploaded to github.com/613Martin/transfer-effect-mistriage before the research was undertaken.
Conflict of interest: The authors declare no conflict of interest.
Source of funding: This work was supported by the Swedish National Board of Health and Welfare, Sweden (grant numbers 22464/2017, 23745/2016 and 22289/2015-3). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Ethics committee approval: This study has been approved by the Regional Ethical Review Board in Stockholm. Ethical review numbers 2015/426-31 and 2016/461-32.