Abstract
OBJECTIVE
The computed tomography-severity score (CT-SS) quantifies the severity of pulmonary involvement and is significantly associated with disease severity, intensive care unit (ICU) admissions, and mortality in coronavirus disease-2019 (COVID-19) patients. There is very limited information on the prognostic value of CT-SS when used in machine learning (ML) models to predict ICU admission in COVID-19 patients. In this study, the prognostic significance of CT-SS in ML model-based prediction of ICU admission among COVID-19 patients was evaluated.
MATERIAL AND METHODS
In this retrospective study, a hospital-based database from 6,854 COVID-19 patients was reviewed. To evaluate the prognostic significance of CT-SS in predicting ICU admission in patients, seven ML methods were trained separately using the most important features, with and without CT-SS data, and their performances were compared.
RESULTS
After applying exclusion criteria, 815 COVID-19 patients remained. Just over half of the patients (54.85%) were male, and the mean age was 57.22±16.76 years. The CT-SS index was the strongest predictor among the parameters examined, and integrating this index into the training dataset enhanced ML model performance. The k-nearest neighbors model with 93.3% accuracy, 97.3% sensitivity, 89.4% specificity, and an area under the curve of approximately 98.8% showed the best performance for predicting ICU admission in COVID-19 patients.
CONCLUSION
The results showed that CT-SS is a key predictor for ML models of ICU admission in COVID-19 patients. The ML models developed using a dataset including CT-SS are efficient risk stratification tools for identifying critical COVID-19 patients, thereby facilitating optimal allocation of limited hospital resources.
Main Points
• For predicting intensive care unit (ICU) admission of coronavirus disease-2019 (COVID-19) patients, computed tomography-severity score (CT-SS) is a highly relevant predictor: it differs significantly between patients admitted to the ICU and those not admitted, highlighting the necessity of including this parameter in machine learning (ML) models.
• The prognostic performance of the ML models for predicting ICU admission in COVID-19 patients was improved by integrating CT-SS data with other prognostic parameters.
• The k-nearest neighbors model, with 93.3% accuracy, 97.3% sensitivity, 89.4% specificity, 90.1% precision, 93.6% F-measure, and an area under the curve of 98.8%, had the best performance in predicting ICU admission in patients with COVID-19.
• The ML models developed using a dataset that includes CT-SS can more effectively identify vulnerable and critical patients, thereby optimizing the allocation of limited hospital resources.
INTRODUCTION
The coronavirus disease-2019 (COVID-19) pandemic presented unprecedented challenges to healthcare systems, leading to an overwhelming influx of patients and a critical shortage of resources.1, 2 Since the initial reports of the COVID-19 outbreak emerged in mid-December 2019, over 689 million individuals have been infected globally as of May 23, 2023.3 The COVID-19 virus exhibits high transmissibility and complex, heterogeneous, and evolving clinical features, resulting in a significant increase in patient morbidity and mortality.4-6
Among patients diagnosed with COVID-19, approximately 14% to 20% experience severe or critical illness,7 characterized by clinical manifestations, including acute respiratory distress syndrome, myocarditis, cardiac or septic shock, and multi-organ failure. These manifestations often necessitate hospitalization in intensive care units (ICUs) and can lead to death.4, 8 Given the high mortality among COVID-19 patients hospitalized in the ICU, reported to be as high as 50%,7 early risk stratification is essential for effective patient management and optimal allocation of medical resources. It is estimated that 20% to 30% of hospitalized patients require ICU admission,9 with this rate varying based on the specific characteristics of the study population.8
ICU resources are severely limited ICU resources, with more than 50% of them frequently occupied under normal conditions.8, 9 Therefore, there is an urgent need for effective tools to predict patient outcomes and to facilitate appropriate triage of patients. Numerous clinical and demographic parameters associated with disease severity and ICU admission have been documented.7, 8, 10-19 These prognostic variables are critical for identifying patients at high-risk who require intensive care during hospitalization. Identifying relevant predictors ultimately leads to enhanced management of high-risk COVID-19 patients and optimal use of ICU capacity. Artificial intelligence (AI) offers a useful approach to develop an effective clinical risk-prediction tool for ICU admission among COVID-19 patients.4, 9 Machine learning (ML) models, as a subset of AI, leverage vast datasets to identify patterns and predict outcomes with high precision. Using ML algorithms, clinicians can stratify patients based on their risk of deterioration, allowing for timely interventions that may improve survival rates and reduce the burden on ICUs.1, 2
Previous studies have employed ML methods that were developed using demographic data, risk factors, clinical symptoms, and laboratory results to predict ICU admissions among hospitalized COVID-19 patients. Notably, most of these studies did not include radiological indicators in their datasets.4, 9, 20-23 The computed tomography-severity score (CT-SS) is a key predictor that measures the severity of pulmonary involvement and is significantly correlated with disease severity, ICU admissions, and mortality in COVID-19 patients.5, 6, 14, 18, 24-27 Therefore, adding CT-SS to other predictors could improve the predictive power of ML models for clinical outcomes in COVID-19 patients and aid their clinical management. There is very limited information on the prognostic value of CT-SS for predicting ICU admission of COVID-19 patients using ML models. In this study, the prognostic significance of CT-SS for predicting ICU admission in COVID-19 patients was evaluated using the ML method. For this purpose, ML models were developed separately using datasets with and without CT-SS data, and their performances were compared.
MATERIAL AND METHODS
Dataset Description
In this retrospective study, a hospital-based database of COVID-19 patient information was analyzed. During initial screening of individuals referred to the COVID-19 referral center between 9 February and 20 December 2020, 6,854 suspected cases were identified. Out of these registered cases, 1,853 were confirmed as positive for COVID-19 through reverse transcriptase–polymerase chain reaction (RT-PCR) testing. The exclusion criteria for the study included: 1) a negative RT-PCR COVID-19 result, 2) absence of laboratory or CT-SS results, 3) age under 18 years, 4) discharge or death without ICU admission, and 5) unknown patient disposition. After applying these criteria, 815 records were retained for analysis. The patient selection flowchart is shown in Figure 1.
The dataset comprised 54 primary features derived from patient information that included 5 demographic variables, 14 clinical presentations, 7 medical histories, 26 laboratory results, 1 radiological measure (CT-SS), and 1 outcome variable (0 for non-ICU-admitted patients and 1 for ICU-admitted patients). The CT-SS index evaluates the severity of pulmonary involvement across five lung lobes, with each lobe assigned a visual score ranging from 0 to 5 based on the extent of involvement. The total CT-SS, ranging from 0 to 25, was derived by summing these scores.1, 2, 5, 27 Two thoracic radiologists, each with a minimum of 10 years of experience, reviewed the high-resolution chest CT images. In the event of disagreement between the two observers, discrepancies were resolved by consulting a senior radiologist. Figure 2 illustrates CT images of COVID-19 patients with different degrees of pulmonary involvement.
Data Pre-processing
Records with more than 70% missing data were excluded. Abnormal, noisy, and irrelevant data were reviewed and corrected in consultation with an infectious disease specialist. Missing values for continuous and discrete variables were imputed using mean and mode values, respectively.
The refined dataset included 185 ICU-admitted patients and 630 non-ICU patients, highlighting a significant imbalance that could bias the results toward non-ICU patients. This imbalance was addressed using the synthetic minority over-sampling technique (SMOTE) (https://imbalanced-learn.org/stable/). The SMOTE technique is an advanced algorithm designed to address data imbalance commonly encountered in data mining. SMOTE operates by generating synthetic examples of the minority class rather than merely duplicating existing instances. It identifies instances of the minority class and, for each instance, calculates the distances to its k-nearest neighbors (k-NN) within the same class. By interpolating between a minority instance and its neighbors, SMOTE creates synthetic data points that enrich the minority-class distribution. This process not only balances the class distribution but also enhances the model’s ability to learn the decision boundaries between classes, leading to improved predictive performance.1, 2
Feature Selection
The feature selection process aimed to identify parameters that were highly correlated with the target outcome, thereby significantly reducing the risk of overfitting in ML algorithms. In this study, the Boruta feature selection package was used to assess the significance of features in predicting ICU admission in COVID-19 patients. This methodology functions as a wrapper around a random forest (RF) algorithm. The Boruta algorithm evaluates the importance of features in predicting the target outcome. It determines the significance of each feature using importance values derived from shadow attributes created by randomly shuffling original attribute values across subjects. The importance of a feature is quantified using a Z-score, calculated as the mean loss in classification accuracy divided by the feature’s standard deviation. The maximum Z-score of the shadow attributes (MZSA) is identified; any attribute with an importance value exceeding this score is considered relevant. On the other hand, attributes with importance values lower than the MZSA are deemed insignificant. This iterative process continues with the elimination of shadow attributes until the significance of all features is reliably determined. The maximum number of importance source runs and the verbosity level (doTrace) were set to 500 and 2, respectively.1, 2
Model Development
To evaluate the prognostic significance of CT-SS for ICU admission among COVID-19 patients, seven ML methods were used: logistic regression (LR), k-NN, multilayer perceptron (MLP), support vector machines (SVM), eXtreme gradient boosting (XGB), RF, and C4.5 decision tree (DT). These ML models were developed using Waikato Environment for Knowledge Analysis software (version 3.9.2, University of Waikato, New Zealand). The attributes selected in the feature selection step were used as input for training and testing ML models.
To optimize the performance of each ML model, hyperparameter tuning was performed using random search. Random search explores the hyperparameter space by sampling values from predefined distributions or ranges. Random search offers great flexibility and efficiency when dealing with a large number of hyperparameters or when the optimal parameter values are less intuitive or not known. The hyperparameters tuned for each ML model are listed in Table 1.
The ML models were trained separately on datasets with and without CT-SS to evaluate the prognostic role of CT-SS in ML-based prediction of ICU admissions among patients with COVID-19. A 10-fold cross-validation method was used to examine the performance of the developed classifiers. The classification performance of each model in predicting ICU admission among COVID-19 patients was assessed using sensitivity, specificity, accuracy, and area under the receiver operating characteristic (ROC) curve (AUC).
Ethical Considerations
This article is extracted from a research project supported by Ilam University of Medical Sciences, and all experimental protocols were approved by the ethical committee of Ilam University of Medical Sciences (approved number: IR.MEDILAM.REC.1402.294, approval date: 11.03.2024). All methods used in the study were performed in accordance with the relevant guidelines and regulations of the Ethics Committee of Ilam University of Medical Sciences. This study used information from a hospital-based registry, and no intervention was performed on patients’ treatment procedures. Patient identification information was anonymized to protect patient confidentiality and privacy. All data generated and analyzed during the current study are not publicly available but will be shared by the corresponding author upon reasonable request.
RESULTS
After applying inclusion/exclusion criteria, a total of 815 COVID-19 patients were included in the study. Slightly more than half of the patients (447, 54.85%) were male, and the mean age was 57.22±16.76 years. Among the included patients, 185 (22.7%) were admitted to the ICU, increasing the number of records in this class to 630 after balancing the dataset.
Feature Selection
In the feature-selection step, the most important features for predicting ICU admission in COVID-19 patients were identified using the Boruta algorithm. Twenty-three variables were selected as the most important features for output prediction. The variable importance diagram for predicting ICU admission among COVID-19 patients is shown in Figure 3. In this diagram, the parameters shown in green, yellow, and red boxes are the confirmed, tentative, and irrelevant features, respectively. The confirmed or most important features selected in this step were used to develop ML models.
Evaluation of the Developed Models
The prediction of ICU admission in COVID-19 patients was performed using seven ML models, including LR, k-NN, MLP, SVM, XGB, RF, and C4.5 DT algorithm. These models were trained separately using the most important features, with and without CT-SS data, and their performance was compared to assess the prognostic significance of CT-SS for ML-based prediction of ICU admission among COVID-19 patients. The sensitivity, specificity, accuracy, precision, F-measure, and AUC indices for the ML models developed using datasets with and without CT-SS data are summarized in Table 2. Among all the models evaluated, the k-NN algorithm demonstrated the best predictive performance for ICU admission in COVID-19 patients. The RF, XGB, and MLP models (AUC >93%) were ranked next and demonstrated performance comparable to that of the k-NN model.
The sensitivity, specificity, accuracy, precision, F-measure, and AUC indices for the k-NN algorithm developed using the dataset without CT-SS data were 96.2%, 86.8%, 91.5%, 88.0%, 91.9%, and 98.4%, respectively. For the CT-SS dataset, the k-NN algorithm yielded 97.3% sensitivity, 89.4% specificity, 93.3% accuracy, 90.1% precision, 93.6% F-measure, and an AUC of 98.8%.
Figure 4 compares ROC curves of ML models developed using datasets with and without CT-SS data.
The integration of CT-SS data into the training dataset, consisting of demographic data, clinical symptoms, and laboratory results, enhanced the performance of the ML models.
DISCUSSION
In disease outbreaks such as the COVID-19 pandemic, healthcare systems face a growing influx of patients and significant constraints on hospital resources.9 Therefore, identifying disease-related risk factors and developing an effective predictive model to evaluate the risk of adverse outcomes can help healthcare systems optimize patient management and efficiently allocate limited medical resources. Pioneering studies have demonstrated that chest CT is a robust tool for identifying and monitoring the disease progression of COVID-19 because of its high sensitivity in depicting the severity of pulmonary involvement.6, 14, 18, 24-26 Although a significant correlation between CT-SS and COVID-19 severity and the likelihood of ICU admission has been established, its prognostic value in ML model-based prediction of ICU admission among COVID-19 patients has not yet been evaluated. In this study, the prognostic significance of CT-SS in predicting ICU admission of COVID-19 patients was evaluated using the ML method. The results showed that CT-SS data are a strong predictor of ICU admission among COVID-19 patients, and that integrating this index with other predictors improves ML models’ performance in predicting ICU hospitalization. Among the models studied, the k-NN model yielded the best performance for predicting ICU admission of COVID-19 patients.
In the first step, the importance of demographic characteristics, underlying diseases, clinical symptoms, laboratory results, and the CT-SS index for predicting ICU admission among COVID-19 patients was evaluated using the Boruta method. The results of the feature-selection step indicated that the CT-SS index was the strongest predictor among the parameters examined. COVID-19 patients admitted to the ICU had a higher mean CT-SS than those not admitted (13.9±6.6 vs. 10.4±4.6); this difference was statistically significant (P < 0.001). Therefore, CT-SS is a strong predictor of ICU admission among COVID-19 patients, and integrating this index with other predictors could improve ML models’ performance in predicting ICU hospitalization.
Twenty-three features, including white blood cell count, blood glucose level, creatinine, bilirubin, lung inflammation, heart disease, diabetes, and CT-SS, were selected as the most relevant parameters to predict ICU admission in COVID-19 patients using the Boruta method. The importance of these parameters in predicting ICU admission among COVID-19 patients has also been confirmed by other studies. Previous studies have indicated that elderly COVID-19 patients exhibiting abnormal laboratory results (such as hemoglobin, hematocrit, lymphocyte count, creatinine levels, etc.) and multiple comorbidities (such as diabetes, heart disease, chronic obstructive pulmonary disease, and chest tightness), along with radiological manifestations of extensive pulmonary involvement, are at an elevated risk of developing severe illness, leading to ICU admission.7,8,12,14-17,19,28,29
The selected predictors were used as input for training the ML models. The models were separately developed using datasets with and without CT-SS data. The performance of the ML models was then compared to determine the role of CT-SS parameters in predicting ICU hospitalization among patients. Across all ML models studied, integrating CT-SS parameters into the input dataset improved the models’ prognostic performance for identifying COVID-19 patients admitted to the ICU. The best classification performance in predicting ICU admission in COVID-19 patients was achieved by the k-NN model. The sensitivity, specificity, accuracy, precision, F1-measure, and AUC indices of the k-NN model developed using the dataset with CT-SS were 97.3%, 89.4%, 93.3%, 90.1%, 93.6%, and 98.8%, respectively.
Prior studies have also evaluated ML techniques for predicting ICU admission of COVID-19 patients. Shanbehzadeh et al.9 employed artificial neural network, DT, k-NN, SVM, and RF algorithms to predict ICU admissions among 1,225 laboratory-confirmed COVID-19 patients. Their findings indicated that the RF model outperformed other techniques. The mean accuracy, mean specificity, mean sensitivity, and root mean square error of the RF model were 99.5%, 99.7%, 99.4%, and 0.015 respectively. The k-NN algorithm, ranked next, achieved a mean accuracy of 94.1%, a mean sensitivity of 99.5%, and a mean specificity of 88.7%. The dataset used to develop the ML models in this study did not include imaging data, and the study aimed to investigate the performance of ML models in predicting the likelihood that COVID-19 patients would be admitted to the ICU. Overall, the performance of the ML models developed in this study for predicting the likelihood of COVID-19 patients being admitted to the ICU was in very close agreement with that of models developed in our research. The slight differences in results could be attributed to variations in the study population and the type of information used as input for model development. The results of this study confirm our findings.
In the study by Zakariaee et al.1, ML models were developed to predict ICU admission and length of stay (LOS) of COVID-19 patients using a dataset comprising demographics, risk factors, clinical manifestations, laboratory results, and CT-SS. For predicting ICU admission of COVID-19 patients, the performance comparison of eight ML models indicates that the k-NN model, with 97.0% sensitivity, 89.7% specificity, 93.3% accuracy, 90.4% precision, 93.6% F-measure, and 99.1% AUC, yields the best results. The RF, MLP, and XGB models (with AUC >91%) were ranked next and had performance comparable to that of the k-NN model. The advantage of this study over similar studies was that the dataset used to develop ML models included radiological manifestations. In this study, the performance of ML models for predicting the likelihood of patient admission to the ICU was examined. However, the prognostic value of the CT-SS parameter was not evaluated. The performance of our models was similar to that of the models developed in this study, which confirms our findings.
The prognostic value of CT-SS parameters for predicting mortality and clinical outcomes in patients with COVID-19 has been investigated in studies by Zakariaee et al.2, 5 In study designs similar to ours, the prognostic value of CT-SS for predicting mortality, hospital LOS, and ICU LOS among COVID-19 patients was investigated using four ML models: k-NN, MLP, SVM, and the C4.5 DT. In these studies, ML models were also developed separately using datasets with and without CT data, and their performances were compared to determine the prognostic value of the CT-SS index. The results of these studies showed that the CT-SS index is among the strongest predictors of COVID-19 progression and related complications, and that integrating this parameter into the datasets used to develop ML models improves their predictive performance.
These studies highlight the potential of ML methods to facilitate the timely and accurate identification of patients at risk for severe illness. An efficient risk stratification tool to predict ICU admissions of COVID-19 patients could significantly mitigate severe complications and associated mortality. Our findings indicate that the integration of CT-SS data with the input dataset for the ML models improves the performance of these models in predicting ICU admission among COVID-19 patients. Consequently, ML models developed using a dataset including CT-SS data can more effectively identify vulnerable and critical patients, thereby optimizing the allocation of limited hospital resources.
Study Limitations
This study has several limitations that should be acknowledged. First, the sample sizes across the data classes were significantly imbalanced. The number of patients not admitted to the ICU was much higher than that of those admitted (630 vs. 185). The SMOTE technique was used to address the dataset’s imbalance. In most data mining studies, class imbalance in sample sizes is a persistent problem. Many solutions, including oversampling of minority-class data, undersampling of majority-class data, and advanced methods such as SMOTE have been proposed to address this problem. In healthcare scenarios, many conditions are underrepresented in datasets due to their rarity, which can lead to biased models that fail to accurately predict outcomes for minority populations. Solutions such as oversampling the minority class cannot solve this problem, but the SMOTE method simulates real-world situations to generate examples. This method certainly has limitations in simulating real-world conditions, but it has advantages over other approaches because new samples are generated from existing ones rather than merely repeating them. By augmenting these datasets using synthetic examples, SMOTE enables more robust training of ML models, thereby improving their ability to generalize across diverse patient populations. This enhanced model performance can lead to more accurate risk assessments, timely interventions, and ultimately better patient outcomes. Furthermore, the use of synthetic data can facilitate the development of predictive models in clinical settings where data collection is constrained by privacy concerns or logistical challenges.
Second, prior studies developed ML models that utilized demographics, risk factors, clinical manifestations, and laboratory predictors but did not integrate radiological imaging data. In our study, the integration of CT-SS data with other predictors enriched the input dataset for developing ML models. Third, the study’s design was retrospective and limited to a single center. Further research with larger sample sizes and multicenter datasets is required to enhance the generalizability of the results. Finally, the prediction of ICU admission for COVID-19 patients was based on clinical predictors at the time of admission. Consequently, the selected features, as relevant predictors, and the developed models may be employed to predict ICU admissions at initial hospitalization of COVID-19 patients.
Conclusion
In this study, the prognostic significance of CT-SS in predicting ICU admissions for COVID-19 patients was evaluated using ML models. Our results showed that the integration of CT-SS data with other clinical predictors enhanced the prognostic performance of ML algorithms for ICU admission among COVID-19 patients. The k-NN algorithm yielded the best predictive performance among the evaluated ML techniques. The ML models developed from datasets including demographics, risk factors, clinical manifestations, laboratory results, and CT-SS data are effective tools for risk stratification to identify critically ill COVID-19 patients, thereby facilitating optimal allocation of limited hospital resources. ML predictive models can significantly enhance the timely and accurate identification of patients at risk for severe illness, ultimately reducing the incidence of severe complications and associated mortality.


