Stacking classifiers based on integrated machine learning model: fusion of CT radiomics and clinical biomarkers to predict lymph node metastasis in locally advanced gastric cancer patients after neoadjuvant chemotherapy

Abstract

Background

The early prediction of lymph node positivity (LN+) after neoadjuvant chemotherapy (NAC) is crucial for optimizing individualized treatment strategies. This study aimed to integrate radiomic features and clinical biomarkers through machine learning (ML) approaches to enhance prediction accuracy by focusing on patients with locally advanced gastric cancer (LAGC).

Methods

We retrospectively enrolled 277 patients with LAGC and randomly divided them into training (n = 193) and validation (n = 84) sets at a 7:3 ratio. In total, 1,130 radiomics features were extracted from pre-treatment portal venous phase computed tomography scans. These features were linearly combined to develop a radiomics score (rad score) through feature engineering. Then, using the rad score and clinical biomarkers as input features, we applied simple statistical strategies (relying on a single ML model) and integrated statistical strategies (including classification model integration techniques, such as hard voting, soft voting, and stacking) to predict LN+ post-NAC. The diagnostic performance of the model was assessed using receiver operating characteristic curves with corresponding areas under the curve (AUC).

Results

Of all ML models, the stacking classifier, an integrated statistical strategy, exhibited the best performance, achieving an AUC of 0.859 for predicting LN+ in patients with LAGC. This predictive model was transformed into a publicly available online risk calculator.

Conclusions

We developed a stacking classifier that integrates radiomics and clinical biomarkers to predict LN+ in patients with LAGC undergoing surgical resection, providing personalized treatment insights.

Background

Asia has the highest gastric cancer incidence globally, although the overall incidence has been steadily declining [1]. Locally advanced gastric cancer (LAGC) continues to pose considerable challenges to gastric cancer treatment and is the third leading cause of cancer-related death worldwide [1]. Currently, the standard surgical treatment for LAGC is a D2 radical resection with lymphadenectomy [2]. The prognosis for LAGC is unfortunately unfavorable, with less than 40% of patients surviving five years post-diagnosis. This poor outcome can be attributed primarily to a combination of tumor-related, host-related, and environmental factors [3].

Several clinical trials have established neoadjuvant chemotherapy (NAC) as the standard treatment for LAGC. This approach effectively downstages the primary tumor, eradicates micrometastases, reduces intraoperative dissemination, and enhances R0 resection rates, positively affecting patient survival [4,5,6]. A recent study indicated that lymph node (LN) downstaging is an independent prognostic factor for LAGC and exerts a more significant positive effect on LAGC than the treatment response following NAC [7, 8]. However, only approximately 30% of NAC recipients exhibit LN regression. LN metastasis can also occur, which can profoundly affect therapeutic strategies and outcomes of patients with LAGC due to the broad heterogeneity of the cancer [9]. Therefore, accurately assessing tumor heterogeneity and identifying LN involvement after neoadjuvant chemotherapy are crucial for determining the optimal extent of surgery and refining pre-treatment decision-making for patients with LAGC.

Recently, computed tomography (CT) has become a routine tool in clinical practice for diagnosing LN positivity (LN+) in patients with LAGC who are undergoing NAC [10]. However, LN+ assessments are subjective, challenging their diagnostic accuracy; they also often yield unsatisfactory results, with a sensitivity rate of approximately 57% [11]. Furthermore, the accurate assessment of lymph node status remains challenging due to inherent limitations in imaging modalities to detect microscopic metastases, a fundamental challenge in oncology that persists regardless of treatment timing [12]. Various clinicopathological factors, including tumor size, tumor infiltration depth, histological type, and the neutrophil-lymphocyte ratio, are associated with LN+ [13, 14]. However, these indicators remain controversial as they have low sensitivity and weak representativeness; thus, there is no consensus on their applicability.

Radiomics involves extracting high-dimensional quantitative features from medical imaging, enabling noninvasive tumor heterogeneity analysis by integrating complex imaging characteristics. Its usefulness in gastric cancer has been increasingly recognized, especially for predicting therapeutic responses and survival outcomes [15]. Notably, a few studies have highlighted the potential of CT radiomics in aiding clinicians in predicting LN+ in patients with LAGC after NAC [16, 17]. However, in practical scenarios, selecting the most appropriate statistical method for the data may be challenging. A single statistical model might overlook certain patterns, limiting diagnostic efficiency. To address this issue, integrated statistical strategies such as hard voting, soft voting, and stacking can be employed. These strategies combine multiple trained models to provide the final prediction results through different approaches, enhancing the precision and accuracy of clinical decisions. This approach also facilitates the construction of robust prediction models for complex situations, reducing computational complexity while maximizing clinical data utilization [18, 19].

Therefore, this study used radiomic features, clinical biomarkers, and various machine learning methodologies to develop a prediction model for LN+ post-NAC in patients with LAGC, aiming to improve LN+ predictions for these patients.

Methods

Patient enrollment

This retrospective study received approval from the Ethics Committee of the Affiliated Cancer Hospital of Guangxi Medical University (Approval No.: KY2025582). We conducted a comprehensive review of datasets from patients clinically staged as cT2-4NxM0, who underwent NAC at our institution between January 2014 and December 2024. The study predominantly comprised patients with T3-4 disease. Although the term 'locally advanced' is traditionally associated with T3-4 disease, we also included selected T2 cases where there was clinical suspicion of advanced features, aligning with recent guideline-supported practice patterns [20]. Details regarding the preoperative NAC regimen, based on tumor regression grade, are provided in Additional File 1.

The inclusion criteria were (1) LAGC confirmed by histopathological examination; (2) availability of a contrast-enhanced CT examination within 7 days before NAC; and (3) completion of radical gastrectomy with D2-regional lumpectomy after NAC, with complete postoperative pathological data. The exclusion criteria were (1) CT images of inferior quality due to noise, respiratory artifacts, or other movement artifacts; (2) having other malignant tumors; (3) having received any anticancer treatment before baseline CT scanning; and (4) having incomplete or ambiguous clinicopathologic data.

Patients were randomly allocated to the training or validation set at a 7:3 ratio (Fig. 1). The training set was used for model development and identifying informative variables, whereas the validation set was used to assess the models’ performances.

Fig. 1

Enrollment criteria for patient selection

CT image acquisition and lesion segmentation

Additional file 2 details the CT scan parameters. The segmentation process was initiated by a radiologist utilizing ITK-SNAP software (version 4.0.0, accessible at www.itksnap.org) to manually delineate each layer of the contrast-enhanced venous CT images, thereby generating a manually contoured region of interest. Subsequently, a senior radiologist carefully examined and refined the segmentation to guarantee accuracy and compliance with the standards. Subsequently, the OnekeyAI platform software was employed for image preprocessing; voxel dimensions were resampled to 1 × 1 × 1 mm³ (X, Y, Z) to ensure dataset consistency. After preprocessing, 1,130 radiomic features were extracted from the region of interest, encompassing first-order intensity, shape, and diverse gray-level matrix features such as the size zone, co-occurrence, dependence, and run length matrices.

Radiomics feature engineering

Radiomic features were normalized to have a mean of 0 and a standard deviation of 1. The feature engineering process comprised 1) feature reduction and 2) screening. In the first step, a batch t-test was performed to exclude features with p-values >0.05. To address feature redundancies, we identified correlation coefficients (r) greater than 0.9 and randomly excluded one of the correlated features. In the second step, a least absolute shrinkage and selection operator (LASSO) analysis with 10-fold cross-validation was employed to identify the most significant radiomic signatures (Additional file 3, Supplementary Figure 1). Features with nonzero coefficients from the LASSO analysis were selected and used to compute each patient’s radiomics score (rad score). The rad score was derived by linearly combining the selected features with each feature weighted by its corresponding coefficient using Equation 1:

$$\text{rad score }= \sum_{i = 1}^{n}{X}_{i}{\beta }_{i}$$

(1)

X_i is the radiomics feature selected by the LASSO regression, and β_i is the corresponding regression coefficient.

Machine learning development

The rad score and clinical biomarkers were incorporated as input features to construct machine learning models. Simple and integrated statistical strategies were used to predict LN+ after NAC.

Simple statistical strategies

Simple statistical strategies rely on individual machine learning models. Random forest (RF), gradient boosting decision tree (GBDT), extreme gradient boosting (XGBoost), light gradient boosting machine (LightGBM), adaptive boosting (AdaBoost), and categorical boosting (CatBoost) are examples of ensemble learning models. These models enhance prediction accuracy and stability by aggregating outputs from multiple decision trees, each trained on different data subsets to capture various patterns. This aggregation reduces overfitting and improves generalization, enabling robust performance across diverse tasks and datasets [21].

Integrated statistical strategies

Integrated statistical strategies involve techniques such as stacking and voting classifiers. Stacking is an ensemble method that minimizes bias by combining multiple prediction outcomes. It operates in two layers: the first layer comprises several base classifiers, and the second layer uses a meta-learner to integrate its outputs. We utilized six base classifiers: RF, GBDT, XGBoost, LightGBM, AdaBoost, and CatBoost (Fig. 2). Voting classifiers employ hard or soft voting methods. In hard voting, the final prediction is the class that garners the most votes from the classifiers. In soft voting, the predicted class has the highest average probability assigned by the classifiers [22].

Fig. 2

Study design flowchart

Optimal model identification

The optimal machine learning model was determined by analyzing receiver operating characteristic curves and the areas under the curves (AUCs). The trained models were assessed using a validation dataset, and their classification performance was quantified using AUC metrics. The model with the highest AUC value in the validation cohort was selected as the final model [23]. To further evaluate the selected model, additional metrics derived from the confusion matrix were calculated, including accuracy, precision, recall, F1 score, negative predictive value (NPV), and positive predictive value (PPV).

Statistics analyses

Statistical analyses were performed using the R software (version 4.4.3, R Core Team, Vienna, Austria). Categorical variables were evaluated using the chi-squared test. Normally distributed continuous variables are presented as means and standard deviations and were compared using the independent samples t-test. Non-normally distributed data are presented as medians with interquartile ranges and were analyzed using the Mann–Whitney U test. Statistical significance was set at p <0.05.

Results

Clinicopathological characteristics

This study included 277 patients with LAGC who underwent NAC followed by surgical resection. Table 1 details the patients’ clinicopathological characteristics. Overall, 136 patients (49.1%) were postoperatively diagnosed with LN+. The training and validation sets included 193 and 84 patients, among whom 93 (48.2%) and 43 (51.2%) were diagnosed with LN+, respectively. The clinicopathological characteristics did not differ between the training and validation sets (Table 1). Furthermore, post-NAC LN+ significantly correlated with advanced clinical T stage, advanced clinical N stage, a reduced number of neoadjuvant chemotherapy lines (nNAC), a lower immunotherapy proportion, and an elevated rad score (p<0.05) (Additional file 4, Supplementary Table 1).

Table 1 Comparison of Baseline Characteristics Between the Training and Validation Sets

Radiomics analysis

Initially, 1,130 radiomic features were extracted from the pre-treatment portal venous-phase CT scans. A batch t-test reduced this number to 446, and further refinement using a batch correlation analysis narrowed the selection to 154 features. After employing a LASSO regression with 10-fold cross-validation, 11 features with nonzero coefficients were identified (Additional file 3, Supplementary Figure 2). The correlation heat map for the final selection of radiomic features indicated low relevance and redundancy (Additional file 3, Supplementary Figure 3). Ultimately, the radiomic model achieved an AUC of 0.788.

Machine performance comparison

Our analysis utilized the radiomic score and clinical biomarkers as input features (x₁, x₂, x₃, …, xₙ) and employed both simple and integrated statistical methodologies to predict LN+ following NAC. The histopathological examination of surgically resected lymph nodes post-NAC serves as the credible gold standard in this binary classification task, providing the definitive criterion for determining lymph node metastasis (ypN+ vs. ypN0). Consequently, the label space is defined such that ypN0 corresponds to y₀ and ypN1 corresponds to y₁, in accordance with previous studies [16, 17]. The LightGBM model demonstrated superior performance among the simple statistical approaches, with an AUC of 0.850 (Fig. 3). In comparison, neither the hard voting classifier (AUC: 0.790) nor the soft voting classifier (AUC: 0.850) improved performance among the integrated statistical strategies. The stacking classifier was the most efficient at predicting LN + following NAC in patients with LAGC, with an AUC of 0.859 (Table 2). Figure 4 presents a detailed overview of the performance metrics, encompassing the AUC, accuracy, recall, precision, F1 score, NPV, and PPV. These results further substantiate the effectiveness of the stacking classifier for this classification task.

Fig. 3

Receiver operating characteristic (ROC) curves depicting the performance of various diagnostic models for predicting lymph node positivity (LN+) in patients with locally advanced gastric cancer (LAGC)

Table 2 Diagnostic Performance Comparison of Simple versus Integrated Machine Learning Models

Fig. 4

Polar map of the comprehensive performance metrics for the diagnostic models. AUC: Area under the curve

Model explanation

We used Shapley additive explanations (SHAP) to further interpret the model [24]. CatBoost had the most significant impact when assessing the first layer’s contribution to the stacking model, followed by XGBoost, LightGBM, GBDT, RF, and AdaBoost (Fig. 5). Additional file 3, Supplementary Figure 4 presents the SHAP values for the first layer, indicating feature importance.

Fig. 5

Shapley additive explanations plot assessing the contribution of the first layer to the stacking model, highlighting their relative contributions

The global interpretation of the stacking classifier for the second layer, which constitutes the final model with optimal input features, was ascertained during the feature reduction process of the stacking classifier. Figure 6 depicts the dynamic response of the AUC throughout this process; the model that incorporated the five top-contributing features had the highest AUC for predicting LN+ following NAC in patients with LAGC. This model demonstrated significant net benefits and robustness. The rad score was the most influential feature among the top five contributing features, followed by the clinical N stage, nNAC, clinical T stage, and CA199 level (Fig. 7).

Fig. 6

The dynamic response of the AUC throughout this process; the model that incorporated the five top-contributing features had the highest AUC for predicting LN+ following NAC in patients with LAGC

Fig. 7

Shapley additive explanations plot ranking the explanatory variables by significance in the stacking model, indicating the key drivers of the model’s predictions

Application for clinical use

The final prediction model that integrated the top five contributing features, the stacking classifier, was developed into a web application designed as a user-friendly online calculator for clinical applications. This online calculator autonomously predicts the risk of LN+ in patients with LAGC following NAC and is available at https://s8xupsxjt5edbdpeuaqyem.streamlit.app/.

Discussion

This study used radiomic features and clinical biomarkers to generate machine learning models for predicting LN+ following NAC in patients with LAGC. We used simple and integrated statistical strategies, finding that the hard voting and soft voting classifiers did not perform better than the simple strategy. In contrast, the stacking classifier, which aggregates information from multiple machine learning models, achieved the highest efficacy and demonstrated superior model performance compared with the simple strategy.

NAC is a standard treatment for LAGC [4,5,6]. Consequently, LN downstaging emerged as an independent prognostic factor, exerting a more pronounced positive impact than the treatment response in LAGC post-NAC [7, 8]. Consequently, the precise assessment of tumor heterogeneity and identification of LN positivity following NAC are essential to determine the optimal extent of surgical intervention and refine pre-treatment decision-making for LAGC. Radiomic analyses comprehensively visualize tumor tissues, capturing morphological and textural characteristics. The distinct textural and spatial gray-level patterns of the extracted radiomic features can reflect variations in cellular composition or properties, thereby aiding tumor heterogeneity assessments [15]. Although some studies have underscored the potential of CT radiomics to assist clinicians in predicting LN positivity in patients with LAGC after NAC [16, 17], these studies were limited using a single, simple statistical approach.

The absence of an integrated strategy has hindered improvements in diagnostic efficiency. These integration strategies have several distinct advantages and disadvantages [21, 22]. First, the selection of these strategies is determined by various factors, such as the data characteristics, model complexity, application scenarios, and prediction objectives. This study implemented three integration strategies to classify LN+ or the LN status after NAC in patients with LAGC. Although majority voting (hard voting) and weighted voting (soft voting) are relatively straightforward methods, they fail to fully leverage performance variations among different models [22]. As a result, neither the hard voting nor soft voting classifiers demonstrated superior performance compared with simpler strategies in this study. Conversely, the stacking classifier enhances the predictive capability by integrating diverse models in its initial layer and employing a meta-learner to refine predictions. This approach addresses individual model limitations, captures intricate data patterns, and improves classification accuracy [25].

SHAP is a technique for elucidating machine learning predictions and was employed on a stacking classifier. This method utilizes the Shapley values derived from game theory to allocate importance to each feature, illustrating their contribution to the model’s output. By decomposing the predictions into the marginal contributions of each feature, SHAP provides equitable credit attribution. Visualizing SHAP values helps users identify the features that most significantly influence the model predictions [24]. Feature reduction of the stacking classifier for this study resulted in a final model with five input features. The rad score emerged as the most influential of these contributing features, exhibiting the highest SHAP value, followed by the clinical N stage, nNAC, clinical T stage, and CA199 level. In patients with LAGC, the LN+ status following NAC was significantly associated with an increased rad score, advanced clinical N stage, decreased nNAC, advanced clinical T stage, and elevated CA199 levels. Notably, the rad score emerged as the most significant factor, underscoring the capacity of radiomics analyses to offer additional insights into tumor heterogeneity that extend beyond the visual assessments conducted by radiologists. This observation is consistent with the findings of previous studies [16, 17]. Furthermore, advanced clinical N and T stages and elevated CA199 levels suggest a higher malignancy grade, while reduced nNAC may indicate incomplete chemotherapy cycles, potentially limiting the downstaging of LN, aligning with previous studies [13, 14]. Finally, we developed a web application using a stacking classifier that integrated five features, designed as a user-friendly online calculator for clinical applications. This online calculator autonomously predicts the risk of LN+ in patients with LAGC following NAC.

Our study had some limitations. First, although the stacking classifier, which integrates information from multiple machine learning models, was the most effective and outperformed the straightforward strategy for our patient population, training each layer requires separate datasets. This could potentially demand greater computational resources, leading to a large computational load and extended computation time for stacking models. Second, radiomic features were extracted only from CT images without considering other multimodal medical imaging techniques, such as magnetic resonance imaging and ultrasound. Third, as a retrospective study, it may have been subject to selection bias and a limited sample size. Fourth, although two experienced radiologists conducted manual tumor segmentation collaboratively to ensure clinical relevance, we did not formally assess inter-observer variability. Future research should include inter-rater reliability analyses and explore semi-automated or deep learning-based segmentation methods to enhance efficiency and reproducibility. Lastly, the study did not include patient follow-up; thus, the long-term effects of the LN status and NAC response on LAGC were evaluated.

Conclusions

We developed a stacking classifier to integrate radiomic features and clinical biomarkers for predicting the LN+ status in patients with LAGC undergoing surgical resection, offering personalized treatment insights. Further prospective analyses with larger cohorts are warranted for validation.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.