AI-Based Early Detection and Stage Prediction of Lung Cancer Using Machine Learning and Statistical Risk Modelling.

Primary supervisor

Jesmin Nahar

Lung cancer prediction research in computer science focuses on overcoming critical data engineering and algorithmic challenges—such as severe class imbalance in Electronic Health Records (EHR), high-dimensional tabular feature interactions, and the opacity of deep neural networks.

To address these AI-specific bottlenecks, this research proposes an end-to-end Predictive Artificial Intelligence pipeline designed for early risk scoring and multi-class stage prediction.

1. Research Objectives & Predictive AI Architecture

  • The Computational Bottleneck: Real-world clinical datasets present extreme minority-class sparsity (far fewer positive cancer instances than non-cancerous records), causing standard ML models to suffer from high false-negative bias.

  • The AI Solution: Design an automated predictive architecture that ingests tabular EMR variables (demographics, behavioral risk factors, routine blood indices) and computes continuous probability scores for early cancer risk.

  • Core Contribution: Evaluate data-centric AI techniques to improve predictive recall, precision, and multi-class stage classification accuracy without relying exclusively on compute-heavy imaging pathways.

2. Machine Learning Pipeline & Algorithmic Design

  • Synthetic Data Augmentation: Solves class imbalance by training Generative Adversarial Networks (e.g., CTGAN) and oversampling frameworks (e.g., K-Means SMOTE) to generate realistic minority-class embeddings.

  • Predictive Modeling: Benchmark ensemble gradient boosting classifiers (XGBoost, LightGBM, Random Forest) and Multi-Layer Perceptrons (MLP) to learn complex, non-linear feature interactions across tabular patient attributes.

  • Performance Metrics: Evaluates system predictive power using AUC-ROC (>0.95), Precision-Recall curves (AUC-PR), and Sensitivity to ensure high detection accuracy on unseen test cohorts.

3. Predictive Risk Modeling & Explainable AI (XAI)

  • Probabilistic Risk Scoring: Translates raw model outputs into calibrated probability scores, stratifying patients into low-, intermediate-, or high-risk tiers for clinical triage.

  • Explainable AI Integration: Implements SHAP (Shapley Additive exPlanations) and LIME to audit model predictions, exposing exact feature attributions (e.g., smoking pack-years, specific symptom combinations) that drive high risk predictions to validate algorithmic fairness and transparency.

Aim/outline

Aim

To construct an interpretable, high-precision predictive AI framework for early lung cancer risk scoring and stage prediction by optimizing ensemble classification architectures, generative data synthesis, and Explainable AI (XAI) feature attribution.

Outline

  1. Predictive Problem Formulation: Framing early detection and stage prediction as multi-class machine learning inference tasks while identifying primary computational bottlenecks (class imbalance, high feature dimensionality, and false-negative misclassifications).

  2. Generative Data Engineering & Imbalance Resolution: Applying advanced oversampling techniques (e.g., CTGAN, K-Means SMOTE) to synthesize minority-class samples, ensuring the predictive model learns unbiased decision boundaries.

  3. Predictive Model Development & Optimization: Benchmarking gradient-boosted decision trees (XGBoost, LightGBM, CatBoost) and Multi-Layer Perceptrons (MLPs), utilizing hyperparameter optimization to maximize predictive sensitivity and stage classification accuracy.

  4. Explainable AI (XAI) & Feature Attribution: Fusing SHAP (Shapley Additive exPlanations) and LIME to interpret predictive outputs, isolating key behavioral, demographic, and clinical features driving high-risk predictions.

  5. Statistical Risk Scoring & Model Validation: Translating model prediction probabilities into calibrated risk scores and validating system performance using diagnostic metrics (AUC-ROC, AUC-PR, Matthews Correlation Coefficient, Sensitivity).

Required knowledge

For IT students conducting AI research on lung cancer prediction, required knowledge encompasses four key computer science competencies:

  1. Tabular & Synthetic Preprocessing: Addressing severe medical class imbalance and incomplete EMR data using generative models (e.g., CTGAN), oversampling algorithms (e.g., K-Means SMOTE), and multivariate statistical imputation.

  2. Ensemble & Neural Architectures: Designing and tuning tree-based ensemble models (XGBoost, LightGBM, Random Forest) and multi-layer perceptrons to learn complex non-linear patterns across tabular demographic and clinical features.

  3. Explainable AI (XAI): Applying local and global interpretability frameworks like SHAP and LIME to eliminate "black-box" decision-making and extract explicit feature attributions for clinical trust.

  4. Diagnostic Metrics & Evaluation: Assessing model reliability using domain-specific AI evaluation metrics—such as AUC-ROC, Precision-Recall curves (AUC-PR), Matthews Correlation Coefficient (MCC), and Sensitivity—to minimize false-negative misclassifications.