Cancer immunotherapy has transformed the treatment landscape for many malignancies by activating or restoring antitumor immune responses. Some patients achieve deep and durable remission, but many show primary resistance, experience disease progression after an initial response, or develop immune-related adverse events. Complex response patterns, including pseudoprogression and hyperprogression, further complicate treatment evaluation and adjustment.
Conventional biomarkers such as programmed death-ligand 1 expression, microsatellite instability, and tumor mutational burden are already used to support patient selection. However, their predictive accuracy and generalizability across cancer types remain limited. No single biomarker can fully capture the dynamic interactions among tumor cells, the immune system, the tumor microenvironment, and therapeutic interventions.
A new review published in Science Bulletin provides a comprehensive framework for understanding how multimodal artificial intelligence could help address these challenges. The article, titled “Emerging advances in multimodal AI in cancer immunotherapy: from multi-scale data integration to clinical decision support,” was led by Professor Bin Wang of Army Medical University in collaboration with researchers from multiple institutions.
The review systematically examines how artificial intelligence can integrate information across molecular, cellular, tissue, and clinical scales. Relevant data modalities include genomics, transcriptomics, proteomics, epigenomics, single-cell and spatial omics, digital pathology, radiological imaging, and longitudinal electronic health records. These datasets differ substantially in acquisition time, spatial resolution, technical platform, and biological meaning. Multimodal AI provides a potential means of learning non-linear relationships and higher-order interactions that are difficult to capture using conventional analytical approaches.
The authors organize the methodological evolution of AI in cancer immunotherapy into six interconnected stages: early machine-learning classifiers, unsupervised representation learning, graph-based modeling of cell–cell interactions, multimodal foundation models and biomedical language models, reinforcement and active learning paradigms, and AI virtual cells.
Early machine-learning approaches use predefined clinical, imaging, or molecular features for patient classification and risk prediction. More recent representation-learning methods, including autoencoders, variational autoencoders, and contrastive learning, can extract latent features directly from high-dimensional biomedical data and reduce reliance on manual feature engineering. Graph neural networks further represent cells, genes, proteins, and their interactions as structured networks, enabling computational analysis of communication among tumor, immune, and stromal cells.
Multimodal foundation models and biomedical language models are beginning to connect omics profiles, medical images, biological sequences, and clinical text within unified representation spaces. Reinforcement learning and active learning may further support dynamic treatment optimization and the selection of informative samples. At the frontier of this progression, AI virtual cells aim to learn computational representations of cellular states and predict how cells respond to genetic, pharmacological, or immunological perturbations. Such systems could eventually complement mechanistic research and facilitate the prioritization of candidate interventions, although extensive experimental validation remains essential.
At the molecular and cellular levels, multimodal AI can integrate genomic, transcriptomic, proteomic, epigenomic, single-cell, and spatial data to characterize the cellular composition, functional states, spatial neighborhoods, and communication networks of the tumor immune microenvironment. These approaches may help identify molecular programs associated with immune response, treatment resistance, and disease progression, while revealing rare cell populations and spatial niches that can be missed by bulk-tissue analyses. AI-assisted virtual staining, cellular phenotyping, and spatial multi-omics integration may also allow researchers to extract more information from limited tissue samples.
At the clinical interface, the review highlights applications in digital pathology and radiological imaging. Models applied to hematoxylin and eosin-stained whole-slide images and immunohistochemistry images have been investigated for predicting PD-L1 expression, microsatellite instability, tumor mutational burden, and immune-cell infiltration, as well as for assessing immunotherapy response and prognosis. Radiomics and deep-learning models based on computed tomography, positron emission tomography, and magnetic resonance imaging can extract phenotypic information from tumors and surrounding tissues for predicting treatment response, progression, recurrence, survival, and immune-related adverse events.
Combining digital pathology, radiological imaging, multi-omics profiles, clinical variables, and longitudinal health records could generate more comprehensive patient representations than any single modality alone. Such representations may improve patient stratification and help distinguish complex response patterns. In particular, longitudinal multimodal analysis may offer new opportunities to differentiate pseudoprogression from true progression or hyperprogression by jointly evaluating imaging changes, pathological features, molecular states, and clinical trajectories.
The review also emphasizes that efficacy and toxicity should not be treated as entirely separate prediction tasks. Future systems will need to jointly estimate the likelihood of therapeutic benefit and the risk of immune-related adverse events. This integrated approach could better support treatment selection, follow-up scheduling, and the timing of preventive or therapeutic interventions.
Despite substantial progress, multimodal AI is not yet ready for routine clinical use. Data modalities are often collected at different times, paired samples may be unavailable, and individual modalities may be missing. Differences among institutions, devices, platforms, and patient populations can lead to distribution shifts and reduced model performance. Models may also learn shortcut features unrelated to disease biology or fail to use all available modalities effectively.
Many published studies remain limited by retrospective single-center designs, insufficient external validation, inconsistent endpoint definitions, and inadequate assessment of calibration and clinical net benefit. Additional concerns include the opacity of foundation models, hallucination and overconfidence in generative systems, limited causal interpretation, and insufficient biological validation of model-derived predictions.
The authors therefore call for multicenter, temporal, and real-world validation, together with systematic evaluation of missing modalities, data drift, calibration, and predictive uncertainty. Decision-curve analysis and net-benefit assessment will be needed to determine whether an AI system can meaningfully improve clinical decisions rather than merely achieve high statistical performance.
Clinical translation will also require transparent data provenance, interoperability across healthcare systems, privacy protection, and lifecycle monitoring. Functional experiments, prospective studies, and clinical trials should be used to assess the biological plausibility and clinical value of AI predictions. AI virtual cells will similarly require high-quality perturbation datasets, standardized benchmarks, and closed-loop collaboration between computational prediction and experimental validation.
The review positions AI as an assistive technology for clinicians and researchers rather than a replacement for clinical judgment, mechanistic investigation, or prospective trials. By organizing the field across methodological evolution, data scales, and clinical tasks, the authors provide a roadmap for moving cancer immunotherapy beyond isolated biomarkers toward decision-support frameworks that jointly consider therapeutic efficacy, toxicity, and dynamic disease evolution.
Science Bulletin
Literature review