Evaluating Generalizability of Artificial Intelligence for Molecular Data

Deep learning has made rapid advances in modeling molecular sequencing data. Despite achieving high performance on molecular sequencing benchmarks, it remains unclear to what extent machine learning models learn general molecular principles and can generalize to novel sequences. Prevailing approaches to interrogate model generalizability evaluate models by generating metadata-attribute based (MB) or sequence-similarity based (SB) train-test splits of input datasets before assessing model performance.

Here, we show how these approaches mischaracterize model generalizability by failing to consider the full spectrum of cross-split overlap or the similarity between train and test sequences. We do so by introducing the spectral framework for model evaluation (SPECTRA). For a given model and input dataset, SPECTRA generates a spectral performance curve (SPC) which plots model performance as a function of decreasing cross-split overlap and reports the area under the SPC (AUSPC) as a metric for model generalizability.

We apply SPECTRA to 18 molecular sequencing datasets with diverse phenotypes ranging from antibiotic resistance in Tuberculosis to amyloid beta protein aggregation in Alzheimer's to protein-ligand binding. In doing so, we demonstrate how existing SB and MB splits provide an incomplete understanding of model generalizability. To demonstrate the insights gained from applying SPECTRA we evaluate the generalizability of ten state-of-the-art machine learning models, including large language models (LLM), a graph neural network (GNN), and convolutional neural network (CNN), using previously published and generated results.

We find that models experience a multi-fold decrease in performance as cross-split overlap decreases, and this decrease is task and model-specific. No machine learning model consistently had the highest AUSPC across all tasks though some models displayed high generalizability for specific tasks. By examining model spectral performance curves, we derived novel insights into the molecular sequence properties learned by models and demonstrate how SPECTRA can be used as a tool to evaluate foundation models in biology. SPECTRA paves the way for a more comprehensive evaluation of foundation models in molecular biology.


Evaluating Generalizability of Artificial Intelligence for Molecular Data
Yasha Ektefaie, Maha Farhat* and Marinka Zitnik*
In Review 2024 [bioRxiv]

  title={Evaluating Generalizability of Artificial Intelligence for Molecular Data},
  author={Ektefaie, Yasha and Farhat, Maha and Zitnik, Marinka},

Code Availability

Pytorch implementation of SPECTRA is available in the GitHub repository.


Latest News

Feb 2024:   Kaneb Fellowship and Dean’s Innovation Award

Feb 2024:   NSF CAREER Award

The lab receives the NSF CAREER Award for our research in geometric deep learning to facilitate algorithmic and scientific advances in therapeutics.

Jan 2024:   AI's Prospects in Nature Machine Intelligence

We discussed AI’s 2024 prospects with Nature Machine Intelligence, covering LLM progress, multimodal AI, multi-task agents, and how to bridge the digital divide across communities and world regions.

Jan 2024:   Combinatorial Therapeutic Perturbations

New paper introducing PDGrapher for combinatorial prediction of chemical and genetic perturbations using causally-inspired neural networks.

Nov 2023:   Next Generation of Therapeutics Commons

Oct 2023:   Structure-Based Drug Design

Geometric deep learning has emerged as a valuable tool for structure-based drug design, to generate and refine biomolecules by leveraging detailed three-dimensional geometric and molecular interaction information.

Oct 2023:   Graph AI in Medicine

Graph AI models in medicine integrate diverse data modalities through pre-training, facilitate interactive feedback loops, and foster human-AI collaboration, paving the way to clinically meaningful predictions.

Sep 2023:   New papers accepted at NeurIPS

Sep 2023:   Future Directions in Network Biology

Excited to share our perspectives on current and future directions in network biology.

Aug 2023:   Scientific Discovery in the Age of AI

Jul 2023:   PINNACLE - Contextual AI protein model

PINNACLE is a contextual AI model for protein understanding that dynamically adjusts its outputs based on biological contexts in which it operates. Project website.

Jun 2023:   Our Group is Joining the Kempner Institute

Excited to join Kempner’s inaugural cohort of associate faculty to advance Kempner’s mission of studying the intersection of natural and artificial intelligence.

Jun 2023:   Welcoming a New Postdoctoral Fellow

An enthusiastic welcome to Shanghua Gao who is joining our group as a postdoctoral research fellow.

Jun 2023:   On Pretraining in Nature Machine Intelligence

May 2023:   Congratulations to Ada and Michelle

Congrats to PhD student Michelle on being selected as the 2023 Albert J. Ryan Fellow and also to participate in the Heidelberg Laureate Forum. Congratulations to PhD student Ada for being selected as the Kempner Institute Graduate Fellow!

Apr 2023:   Universal Domain Adaptation at ICML 2023

New paper introducing the first model for closed-set and universal domain adaptation on time series accepted at ICML 2023. Raincoat addresses feature and label shifts and can detect private labels. Project website.

Apr 2023:   Celebrating Achievements of Our Undergrads

Undergraduate researchers Ziyuan, Nick, Yepeng, Jiali, Julia, and Marissa are moving onto their PhD research in Computer Science, Systems Biology, Neuroscience, and Biological & Medical Sciences at Harvard, MIT, Carnegie Mellon University, and UMass Lowell. We are excited for the bright future they created for themselves.

Apr 2023:   Welcoming a New Postdoctoral Fellow

An enthusiastic welcome to Tianlong Chen, our newly appointed postdoctoral fellow.

Apr 2023:   New Study in Nature Machine Intelligence

New paper in Nature Machine Intelligence introducing the blueprint for multimodal learning with graphs.

Mar 2023:   Precision Health in Nature Machine Intelligence

New paper with NASA in Nature Machine Intelligence on biomonitoring and precision health in deep space supported by artificial intelligence.

Zitnik Lab  ·  Artificial Intelligence in Medicine and Science  ·  Harvard  ·  Department of Biomedical Informatics