Multimodal Deep Learning for Pathology Applications

Subjective clinical diagnosis is often based on multimodal information from microscopic and molecular information as well as information from patient and familial histories. Most recent work in objective pathology image analysis using deep learning does not take into account additional information that often acts as an important source of diagnostic and prognostic cues. In this talk we will present a variety of different computational paradigms to fuse information from microscopic images of tissue biopsies, corresponding genomic data as well as patient and familial histories. We demonstrate that fusing multimodal information significantly improves survival prediction, characterization, prognostication and patient stratification for targeted theripies. We further demonstrate that such a multimodal fusion paradigm can be used to identify new biomarkers and morphological features that can lead to the development of new grading schemes.

Speaker's bio: Dr. Faisal Mahmood received his PhD in Biomedical Imaging from Okinawa Institute of Science & Technology and did a postdoctoral fellowship at the Johns Hopkins Whiting School of Engineering. He is currently an assistant professor of computational pathology at Harvard Medical School and the Brigham and Women’s Hospital. He is also an associate member of the Broad Institute of Harvard and MIT and a member of the Cancer Data Science Program at the Dana-Farber Cancer Institute. His laboratory focuses on pathology image analysis, morphological feature and biomarker discovery using deep learning, data fusion, and multimodal analysis.