Lecture 4 – Multimodal Alignment (MIT How to AI Almost Anything, Spring 2025)

In this lecture, Paul Liang introduces multimodal AI, covering logistics such as project proposal feedback and reading assignments due tomorrow. He outlines the course's progression from basic AI foundations to the current focus on multimodal AI paradigms, including alignment, fusion, and transfer learning. Liang discusses the behavioral history of multimodal AI, highlighting the McGurk effect, and reviews various tasks and applications, such as audio-visual speech recognition, image captioning, and autonomous agents. He defines modalities and the challenges of heterogeneity, connections, and interactions in multimodal problems, further detailing six specific challenges: representation, alignment, reasoning, generation, transference, and quantification, with a deeper dive into alignment methods like contrastive learning and its applications, such as CLIP.