Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 14: Data 2

This lecture focuses on data filtering and deduplication techniques used in training language models. It begins by outlining filtering algorithms, including N-gram models, FastText classifiers, and importance resampling, emphasizing speed and generalization. The lecture then discusses how these filtering methods can be applied to language identification, quality enhancement, and toxicity reduction in datasets. The second half shifts to deduplication, distinguishing between exact and near duplicates, and introduces hashing techniques like Bloom filters and MinHash LSH to efficiently identify and remove duplicate data, ultimately improving training efficiency and preventing memorization in language models.