Jack Morris: Stuffing Context is not Memory, Updating Weights is

Jack Morris discusses the limitations of ChatGPT and explores methods for improving knowledge injection into language models. He contrasts full context and Retrieval Augmented Generation (RAG) with training information into model weights, advocating for the latter. Morris explains the challenges of context windows and the inefficiencies of RAG, including security concerns and adaptability issues. He then proposes training data into the model's parameters, discussing data generation strategies, catastrophic forgetting, and various architectural approaches like LoRa and memory layers. The podcast concludes with a Q&A session, addressing the trade-offs between training and RAG, synthetic data generation, and practical implications of personalized models.