Memory in LLMs: Weights and Activations - Jack Morris, Cornell | AI Engineer

Jack Morris discusses the limitations of ChatGPT and explores methods for improving knowledge injection into language models. He contrasts full context and Retrieval Augmented Generation (RAG) with training information into model weights, advocating for the latter. Morris explains the challenges of context windows and the inefficiencies of RAG, including security concerns and adaptability issues. He then proposes training data into the model's parameters, discussing data generation strategies, catastrophic forgetting, and various architectural approaches like LoRa and memory layers. The podcast concludes with a Q&A session, addressing the trade-offs between training and RAG, synthetic data generation, and practical implications of personalized models.

Outlines

Part 1: Context and Limitations

Part 2: RAG Systems and Embeddings

Part 3: Training Data into Weights

Part 4: Q&A and Future Outlook

Sign in to continue reading, translating and more.

Continue

Memory in LLMs: Weights and Activations - Jack Morris, Cornell

AI Engineer

Part 1: Context and Limitations

Introduction to ChatGPT's Limitations and the Context Window Problem

Context Fraud, Efficient Architectures, and the Shift to Retrieval Augmented Generation (RAG)

Part 2: RAG Systems and Embeddings

RAG Systems, Vector Databases, and Embedding Vulnerabilities

Limitations of Embeddings and the Potential of Agentic Search

The Trade-offs of RAG and Introduction to Training Things into Weights

Part 3: Training Data into Weights

Training Data into Weights: Challenges and Solutions

Synthetic Data Generation and Architectural Considerations for Training

Properties of Effective Training Methods and Parameter-Efficient Techniques

Part 4: Q&A and Future Outlook

Q&A: RAG vs. Training, Synthetic Data, and Model Security

Q&A: Version Control, Conflicting Information, and Federated Tuning

Memory in LLMs: Weights and Activations - Jack Morris, Cornell

AI Engineer

Part 1: Context and Limitations

00:20Introduction to ChatGPT's Limitations and the Context Window Problem

Introduction to ChatGPT's Limitations and the Context Window Problem

07:07Context Fraud, Efficient Architectures, and the Shift to Retrieval Augmented Generation (RAG)

Context Fraud, Efficient Architectures, and the Shift to Retrieval Augmented Generation (RAG)

Part 2: RAG Systems and Embeddings

10:44RAG Systems, Vector Databases, and Embedding Vulnerabilities

RAG Systems, Vector Databases, and Embedding Vulnerabilities

14:40Limitations of Embeddings and the Potential of Agentic Search

Limitations of Embeddings and the Potential of Agentic Search

21:31The Trade-offs of RAG and Introduction to Training Things into Weights

The Trade-offs of RAG and Introduction to Training Things into Weights

Part 3: Training Data into Weights

24:36Training Data into Weights: Challenges and Solutions

Training Data into Weights: Challenges and Solutions

30:33Synthetic Data Generation and Architectural Considerations for Training

Synthetic Data Generation and Architectural Considerations for Training

37:02Properties of Effective Training Methods and Parameter-Efficient Techniques

Properties of Effective Training Methods and Parameter-Efficient Techniques

Part 4: Q&A and Future Outlook

44:48Q&A: RAG vs. Training, Synthetic Data, and Model Security

Q&A: RAG vs. Training, Synthetic Data, and Model Security

55:16Q&A: Version Control, Conflicting Information, and Federated Tuning

Q&A: Version Control, Conflicting Information, and Federated Tuning