Blitzy's CEO Brian Elliott and CTO Sid Pardeshi discuss autonomous software engineering, focusing on how their platform achieves AGI-type effects using non-AGI LLMs. They emphasize the importance of orchestrating LLMs within long-running complex systems, highlighting the limitations of standalone LLMs, such as context window constraints and tool selection challenges. Blitzy employs dynamic agent generation, prompts written by other agents, and iterative planning to overcome these limitations. They use a hybrid approach combining relational and semantic understanding, building a programming language-agnostic knowledge graph to deeply understand code relationships. The discussion also covers model evaluation strategies, the role of taste in assessing model performance, and the labor market implications of AI in software engineering, predicting a shift towards junior engineers skilled in AI.
Part 1: Introduction, Blitzy’s Vision Blitzy's AI-Driven Approach to Enterprise Software Modernization
Autonomous Software Engineering: Blitzy's Architecture and Capabilities
Blitzy's AGI Perspective: Orchestration Over Standalone LLM Capabilities
Part 2: LLM Limitations, Context Engineering LLM Limitations: Context Window Depreciation and Tool Management
Designing Systems for AGI-Type Effects: Harnessing LLM Limitations
Schematizing Code: Building Relational Understanding for Autonomous Development
Knowledge Graphs: A Deep Relational Understanding of Code
Part 3: Technical Implementation, Parallel Environments Ingesting Enterprise Code: Building a Programming Language Agnostic Understanding
Building and Running Applications: Understanding Relationships in Production
Parallel Application Instances: QA and Recursive Correction Loops
Implementation Challenges: Building Instructions and Modernizing Technology Stacks
Build Sophistication: Automating Windows Application Builds
Part 4: Dynamic Architecture, Evaluation Dynamic Design: Generating Agents Just-In-Time for Ever-Improving LLMs
Dynamic Systems: Referencing the Latest Prompting Instructions
Evals: Mapping Onto the Real World with Taste
Large-Scale Evaluation: Taste and Technical Design
Blitzy's Technical Taste: Instantiating Technical Taste into the Outcome
Evaluating New Models: Tracing Agentic Interactions
Part 5: System Steering, Knowledge Management Steering the System: Dynamically Addressing Instances
Algorithmic Steering: Tweaking Algorithms for the Right Outcome
Knowledge Graphs and RAG: Hybrid Sources of Truth
Semantic Matching: An Efficiency Search Mechanism
Guiding Models: Making Judgement Calls on When to Stop Searching
Part 6: Human-AI Collaboration, Workflow Structuring Requests: Creating the Right Interface Experience
Human Approval: Clarity on What Needs to Be Read
System Capabilities: Generating In-Depth Implementation Plans
Recognizing Failure: Building Independent Evaluation Systems
Project Guides: Identifying Functions That Need Human Help
Part 7: Model Selection, Memory Strategies Model Scouting Report: Leveraging Multiple Model Families
Model Selection: Dynamic Algorithms and Validation
Fine-Tuning vs. Memory: A Bet on Improved Models
Enterprise Memory: Sustaining Actions of the Best People
Long-Term Memory: A Problem to Be Solved at the System Layer
Context Management: Storing Memory and Preferences Based on Actions
Local Specificity: Memory on Enterprise Code Bases
Bifurcating Memory: Global Truths vs. Local Contextual Decisions
Part 8: Development Fundamentals, Business Value Planning and System Understanding: The Wise Developer Emotion
Parallelism: Limits and Software Development Fundamentals
Quality Preference: Paying More for Better Results
Human Labor: Working on Problems on the Edge
Pricing: Maximizing Value and Closing the Autonomy Gap
Value Creation: Pushing Hard on Value Creation
Part 9: Onboarding, Testing, Future Outlook Onboarding: Identifying Ambiguity and Extracting Information
Documentation and Test Coverage: Addressing with Blitzy
Spec Perspective: Getting to Truth
Human Involvement: From Go Time to 80% Plus
Human Report: The Human Pickup on the Back End
The Last 20%: Unanticipated Scenarios and Missing Judgement
Testing Strategy: Unit Tests and Integration Tests
The Human Task: Figuring Out Where There Is Conflict
Expressing Intent: Spec-Driven Development
Refining Pull Requests: Adjusting to Preferences
Model Improvement: Trade-Off Decisions and Raw Intelligence
The Future: Closer Than People Think
Sid Pardeshi: Prolific Inventor at NVIDIA
LLMs as Judges: Getting AIs to Correct Each Other's Work
LLM Peculiarities: Constant Conditions and Different Reactions
Ambiguous Situations: Probabilistic Models and Different Trajectories
LLM as a Judge: Dissimilar Models Evaluating Each Other's Work
Strange Behaviors: Reluctance to Use Tools
Tool Calling: GPT-4 Outshining Claude
Over Eagerness: Tool Schema Errors
Context Anxiety: Giving Up with Real Context
External Factors: Solving Issues with Model Providers
Information Theory: Reducing Entropy for Reliable Outcomes
Entropy: From Temperature to Thinking Budget
Reasoning Budget: Higher Quality Answers
Test Time Inference: The Models Are Getting Better
Model Improvements: Primarily Driven by Test Time Inference
Fully Autonomous Software Development: The Vision
Success Criteria: High Quality Code That Works
Context: The Problem That Has Been Solved
Elite AI Users: Doing a Lot of Hard Work
LLMs as Commodities: Code That Just Works
Expectations: What You'd Expect From a Human Development Team
The Goal: Code That Just Works
Test Time Training: An Obsession
Fine-Tuning: The Gap Between What You Can and Can't
Fine-Tuning: A Risky Game
Leaderboards: Insufficient
Leaderboard Design: A Complicated Problem
RKGI: Problems That Are Different
Sign in to continue reading, translating and more.
Open full episode in Podwise 
