Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize

The podcast centers on prompt learning, a technique for optimizing prompts using human feedback and LLM evaluations. It addresses why agents fail, emphasizing weak environments and instructions over model weaknesses. Prompt learning is presented as an alternative to reinforcement learning and meta-prompting, utilizing English feedback to pinpoint issues. A case study demonstrates a 15% performance improvement in coding agents by adding rules to system prompts, achieving performance near state-of-the-art models at a lower cost. The discussion covers overfitting, reframing it as expertise gained through continuous optimization. Benchmarking against GEPA shows prompt learning's effectiveness with high-quality evaluations. The presenters also address questions about setting up evaluations for non-quantifiable prompts and dynamically changing instructions over time.