The good, the bad, and the AI apps

Defining "good" for AI applications requires moving beyond sparse, qualitative feedback toward structured, repeatable evaluation processes. Benny Chen, co-founder of Fireworks AI, highlights that effective AI development involves decomposing complex problems into measurable symptoms that language models can judge. This iterative "whack-a-mole" process allows teams to identify specific failures and refine model behavior through targeted evaluations. While deterministic unit tests remain essential for safety and basic functionality, product-level improvements often rely on aggregating scores that align with long-term user satisfaction. As infrastructure scales to handle massive inference workloads, the industry is shifting toward agentic workflows and online supervised fine-tuning. These techniques enable organizations to create data flywheels, turning production traces into continuous improvements while maintaining high performance and efficiency in compute-constrained environments.