I am a machine learning engineer with over 20 years of experience. I have worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. I have also led and contributed to numerous popular open-source machine-learning tools. I am currently an independent consultant helping companies operationalize Large Language Models (LLMs) to accelerate their AI product journey.
LLM inference performance hinges on understanding the distinct bottlenecks of the pre-fill and decode phases. Pre-fill, which determines time to first token, is compute-bound, whereas the decode phase is limited by memory bandwidth and dictates overall latency. In multi-step agentic systems, cumulative cost and latency...
The podcast episode centers on a review and discussion of Braintrust, an evaluation tool, with Wayde Gilliam from Braintrust demonstrating its functionalities for completing specific homework assignments related to building a recipe chatbot. The discussion covers creating datasets by importing user queries and metadata...
Follow this podcast in Podwise
Sign in to get AI summaries, transcripts and mind maps for any episode, including new ones.
