Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah-Hill Smith

The podcast explores the evolution and business model of Artificial Analysis (AA), a company specializing in independent AI benchmarking. Founders Micah-Hill Smith and George Cameron discuss AA's origins as a side project born from the need for objective model evaluation, detailing their transition to a sustainable business with over 20 employees. They generate revenue through enterprise subscriptions and private benchmarking services, ensuring that public benchmarks remain unbiased. The conversation covers the technical challenges of AI evaluation, including prompt engineering, parsing, and the importance of statistical rigor. They also discuss the company's AI Grant experience and their efforts to develop new evaluation metrics, such as the Omniscience Index for measuring hallucination. The podcast further examines trends in AI, including the declining cost of intelligence, hardware efficiency, and the increasing importance of token efficiency.