Vincent Chen, co-founder at Snorkel AI, announces an expansion of Snorkel's Open Benchmarks Grants program to $30 million to keep evaluation infrastructure in line with accelerating frontier model capabilities. He breaks down the program's three core pillars—funding new benchmarks, launching the Open Benchmarks Red Team, and establishing the Snorkel Research Fellowship—while addressing continuous benchmark maintenance and AI alignment. Check out our sponsor Lovable: https://lovable.dev. Turn ideas into software people love.

MTS is a live news and interview show covering technology, business, politics, and culture as it happens.

Follow MTS:

YouTube | ⁠https://www.youtube.com/@mtsituation⁠

Spotify | ⁠https://open.spotify.com/show/4HUNNmV1pjV7CW6Lphm1rM⁠

Apple | ⁠https://podcasts.apple.com/us/podcast/mts/id1891088763⁠

Substack | ⁠https://mtslive.substack.com/⁠

Interested in sponsoring the show? ⁠⁠⁠⁠⁠⁠sponsors@mts.now⁠⁠

Brought to you by:

Mongo DB - Anyone can prototype an agent. Learn how to get them into production on September 30th in New York - register at https://www.mongodb.com/events/mongodb-local/nyc

Lovable - Turn ideas into software people love with Lovable. Start today at https://lovable.dev/

Larridin - Finally prove what your AI investment is worth. Head to https://larridin.com/ today to start measuring your AI’s bottom line.

Neon (by Databricks) - Get up to $100K in credits for your startup at ⁠https://neon.com/mts⁠

ElevenLabs - AI that communicates at human level across every channel and modality. https://elevenlabs.io/mts

Arena - Measuring AI performance in the real world. ⁠https://arena.ai/⁠

Kong - The AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. https://konghq.com

Today’s stream is sponsored by VCX, the public ticker for private tech. Head over to http://getvcx.com to learn more

Blitzy - Autonomous software development for enterprise codebases. Ship 5x faster. https://blitzy.com

AdQuick - Make your brand a billboard. Out-of-home advertising as easy to scale as digital. https://adquick.com

Chargebee - Billing and monetization infrastructure for the AI era. https://www.chargebee.com/

Timestamps:

(00:00) Introduction

(02:09) Snorkel Research Fellowship

(03:54) Open Benchmarks Red Team

(05:02) Auditing Broken Benchmarks

(07:32) Benchmarks in AI Alignment

(08:17) Science of Empirical Benchmarking

(09:48) Closing the Capability Gap

Note: This podcast is not investment, legal, or tax advice, and is intended for informational and entertainment purposes only. Hosts and guests may hold positions in the companies and securities discussed; do your own research before acting on anything you hear.

Why Frontier AI Evals Are Falling Behind | Vincent Chen

MTS

Why Frontier AI Evals Are Falling Behind | Vincent Chen