Modal

Nemotron: Open models, data, & weights | Bryan Catanzaro VP at NVIDIA, Runtime 2026

The future of software engineering | Cat Wu, Dax Raax, Swyx, Runtime 2026

It's time to be ambitious | Scott Wu Co-founder & CEO at Cognition, Runtime 2026

Computational Biology | Theo Karaletsos, Kim Branson, Amir Shanehsazzadeh, Runtime 2026

jev: on | Diogo Almeida founder of Typesafe AI, Runtime 2026

Runtime 2026 Product Launch Keynote | Erik Bernhardsson, CEO @ Modal

The rapid evolution of generative AI necessitates a fundamental shift in software infrastructure, moving beyond traditional cloud limitations to support high-performance, scalable applications. New infrastructure building blocks are essential for managing the complexities of LLM inference, such as KV caching and dynami...

Inside Cognition's inference stack: RL, speculative decoding & DFlash

Reinforcement learning (RL) and inference optimization are increasingly intertwined, as RL rollouts rely heavily on efficient inference throughput and low latency. The development of DFlash, a diffusion-based speculative decoding method, marks a significant advancement by achieving higher token acceptance lengths and c...

Making Agents Useful: MCP, Tool Calling, and Sandboxed Execution

FastMCP provides an ergonomic Python framework for building Model Context Protocol (MCP) servers, enabling agents to interact with external tools. Traditional MCP implementations suffer from "context bloat," where injecting extensive tool schemas into an LLM's context window degrades performance and increases latency d...

Running real-time applications on Modal: Low-Latency Voice Agents

Voice AI application development requires robust serverless infrastructure to manage fluctuating workloads and minimize latency. Core components—including speech-to-text, text-to-speech, voice activity detection, and diarization—perform most efficiently when deployed as independent, auto-scaling services. Optimizing pe...

Specific Intelligence at Scale: How Applied Compute Trains Custom Agent Workforces on Modal

"Specific intelligence" enables enterprises to transcend off-the-shelf frontier models by embedding proprietary data and institutional expertise into autonomous, non-static AI agents. Applied Compute, led by CEO Yash Patil, facilitates this transition through reinforcement and continual learning, allowing companies to ...

Intro to Fine-Tuning on Modal: SFT, RL, and more

Training custom machine learning models offers a path to greater differentiation and performance optimization compared to relying solely on general-purpose frontier APIs. As products evolve into domain-specific applications, businesses often reach bottlenecks in latency, throughput, and cost, signaling a need for custo...

Scaling Reinforcement Learning on Modal

Scaling reinforcement learning (RL) infrastructure on Modal relies on leveraging core primitives like functions, volumes, and sandboxes to optimize stability and throughput. Training efficiency improves by utilizing multi-node clusters and parallelizing workloads, while maintaining warm pools of sandboxes ensures rapid...

Building AI agents from scratch using the OpenAI Agent SDK and Modal.

Building custom AI agents requires a robust "harness"—the supporting infrastructure surrounding an LLM—to manage context windows and prevent performance degradation known as "context rot." Effective agent design relies on strategies like sub-agents, which encapsulate complex tasks to keep the primary orchestrator's con...

Truly Serverless GPUs: A Deep Dive Inside Modal's Fast Cold Starts

Serverless GPU infrastructure requires balancing variable inference demand with resource efficiency to avoid the high costs of over-provisioning. Modal addresses this by maintaining a buffer of pre-warmed instances and employing lazy-loading file systems to bypass standard container startup delays. By utilizing gVisor ...

Speculation is all you need: Intro to Speculative Decoding for High Performance Inference

Speculative decoding optimizes large language model (LLM) inference by employing a smaller, faster draft model to propose token sequences that a larger target model verifies in parallel. This technique exploits spare GPU compute capacity to generate multiple tokens per forward pass, significantly reducing inter-token l...

Inside Modal Sandboxes: How Agents Code at Scale

Modal Sandboxes provide isolated environments for executing untrusted AI-generated code at scale, enabling iterative development loops for autonomous coding agents. Key architectural patterns include separating the agent harness from the execution environment to enhance security and modularity, while utilizing file sys...

How Ramp built a background coding agent that writes over half of its pull requests

High Performance LLM Inference in Production

High-performance LLM inference in production requires tailoring infrastructure to specific workload demands, categorized into online, offline, and semi-online patterns. Online applications, such as chat interfaces, prioritize low latency and interactivity, while offline jobs focus on throughput for large-scale batch pr...

⚡Blazing Fast LLaMA 3: Crush Latency with TensorRT LLM

Your Self-Hosted Chatbot Just Went Viral—Can It Handle the Traffic?