Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal

RL post-training efficiency hinges on decoupling the trainer from the rollout fleet to overcome the constraints of tightly coupled RDMA clusters. By shifting from full checkpoint synchronization to sparse, bit-level weight diffs, rollout workers can operate on elastic, distributed compute across global regions. This approach leverages the observation that Adam optimizer updates and finite precision rounding cause only about 1% of weights to change per step, reducing data transfer requirements from hundreds of gigabytes to mere megabytes. The "Stitch" framework implements this architecture by allowing trainers to publish mutable weight versions that rollout engines pull and reconstruct locally. This paradigm transforms scattered inference capacity into a unified, scalable RL fleet, enabling more flexible and cost-effective training workflows without sacrificing bit-level accuracy or requiring constant, high-bandwidth interconnects between all components.