Efficient GPU infrastructure at LinkedIn // Animesh Singh // MLOps Podcast #299

Scaling large language models (LLMs) and agentic applications requires significant infrastructure investment, particularly in GPU efficiency and memory management. While training costs have stabilized through open-source models and fine-tuning techniques, inferencing remains a major financial and technical bottleneck. Optimizing GPU utilization through kernel fusion—such as the Liger kernels—and implementing hierarchical checkpointing strategies are essential to managing long-running training jobs and reducing idle capacity. Moving forward, the industry faces a critical architectural decision: whether to integrate LLMs into traditional recommendation ranking systems or maintain separate, specialized pipelines. Standardizing machine learning platforms on flexible, experiment-oriented orchestration engines like Flyte allows organizations to bridge the gap between traditional ML and generative AI, ultimately simplifying talent acquisition and compliance while maximizing infrastructure ROI.