ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs

Xiaol.x - ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs