arxiv preprint - Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck

AI Breakdown - arxiv preprint - Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck