Scaling Laws and Interpretability of Learning from Repeated Data

This research paper studies the effects of repeating small fractions of training data on large language models, revealing a strong double-descent phenomenon where certain repetition frequencies cause large drops in test performance by inducing memorization and damaging internal generalization mechanisms such as induction heads.