A1 Vertaisarvioitu alkuperäisartikkeli tieteellisessä lehdessä

Scaling Data-Constrained Language Models




TekijätMuennighoff, Niklas; Rush, Alexander M.; Barak, Boaz; Le Scao, Teven; Piktus, Aleksandra; Tazi, Nouamane; Pyysalo, Sampo; Wolf, Thomas; Raffel, Colin

KustantajaMICROTOME PUBL

KustannuspaikkaBROOKLINE

Julkaisuvuosi2025

JournalJournal of Machine Learning Research

Tietokannassa oleva lehden nimiJOURNAL OF MACHINE LEARNING RESEARCH

Lehden akronyymiJ MACH LEARN RES

Artikkelin numero53

Vuosikerta26

Sivujen määrä66

ISSN1532-4435

eISSN1533-7928

Verkko-osoitehttps://www.jmlr.org/papers/v26/24-1000.html

Rinnakkaistallenteen osoitehttps://research.utu.fi/converis/portal/detail/Publication/492253404


Tiivistelmä
The current trend of scaling language models involves increasing both parameter count and training data set size. Extrapolating this trend suggests that training data set size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approach esmitigating data scarcity, including augmenting the training data set with code data or removing commonly used filters. Models and data sets from our 400 training runs are freely available athttps://github.com/huggingface/datablations.

Ladattava julkaisu

This is an electronic reprint of the original article.
This reprint may differ from the original in pagination and typographic detail. Please cite the original version.





Last updated on 2025-05-06 at 12:33