Scaling Laws · Compute-Optimal Training

Hoffmann Scaling Laws in Your Browser

Chinchilla Lite reproduces the Hoffmann et al. 2022 compute-optimal allocation rule end-to-end on a laptop-scale nine-run grid. A Python driver fits the parametric loss surface L(N, D) = E + A / Nα + B / Dβ, writes the coefficients to data/grid.json, and this page reads them back and re-derives the compute-optimal frontier interactively. The same fit is reused across every panel so every number you see traces to one source of truth.

Fit summary

alpha (model exponent)
— —
beta (token exponent)
— —
Irreducible loss E
—
R² on log-loss
—
Source
—

Confidence intervals are 5th and 95th percentiles of a 200-sample bootstrap over the nine runs.

Explore the fit

Six panels, one data payload. Tab between them with the arrow keys; each panel carries both a chart and a tabular mirror so the exploration is identical for sighted and screen-reader users.

Final loss vs. parameter count

Each marker is one run. Shape and colour jointly encode the token budget, so the chart reads without colour vision. Use Tab to focus individual points; each point announces its configuration and final loss.

Method in one paragraph

We synthesise a nine-run grid across three parameter counts (1.2 M, 4.8 M, 19.2 M) and three token counts (10 M, 40 M, 160 M) drawn from a Hoffmann-form ground truth close to the published values, then refit the four free parameters on the log-loss using Nelder-Mead with a Huber loss. The fit normalises N by 107 and D by 109 for numerical stability and undoes the scaling on return, so the reported coefficients are in native units and line up directly with the ground truth. The same Python driver includes a real-training variant at scripts/train_one.py that trains a small transformer on TinyStories and writes the same JSON schema — either source works here.

Credits and references