Hoffmann Scaling Laws in Your Browser
Chinchilla Lite reproduces the Hoffmann et al. 2022 compute-optimal allocation rule end-to-end on a laptop-scale nine-run grid. A Python driver fits the parametric loss surface L(N, D) = E + A / Nα + B / Dβ, writes the coefficients to data/grid.json, and this page reads them back and re-derives the compute-optimal frontier interactively. The same fit is reused across every panel so every number you see traces to one source of truth.
Fit summary
- alpha (model exponent)
- — —
- beta (token exponent)
- — —
- Irreducible loss E
- —
- R² on log-loss
- —
- Source
- —
Confidence intervals are 5th and 95th percentiles of a 200-sample bootstrap over the nine runs.
Explore the fit
Six panels, one data payload. Tab between them with the arrow keys; each panel carries both a chart and a tabular mirror so the exploration is identical for sighted and screen-reader users.
Final loss vs. parameter count
Each marker is one run. Shape and colour jointly encode the token budget, so the chart reads without colour vision. Use Tab to focus individual points; each point announces its configuration and final loss.
Predicted loss surface and compute-optimal frontier
A predicted-loss colour field across the (N, D) plane derived from the fitted coefficients. The overlaid line is the compute-optimal frontier from 1019 to 1022 FLOPs. A 5 by 5 tabular mirror of the same surface follows below the figure.
Compute-optimal allocation at a chosen budget
Drag the slider or press the arrow keys to pick a budget in log10 FLOPs. Arrow keys move by one decile; the slider is lined up with the seven preset ticks baked into the grid. The predicted optimum updates live, announced politely to screen readers on every change.
Per-run training loss curves
One polyline per run, each with a distinct dash pattern so the panel is legible in monochrome. Each line is keyboard-focusable and announces its configuration, final loss, and the number of logged steps.
Results table
All nine runs with observed loss, fit prediction, and residual. Column headers are buttons — click or press Enter to sort; the new ordering is announced via the polite region.
Fit residuals
Observed minus predicted final loss. A zero line marks perfect agreement; vertical stems make the sign of each residual obvious without relying on colour. Focus a point to hear its observed value, prediction, and residual.
Method in one paragraph
We synthesise a nine-run grid across three parameter counts (1.2 M, 4.8 M, 19.2 M) and three token counts (10 M, 40 M, 160 M) drawn from a Hoffmann-form ground truth close to the published values, then refit the four free parameters on the log-loss using Nelder-Mead with a Huber loss. The fit normalises N by 107 and D by 109 for numerical stability and undoes the scaling on return, so the reported coefficients are in native units and line up directly with the ground truth. The same Python driver includes a real-training variant at scripts/train_one.py that trains a small transformer on TinyStories and writes the same JSON schema — either source works here.
Credits and references
- Hoffmann et al. 2022, Training Compute-Optimal Large Language Models, the source of the parametric loss form and the compute-optimal allocation rule.
- Kaplan et al. 2020, Scaling Laws for Neural Language Models, the precursor work that motivated the extrapolation protocol.
- Accessibility scaffolding follows the W3C ARIA Authoring Practices Guide tabs pattern and WCAG 2.2 AA targets.
- Fonts: Inter (Rasmus Andersson) and JetBrains Mono (JetBrains), both under the SIL Open Font License.