DelphiGet started →

Never outsource your understanding

Delphi makes understanding new things scalable.

DelphiReader
SSCSW
SCALING LAW · 15 SECTIONS
🍷
Scaling Laws, Carefully | Lil'Log
https://lilianweng.github.io/posts/2026-06-24-scaling-laws/
Lil'Log|
PostsArchiveSearchTagsFAQ

Scaling Laws, Carefully

Date: June 24, 2026 | Estimated Reading Time: 25 min | Author: Lilian Weng
▸ Table of Contents

Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss L decreases predictably as we scale up model size N, dataset size D, and compute C, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between N and D.

This predictability makes scaling laws highly valuable in practice. A common workflow is to fit scaling laws on a handful of small runs and then extrapolate to estimate the token and compute requirements for larger models.

Reply
Scaling Laws for Efficient MoE Language Modelsp. 1
(a) Activation ratio
(b) Granularity
(c) Compute budget
Figure 8  Scaling Behavior of Efficiency Leverage (EL). (a) With fixed granularity, EL follows a power law with respect to the activation ratio across all tested compute budgets. (b) With a fixed activation ratio, EL's scaling with granularity conforms to a log-polynomial law. (c) With both held constant, EL scales with compute according to a standard power law.

Interaction of Efficiency Leverage and Granularity. There exists an optimal granularity that maximizes the EL; under fixed FLOPs budget C and activation ratio A, the relationship between EL and granularity follows a log-polynomial pattern.

Prev14 / 37Next
RELATED KNOWLEDGEFull Graph →
+13 more
Empirical scaling laws
Power laws between size, data and loss.
Scaling Law
Domain hub
MoE scaling
Efficiency leverage and sparsity.
Harness engineering
The eval loop around the model.
@Scaling Laws for Efficient MoE… , what's new in that MoE scaling paper?|
Waiting for Standard capacity · 2s
The MoE paper adds an architecture-specific efficiency law. It does not replace the post's Chinchilla-style question.
Its main extension beyond ordinary compute-optimal sizing is Efficiency Leverage MoE paper, pp. 14–15.
Ask questions in context
GPT-5.6 Luna xHighADHD↑
RELATED KNOWLEDGEFull Graph →
Empirical scaling laws
Harness engineering
1Scaling Laws for Efficient MoE Language Models — Ant Group (2507.17702v2)
2Chinchilla (2203.15556v1)
3Scaling Laws for Native Multimodal Models (2504.07951v4)
4Why Larger Models Learn More (2605.29548v2)
…it is about how to allocate precious compute optimally between N and D.×
@Scaling Laws for Efficient MoE… , what's new in that MoE scaling paper?|
GPT-5.6 Luna xHighADHD↑
Highlight any live web page, and reply from it

Knowledge base

Less climb.
Same summit.

Chat made you faster, not smarter: a single thread forgets what you know and restarts at the summary. In Delphi every answer writes into a cited wiki, and every new question reads it first — skipping what you already worked out.

DelphiKnowledge
CONCEPT

Model Scaling Versus Data Scaling

At a fixed training-compute budget, Chinchilla finds that model parameters and training tokens should grow at approximately equal rates, making data allocation as important as model size.

›  Compiled from Chinchilla (2203.15556v1) · 5 passages10 pages link here

Optimal parameter count N and training-token count D should grow at roughly equal rates with compute: doubling the model should be accompanied by roughly doubling the data.E1 E4

The fitted scaling view that makes the model-size / data trade-off legible.
RELATED KNOWLEDGEFull Graph →
+60 more pages in list
Chinchilla
Compute-optimal parameter and token counts.
Why Larger Models Learn More
Capacity, interference and rare-task retention.
Cosine schedules must match
The caveat behind the fitted frontier.

Write — the answer you just read is filed as a cited page, not a scrolled-past message.

Features

Built for the way you actually read

MCP

Your knowledge base, everywhere else

Not shipped yet, and next in the backlog: Notion, your coding agent and Slack reading the wiki Delphi wrote — so the tools you already use start from what you already know.

1 / 4
MCPNEXT IN THE BACKLOG

Your knowledge base becomes a server the tools you already use can read: Notion for the writing, a coding agent for the work, Slack for the asking.

Try Delphi now

Try web appDownload for macOSSoon@curiouskid423

Delphi does not do your thinking for you. It makes you smarter.