For plain SGD, the scaling rule with the most primary support is linear: multiply η by k when the batch grows by k. Goyal et al. (2017) write it as η = 0.1 × kn/256 for minibatch size kn, and report that it matched the original training curve for batch sizes up to 8k on ImageNet, with warmup. For Adam-style optimizers the evidence is weaker, and a square-root rule is the main alternative. Neither is universal.
Terms and conventions
Batch size B is the number of examples per optimizer step. The learning rate η is the step-size multiplier applied to the gradient. An epoch is one pass over the dataset. Sources differ on whether convergence is counted in steps or in examples processed, and this article states which. McCandlish et al. define the critical batch size B_crit as the batch size beyond which larger batches no longer improve convergence speed measured in examples processed. Their gradient noise scale, B_noise, is the companion quantity. Global batch size is often set by device count and interconnect; see network topology for distributed training.
The learning rate batch size scaling rule for SGD
Goyal et al. tested the rule on ImageNet with ResNet-50 and a warmup schedule. The 8k result belongs to that recipe and should not be generalised. A secondary guide (arXiv 1810.11787) describes the rule as multiplying the learning rate by k when the batch is multiplied by k, and describes warmup of about five epochs from a small learning rate. It says applying the full scaled rate from the first step leads to divergence. The five-epoch figure has not been checked against the primary text.
Why linear scaling works, and what is disputed
The noise account runs as follows. A larger batch gives a less noisy gradient estimate, so a larger step is safer, and scaling η with B keeps noise per update roughly comparable. McCandlish et al. (arXiv 1812.06162) develop this as an empirical model; the exact expression was not checked here. The existence of the effect is better supported than the explanation for it. A Random Matrix Theory paper (arXiv 2006.09092) calls the assumption behind the Goyal justification unrealistic. The research locators for that critique come from a JMLR paper, so the attribution needs confirming. The JMLR paper (volume 23, 20-1258) derives a linear rule for SGD up to a threshold, under its stated assumptions.
Adam: the square-root rule and the evidence against it
For adaptive methods the proposed rule is square-root scaling, multiplying η by √k. You et al., as reported in a 2026 preprint (arXiv 2601.05034), found empirically during BERT training that square-root scaling performed better. Liu et al., per the same preprint, argue that under Adam the gradient noise variance scales with η²/B, so holding variance constant requires square-root scaling. The JMLR paper reports the same result for adaptive methods with small fixed damping.
The counter-evidence is narrower. In a single-task language-model LSTM study (arXiv 1808.01371), Adam diverged at large batch sizes under both linear and square-root scaling. Batches of 2k–8k converged with a learning rate of 3e-3 decayed to zero over one epoch. Li et al. (NeurIPS 2024, arXiv 2405.14578) report that the optimal learning rate first rises and then falls as batch size grows, with the peak moving toward larger batches as training progresses. They argue that neither linear nor square-root scaling is the optimum for Adam at large batch sizes. Their theory assumes sign-based updates and a normal gradient distribution, so it is an approximation. The research does not establish which rule holds across general workloads. The Krizhevsky 2014 attribution is also unresolved: a 2026 preprint describes it as proposing square-root scaling for SGD, while a 2019 paper (arXiv 1905.06236) says he reported linear scaling worked best. The primary paper was not retrieved.
Critical batch size and the optimal learning rate
A 2026 preprint (arXiv 2601.05034) gives the optimal learning rate as η_opt = η_max / (1 + B_noise/B). This expression has not been verified against McCandlish et al. Under it, when B is much less than B_noise the optimal rate scales roughly linearly with B, and when B is much greater than B_noise it approaches its maximum. No source gives a fixed B_crit for general use. Because Li et al. report that the optimum drifts during training, a critical batch size measured early may not hold later. That inference is the author's reading.
Warmup: what it is for and where it fails
Warmup ramps the learning rate from a small value up to the scaled target, which avoids the early divergence described above. Warmup and learning-rate decay are covered in learning rate schedules and training stability. A 2018 slide deck from a Lawrence Berkeley National Laboratory event (Mustafa) says linear warmup breaks down at extreme batch sizes and cites an 8k ImageNet ceiling. That is a secondary, low-confidence source, and the 8k figure should not be generalised.
What breaks: divergence, step count and generalization
Divergence is the most direct failure and is tied to missing warmup or over-aggressive rates (arXiv 1810.11787; arXiv 1808.01371). A separate LLM-scaling study (arXiv 2412.01505) reports a learning-rate ceiling caused by divergence. Its date conflicts with its identifier and needs checking.
At a fixed data budget, a larger batch means fewer steps. The author's reading is that below B_crit, doubling B roughly halves the steps needed, while above it the step count barely falls as examples consumed keep rising. Only secondary blogs in the research support this, so treat it as a reading of the B_crit definition, not a measurement.
Keskar et al. (arXiv 1705.08741) observed that large batches train to models that generalise less well, even when training runs until the loss stops improving. Hoffer et al. argue the gap is largely about the number of updates and can be reduced by adapting the training regime. The sharp-minima explanation for the gap remains a hypothesis here, and its critiques were not retrieved.
Beyond SGD and Adam
LARS and LAMB are layerwise adaptive optimizers used at very large batch sizes. The LAMB report (UC Berkeley EECS-2019-103) uses untuned square-root scaling with linear-epoch warmup for BERT and ResNet-50. That is one lab's claim. SimCLR (arXiv 2002.05709) found square-root scaling improved performance at smaller batches with fewer epochs, with saturation around batch size 8192 and longer training helping. In federated learning, linear scaling is often too aggressive, and a federated paper (arXiv 2106.07820) reports mixed efficacy for server learning-rate scaling. None of these settings transfers the SGD or Adam rules automatically.
Checks before relying on these rules
- Verify the Goyal et al. expression, warmup length and 8k conditions against the primary PDF.
- Verify the optimal learning-rate formula against McCandlish et al. before quoting it.
- Retrieve Shallue et al. (2018) and Krizhevsky (2014). Neither was checked here.
- Resolve the date of arXiv 2412.01505.
- Confirm which paper makes the Random Matrix critique.
No established diagnostic detects that a run has passed its critical batch size. As the author's reading, a practical test is a short sweep at two batch sizes with identical warmup, comparing steps, not wall-clock time, to a target loss.