On learning rate schedule training stability, the sources reviewed here support warmup most strongly. Decay shape is contested, and most decay studies measure final loss rather than spike frequency or divergence. I did not check the original documents for any figure below; each comes from a retrieved summary or search-result text, so verify before reuse.
Definitions and scope
A learning rate schedule maps the training step to the step size the optimizer uses. Warmup is an initial phase in which that step size rises to a peak. Decay is a later phase in which it falls. A loss spike is a sudden, transient rise in training loss. This article uses divergence loosely, for a run that fails to converge; I could not confirm a single operational threshold shared by the cited papers. The NeurIPS 2024 paper calls a large loss increase after a high-rate start a loss catapult and ties it to sharpness, the curvature of the loss surface. Most of the evidence comes from Adam or AdamW language-model pretraining and from SGD on small image models.
Warmup: the best-supported stabilizer
A 2024 NeurIPS paper (arXiv 2406.09405) reports that Adam trained at high initial learning rates without warmup can produce large catapults that cause training failures. Longer warmup, it reports, lets preconditioned sharpness decrease naturally before the rate is high, which prevents early spikes. It also reports that longer warmup makes catapults milder, pushing the divergence boundary to higher peak rates. This is one research group's analysis, and the authors note that momentum oscillations make warmup-induced spikes hard to separate from ordinary fluctuation.
An early Transformer tuning report (arXiv 1804.00247) gives the warmup-length result most often cited. There, divergence usually occurred within the first hours, when the actual learning rate was highest. Raising warmup from 16k to 32k steps allowed training at peak rates of 0.30 and 0.50 without divergence. The retrieved summary gives no model or batch size, and the posting date is not confirmed.
Where the stability boundary sits
A separate preprint on a 124M-parameter GPT-2 model (arXiv 2410.23922) reports that warmup stabilizes higher learning rates and prevents a performance gap that longer training does not close. Each of these results holds for specific runs. I found no replication at other scales and no general law linking warmup length to a maximum safe rate. The author's reading, not a finding: the right warmup length probably depends on peak rate, depth and batch size, and none of the retrieved sources varied those systematically.
Architecture changes and the curvature proposal
The Taming Curvature preprint (arXiv 2606.16768) proposes that in large Transformers, preconditioned curvature spikes mostly during learning rate warmup and grow with depth. In its runs, 16- and 32-layer models diverged while an 8-layer model converged. It also proposes architecture warm-up as a substitute for learning rate warmup. In one setup, a 16-layer, 1B-parameter Transformer trained this way with no learning rate warmup converged on par with the baseline and was more stable. This is one preprint at one scale, and its posting date is unverified. Its QK-Norm and Softcap baselines spiked frequently at a peak rate of 8×10^-3. Those runs probed a lower stability threshold and are not typical settings.
Decay shape: contested
The original Transformer schedule used inverse-square-root decay after warmup, which did not require the total step count in advance. Cosine decay later became common in LLM pretraining. This history comes from a 2025 survey (arXiv 2510.06826), a secondary source.
Defazio et al., as cited in that survey, report linear decay consistently outperforming cosine across optimizers. The primary paper (arXiv 2310.07831) reports linear decay beating cosine across all model sizes it tested, and notes that cosine under-performs when training for under 30 epochs on CIFAR10. Bergsma et al. (arXiv 2502.15938, a preprint) report that linear decay to zero outperforms other schedules at compute-optimal dataset sizes, with the benefit growing as dataset size grows. Cosine remains common in practice. I found no independent systematic replication across these schedules, so any statement that linear is better needs those qualifications.
An Index SLM technical report (arXiv 2607.09885) found cosine, linear and WSD reached essentially the same final validation loss on a 0.1B model. Its WSD run had higher loss during the constant phase, then dropped rapidly once decay began. The report says the schedule alone 'matters little at convergence'. It compared final loss, not spike frequency.
WSD (warmup, stable constant phase, decay) is widely used. The primary paper (arXiv 2507.09846) describes it and its checkpointing benefits. Claims that WSD is superior in final loss and stability mostly trace to aggregator pages citing Hu et al. (2024). I could not retrieve that original, so I give no figures attributed to it.
Does a stable schedule give a better model?
Smoother curves do not establish a better model. In the Avey study (arXiv 2506.11305), cosine gave lower loss than a constant schedule across the rates tested, but the lowest loss did not always match the best downstream performance. The setting is architecture-specific. A paper listed under ICLR 2026, summarised on a paper-notes page, proposes Warmup-Stable-Only, which removes the decay phase. It reports worse pre-training metrics but better results after supervised fine-tuning, which it attributes to flatter minima. I have not checked this against arXiv 2603.16127, so treat it as unconfirmed.
Open questions
- No retrieved study measures spike frequency or divergence rate across cosine, linear and WSD on the same model.
- No primary source covers recovery from a mid-run divergence, such as restarting from a checkpoint at a lower rate or with longer warmup.
- Whether schedule choice, not only peak rate, causes instability at LLM scale is not established.
- Schedule-free optimization is within the scope of arXiv 2507.09846, but I did not verify its performance claims.
Author's reading, not a finding: when a run spikes, the evidence here most supports checking warmup length and peak rate before changing decay shape. Whether a difference between schedules exceeds run-to-run noise needs several seeds. The sample-size reasoning is in statistical power when comparing machine learning models.