Optimizers (SGD · Momentum · Adam)

FoundationsIntermediate~8 min

Optimizers (SGD · Momentum · Adam) — Optimizers are the update rules that drive gradient descent. Stochastic gradient descent steps on noisy mini-batch gradients; momentum accumulates velocity to power through ravines; Adam adapts a per-parameter step size. They differ most on hard surfaces like ravines and saddles.

Plain gradient descent zig-zags through narrow valleys and stalls on plateaus. Momentum gives it inertia like a rolling ball; Adam adapts each parameter’s step size automatically. Race all three down the same surface and the differences are obvious.

Loading 3D view…

SGD
Momentum
Adam

Optimizer controls

Data

Surface

Steep one way, shallow the other — the case momentum and Adam were invented for.

Start θ₁-2.6

Start θ₂2.4

All three optimizers launch from the same point — drag it on the contour map.

Model

Learning rate α0.080

Step budget120

Momentum β0.90

β = 0 makes Momentum identical to SGD; near 1 it sails past the valley floor.

Adam β₂ (per-axis scale)0.999

Inspect

β₁ = 0.9, β₂ = 0.999, ε = 1e-8 — a running mean of the gradient divided by a running size, so every axis gets its own step length.

SGD14.528

Momentum14.528

Adam14.528

Playback

Step 0 / 120

Speed

Plain step
+ momentum
+ per-axis scale
↻

Step 0 of 120 — SGD 14.528, Momentum 14.528, Adam 14.528 — SGD is ahead, and Adam sits at (-2.60, 2.40)

Break it

At α = 0.6 the ravine's steep axis has |1 − αk| = 1.4 > 1, so SGD and Momentum overshoot further every step. Adam divides by its own running gradient size, which is exactly why its step stays bounded when theirs does not.

The idea in plain words

Plain gradient descent always steps straight downhill, which zig-zags painfully across a narrow valley. Optimizers change the update rule. Momentum accumulates velocity like a rolling ball, powering through ravines. Adam adapts a separate step size for each direction, so steep and shallow axes both move sensibly.

Race all three down the same surface and the differences are obvious: on a ravine, SGD stutters, Momentum overshoots and recovers, Adam glides. On a saddle, plain SGD can stall where the gradient nearly vanishes.

Now, the math

Each optimizer transforms the raw gradient before stepping:

\text{SGD:}\quad \theta \leftarrow \theta - \eta\, g

\text{Momentum:}\quad v \leftarrow \beta v + g,\quad \theta \leftarrow \theta - \eta\, v

\text{Adam:}\quad \theta \leftarrow \theta - \eta\, \frac{\hat{m}}{\sqrt{\hat{s}} + \epsilon}

$g$: the gradient of the loss at the current point.
$v$: momentum’s velocity — an exponential average of past gradients.
$\hat{m},\ \hat{s}$: Adam’s bias-corrected first and second moment estimates.

▸ Show the derivation

Momentum’s β (here 0.9) means each step remembers ~10 previous gradients, cancelling the side-to-side oscillation in a ravine while reinforcing the consistent downhill direction. Adam divides by √ŝ, so a direction with large gradients gets a smaller effective step — which is why it handles badly-scaled surfaces that cripple plain SGD.

Trace it by hand

All three optimizers start at (2, 2) on the ravine surface — loss equals half of (4 times x squared plus 0.35 times y squared) — with eta = 0.1 and momentum beta = 0.9, exactly as in the race interactive. Numbers come from the repo's own optimizer code, rounded to 2 decimals.

The gradient at the shared start
$g = (4x,\ 0.35y) = (8,\ 0.7), \qquad L(2,\ 2) = 8.7$
The steep wall of the ravine makes the x gradient about 11 times the y gradient — this imbalance is the whole story.
SGD: step straight downhill
$\theta_1 = (2,\ 2) - 0.1\,(8,\ 0.7) = (1.2,\ 1.93), \qquad L = 3.53$
Its second step lands at (0.72, 1.86) with loss 1.64 — progress along the valley floor stays painfully slow.
Momentum: velocity accumulates
$v_2 = 0.9\,(8,\ 0.7) + (4.8,\ 0.68) = (12,\ 1.31), \quad \theta_2 = (1.2,\ 1.93) - 0.1\,v_2 = (0,\ 1.80)$
Momentum's first step matches SGD (velocity starts at zero); by step two the remembered gradient makes the x move 2.5 times bigger.
Adam: normalize each direction
$\frac{\hat{m}}{\sqrt{\hat{s}}} = \frac{g}{|g|} = (1,\ 1) \;\Rightarrow\; \theta_1 = (2,\ 2) - 0.6\,(1,\ 1) = (1.4,\ 1.4)$
At step one the bias corrections cancel, leaving g over its magnitude per axis. The demo scales Adam's eta by 6, giving the 0.6 step; step two lands at (0.81, 0.81).
Loss after two steps of each
$L_{\text{SGD}} = 1.64, \qquad L_{\text{Mom}} = 0.57, \qquad L_{\text{Adam}} = 1.44$
Same surface, same start, same learning rate — only the update rule differs.

What just happened: From an 11-to-1 gradient imbalance, SGD crawled along y while momentum's accumulated velocity slammed x to exactly 0 in two steps, and Adam ignored the imbalance entirely, stepping 0.6 on both axes. The transform applied to the raw gradient is what separates them.

Now Break It

Try this: Raise the learning rate until all three diverge; pick a saddle where plain SGD stalls.

Control: Learning-rate slider (set high) / surface picker (saddle)

What happens: Diverged! The learning rate is too high — every optimizer overshoots and the loss explodes.

Where optimizers (sgd · momentum · adam) is used

Optimizers decide how the gradient turns into an actual parameter update, and the difference between them shows up in real training runs. Plain stochastic gradient descent, often with a momentum term, remains the standard for training image classifiers like ResNet, where it frequently produces the best final accuracy. Momentum accelerates progress by accumulating a velocity across steps, helping the update roll through flat regions and dampen oscillations in narrow valleys. Adam, which adapts a separate step size for each parameter using running estimates of the gradient and its square, is the default for training transformers and large language models because it converges quickly and tolerates sparse, noisy gradients. Variants such as AdamW, RMSProp, and Adagrad appear across recommendation systems, speech models, and reinforcement learning where different gradient behaviors demand different update rules.

A common misconception is that Adam is always better than SGD. Adam usually trains faster and needs less learning-rate tuning, but well-tuned SGD with momentum often generalizes better on vision benchmarks, which is why both persist. A second pitfall is thinking an optimizer removes the need to set a learning rate. Every optimizer here, including Adam, still has a base learning rate that must be chosen, and a bad value causes divergence or stagnation regardless of the method. People also confuse the optimizer with the loss function: the loss defines what to minimize, while the optimizer defines how to move parameters toward that minimum. Choosing an optimizer is about the update rule, not the objective itself.

Frequently asked questions

What does an optimizer do in machine learning?

An optimizer is the rule that turns the computed gradient into an actual change to the model's parameters. It decides the direction and size of each update, and may incorporate memory of past gradients to move more effectively. Gradient descent is the simplest optimizer, while SGD with momentum and Adam are more sophisticated variants.

What is the difference between SGD, Momentum, and Adam?

Plain SGD updates parameters using only the current gradient. Momentum adds a running velocity so updates build up speed in consistent directions and smooth out noise. Adam goes further by adapting a separate step size for each parameter based on running estimates of the gradient's mean and variance, which speeds up convergence on many problems.

Which optimizer should I use?

Adam or AdamW is a strong default for most deep learning, especially transformers and language models, because it converges quickly with little tuning. SGD with momentum often reaches better final accuracy on image classification when carefully tuned. A practical approach is to start with Adam and try tuned SGD if you need to squeeze out more generalization.

Is Adam always better than SGD?

No. Adam typically trains faster and is more forgiving about hyperparameters, but well-tuned SGD with momentum frequently generalizes better on vision tasks. The best choice depends on the model, the data, and how much tuning effort you can spend. Both remain widely used for good reason.

Do I still need to set a learning rate when using Adam?

Yes. Even though Adam adapts a per-parameter scale, it still relies on a global base learning rate that you must set. A common default works often but not always, and a poorly chosen value can still cause divergence or slow training. Learning-rate schedules and warmup are frequently combined with Adam.

What is momentum and why does it help?

Momentum accumulates a moving average of past gradients, giving the update a velocity that carries it forward. This helps the optimizer push through flat regions and dampen the back-and-forth oscillations that occur in narrow, steep valleys of the loss surface. The result is usually faster and more stable convergence than plain SGD.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.