Clicked Gallery

What is Stochastic Gradient Descent (SGD)?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

Engineering Notes · AI Systems

To optimize the training loop for large datasets, the ML engine relies on stochastic gradient descent.

The reader highlighted one word in the docs. Clicked explained the technical term “stochastic gradient descent” in simple terms:

Explained in three depths

Same facts, different vibe — Slang mode 😎

The Clicked way

●○○

Overview

SGD is the method most AI models use to learn: make a prediction, measure how wrong it was, nudge every internal setting a tiny step in the direction that reduces the error, and repeat millions of times. The stochastic part is that each step checks a small random batch of examples instead of the entire dataset.
●○○

Overview

SGD is how models learn: guess, get told how wrong, adjust a tiny bit, repeat a few million times. The stochastic part is the honest bit — we check a random handful of examples per step, because checking everything every time would take a lifetime. 😎

A quick take — often all you need.

●●○

Detail

A model is a machine with millions of tiny dials, and its performance is one wrongness score called the loss. For each dial, math can work out which way to turn it so the score gets smaller, and that direction is called the gradient. Descent means turning every dial a little that way, measuring the score again, and repeating millions of times. The choice SGD makes is how many examples to check before each turn. Checking all of them, which for a large model can mean millions of images, points in the most accurate direction but is so slow that few turns happen per hour, while a batch of 32 or 64 random examples points in a slightly wrong direction and allows thousands of turns in the same time. Thousands of slightly wrong turns lower the score faster than a handful of perfect ones. Because each batch differs, the direction also wobbles, and that wobble can shake the model off a setting that looked good but wasn't the best available.
●●○

Detail

The pieces: loss is the wrongness score, and the gradient is an arrow pointing toward less wrong for every knob in the model. The learning rate is the step size — too big and you faceplant past the target, too small and training takes a geological era. Mini-batches make each step cheap and slightly drunk, and the drunkenness is weirdly useful: a perfectly sober walker settles into the first comfortable rut, while the wobbly one stumbles out and finds something better. Why random batches win: a rough direction you can compute a thousand times an hour beats a perfect direction you compute twice. That's the whole trick: speed over precision, repeated until the score stops falling. And the wobble is not a bug you tolerate, it's the part that keeps the model from settling for the first mediocre answer it finds. 😎

Want more? One click digs deeper.

●●●

Analogy

Walking down a mountain in thick fog. You can't see the valley, so you feel the slope under your feet and step downhill, over and over — surveying the entire mountain before each step would give a perfect direction and take forever. SGD is reading only the ground under your boots: rougher readings, far more steps, and the occasional stumble that knocks you out of a small dip that wasn't the real bottom.
Deepest level reached.
●●●

Analogy

It's fixing your free throw. Shoot, watch the miss, adjust a little, shoot again — you don't rewatch every shot you've ever taken before each tweak, because the last few misses tell you which way to lean. Occasionally a lucky bounce lies to you and you adjust the wrong way, and a thousand shots later the form is dialed in anyway.
Deepest level reached.

Unfamiliar concept? A real-world example makes it click — fresh analogies on tap.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

Stochastic gradient descent is an iterative optimization algorithm that updates model parameters in the direction of the negative gradient of the loss function, where the gradient is estimated from a randomly sampled mini-batch rather than the full dataset. This yields computationally inexpensive, unbiased but noisy updates; the induced noise can aid escape from poor local minima and saddle points. Convergence behavior is governed principally by the learning-rate schedule, with momentum-based and adaptive variants (e.g., Adam) widely used in practice.

Want Clicked to explain terms like “stochastic gradient descent” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required