Stochastic Gradient Descent (SGD)
Definition
SGD updates a model based on randomly chosen examples or small groups of examples.
Stochastic gradient descent estimates the gradient of the objective function using randomly drawn examples or mini-batches and updates the model parameters accordingly. The common mini-batch SGD is a practical variant.
Simply explained
The model improves step by step on changing small portions of training data instead of evaluating the entire dataset each time.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A classifier updates its weights after each mini-batch and monitors loss as well as validation error.
Limits and pitfalls
- Not shuffling data sufficiently, coupling learning rate and batch size without checks, interpreting noisy training curves as errors, or mixing training and evaluation data.
Origin & history
Stochastic approximation was formalized in 1951; SGD became the fundamental method for large statistical and neural models.
Use in mid-sized companies
- Possible example: A classifier updates its weights after each mini-batch and monitors loss as well as validation error.
Frequently asked questions
What does Stochastic Gradient Descent (SGD) mean?
Stochastic gradient descent estimates the gradient of the objective function using randomly drawn examples or mini-batches and updates the model parameters accordingly. The common mini-batch SGD is a practical variant. What mistakes should be avoided? Not shuffling data sufficiently, coupling learning rate and batch size without checks, interpreting noisy training curves as errors, or mixing training and evaluation data.
Sources
- A Stochastic Approximation Method · Robbins and Monro, A Stochastic Approximation Method
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: