Gradient Centralization
Definition
Gradient centering modifies the gradients computed during model training by subtracting their mean.
Gradient centering is an optimization method for neural networks. It acts directly on the gradients, i.e., the computed directions of change for model weights. The gradient vectors are centered so that their mean is zero.
Simply explained
Like a group of signposts whose common shift you remove to examine their differences. The analogy explains only the centering; a training gradient is not a true path description.
Why it matters in practice
For own training experiments, it is relevant at which point an optimization intervenes. The paper explicitly distinguishes processing gradients from methods on activations or weights.
Example from business
Possible example: A research team compares the same optimization with and without gradient centering when training an image classifier.
Limits and pitfalls
- Gradient centering is not merely a renaming of batch normalization or weight standardization. The cited paper targets a different quantity. Its experimental results are not a general guarantee of improvement for every network.
Origin & history
Yong, Huang, Hua, and Zhang submitted the cited initial version on April 3, 2020.
Use in mid-sized companies
- Possible applications include controlled training comparisons and experiments to adapt already trained neural networks.
Frequently asked questions
Are the input data centered directly in this process?
The described method operates on the gradients.
Can it be integrated into existing optimization methods?
The work describes integration into gradient-based optimizers and experiments for fine-tuning.
Sources
- Gradient Centralization: A New Optimization Technique for Deep Neural Networks · Yong et al. (2020): Gradient Centralization: A New Optimization Technique for Deep Neural Networks
Editorially reviewed on: