Layer Dropping
Definition
Layer Dropping disables or removes entire model layers to influence robustness, training, or inference costs.
LayerDrop trains transformers with structured dropout on entire layers so that subnetworks of reduced depth can be used during inference.
Simply explained
Like occasionally skipping entire processing stages during training so the network also works with fewer stages.
Why it matters in practice
Relevant for model compression and adaptive inference, but not a standard procedure for every model.
Example from business
A research team tests how many transformer layers can be removed before quality limits are undershot.
Limits and pitfalls
- Delete arbitrary layers without evaluation; equate LayerDrop with normal neuron dropout.
Origin & history
Fan, Grave, and Joulin published LayerDrop in 2019/2020 as structured dropout for transformers.
Use in mid-sized companies
- Possible application in SMEs: Reduce model size or investigate inference costs when controlled quality tests are available.
Frequently asked questions
Is LayerDrop normal dropout?
No, entire layers are treated in a structured way.
Sources
- LayerDrop · Fan et al. (2019), LayerDrop
Sources checked on 10.10.2026.
Editorially reviewed on: