Tensor Parallelism
Definition
Tensor parallelism distributes computations within a model layer across multiple accelerators.
A form of distributed training that splits tensor operations of a single model layer across multiple accelerators.
Simply explained
Multiple computing devices share the work within the same model component instead of each computing the entire component alone.
Why it matters in practice
The term contributes to secure, maintainable, and high-performance web applications as well as resilient technical decisions.
Example from business
A medium-sized company distributes an oversized language model across four GPUs of a server.
Limits and pitfalls
- Communication and synchronization can negate the benefits; tensor, data, and pipeline parallelism must be distinguished.
Origin & history
Megatron-LM demonstrated practical intra-model parallelism for large transformers in 2019.
Use in mid-sized companies
- Possible application: A medium-sized company distributes an oversized language model across four GPUs of a server.
Frequently asked questions
What does Tensor Parallelism mean?
A form of distributed training that splits tensor operations of a single model layer across multiple accelerators. What should be watched out for? Communication and synchronization can negate the benefits; tensor, data, and pipeline parallelism must be distinguished.
Sources
- Megatron-LM · Shoeybi et al., Megatron-LM
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: