On-Device Inference
Definition
A trained model is executed directly on the device.
The prediction calculation takes place on a smartphone, PC, or embedded device. Training and supplementary services can still occur elsewhere.
Simply explained
The device answers the task with its own AI engine.
Why it matters in practice
This can reduce network dependency and data transmission.
Example from business
Hypothetical example: A device recognizes a wake word without transmitting any audio recording.
Limits and pitfalls
- Memory, power consumption, and model size limit the possibilities.
Origin & history
The approach belongs to edge computing and embedded AI.
Use in mid-sized companies
- SMEs can consider local processing for mobile or offline-capable applications.
Frequently asked questions
Is On-Device automatically fully offline?
Only if all required additional functions are also available locally.
Sources
- LiteRT — · Google AI Edge: LiteRT —
Editorially reviewed on: