Inference as a Service
Definition
Inference as a service provides execution of trained models via a service.
Inference as a service here means retrieving predictions of a trained model via a separate service interface. The application calls the prediction service instead of embedding model execution directly into its own workflow.
Simply explained
Like a request to a specialist unit: The application sends inputs and receives results back. Here, however, the specialist unit is software, not an independently judging person.
Why it matters in practice
In an architectural decision, the service interface and the requirements for response time and load should be explicitly described.
Example from business
Possible example: An internal application sends approved features to a prediction service and displays the result for professional evaluation.
Limits and pitfalls
- A prediction service is not automatically a public paid offering. The cited Clipper architecture demonstrates the technical service separation, not specific tariffs or contractual services.
Origin & history
The cited 2017 Clipper version describes a modular prediction service between applications and various model frameworks.
Use in mid-sized companies
- Possible fields of application are internal search and classification applications that call a jointly operated prediction service.
Frequently asked questions
Is the model trained on every call?
The described service provides model predictions, not training as a query result.
Must the service be hosted by an external provider?
The technical service separation does not specify a particular operator.
Sources
- Clipper: A Low-Latency Online Prediction Serving System · Crankshaw et al. (2017): Clipper: A Low-Latency Online Prediction Serving System
Editorially reviewed on: