Safety Classifier
Definition
A safety classifier rates inputs to a language model and its responses as safe or unsafe based on safety risks.
A safety classifier is a separate model that checks inputs and outputs of a language model for defined safety risks. It can be used before and after the actual language model. What content it flags depends on the defined risk categories and its training.
Simply explained
Like a gate with a checklist that controls visitors and outgoing mail, regardless of who works inside the building.
Why it matters in practice
A safety classifier can additionally check inputs and outputs. Its categories must fit the use case; successful safety tests do not guarantee general protection.
Example from business
Hypothetical example: A travel provider has every chat input and every response of its assistant pre-checked by a safety classifier.
Limits and pitfalls
- Inappropriate categories cause false alarms or overlooked risks. Checking inputs only leaves outputs uncontrolled. Classifiers can misjudge, be vulnerable to prompt injection, and perform worse in other languages.
Origin & history
Inan and co-authors (GenAI at Meta) introduced Llama Guard in 2023 as a Llama2-7b model with a safety taxonomy and made the model weights available. Sharma and co-authors (Anthropic) described Constitutional Classifiers in 2025 as a protective measure against universal jailbreaks.
Use in mid-sized companies
- Possible applications: Check inputs and outputs in customer chats and internal assistants. Suitable output classifiers can abort generation as soon as they detect harmful content.
Frequently asked questions
Does a safety classifier replace model alignment?
No, it complements it as an additional layer of protection.
Does it detect every harmful input?
No, misjudgments and attacks like prompt injection remain possible.
Can categories be adapted?
Llama Guard supports task-specific categories, also via prompts with few or no examples.
Sources
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations · Inan et al. (2023): Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming · Sharma et al. (2025): Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming
- Llama Guard 3 8B, Model Card · Meta (o. J.): Llama Guard 3 8B, Model Card
Sources checked on 14.10.2026. Editorial application examples are hypothetical; no proof of real customer results.
Editorially reviewed on: