X-Robots-Tag
Definition
X-Robots-Tag transmits instructions to search engines on how to handle webpages and other resources.
An HTTP response header that transmits instructions like noindex or noarchive to search engines for arbitrary resources.
Simply explained
The server provides search engines with hints, for example that a file should not appear in search results.
Why it matters in practice
The term is relevant for systematically improving digital visibility, qualified traffic, and content findability.
Example from business
An operator marks generated PDF duplicates as noindex but keeps them accessible to authorized crawlers.
Limits and pitfalls
- robots.txt prevents retrieval and thus reading of the header; search engines treat directives as guidelines, not access control.
Origin & history
Search providers introduced the header as the HTTP equivalent to robots meta tags.
Use in mid-sized companies
- Possible application: A company marks generated PDF duplicates as noindex but keeps them accessible to authorized crawlers.
Frequently asked questions
What does X-Robots-Tag mean?
An HTTP response header that conveys instructions such as noindex or noarchive to search engines for arbitrary resources. What should be considered? robots.txt prevents fetching and thus reading the header; search engines treat directives as guidelines, not access control.
Sources
- Robots meta tag and X-Robots-Tag · Google Search Central, Robots meta tag and X-Robots-Tag
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: