< X-Robots-Tag: noindex, nofollow

TL;DR: X-Robots-Tag: noindex tells search engines not to list a URL, and it works on PDFs, images and other files where a meta tag is impossible. The crawler has to be allowed to fetch the URL to see it, so never combine it with a robots.txt Disallow on the same path.

Syntax

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow

The value is a comma-separated list of rules. You can send several headers, and you can scope a rule to one crawler by prefixing its user agent token:

X-Robots-Tag: googlebot: noindex
X-Robots-Tag: otherbot: noindex, nofollow

Rules without a user agent apply to every crawler that honours the header. Google documents that the header name, user agent names and values are not case sensitive. When rules conflict, the more restrictive one wins.

Directives Google supports

Per Google Search Central’s robots meta tag specification:

Rule	Effect
all	No restrictions. This is the default.
noindex	Do not show the page, media or resource in search results.
nofollow	Do not follow links on the page.
none	Shorthand for noindex, nofollow.
nosnippet	Show no text snippet or video preview.
indexifembedded	Allow indexing of the content when it is embedded in another page through an iframe or similar, even though the page carries noindex. It only takes effect together with noindex.
max-snippet: [number]	Limit the text snippet to that many characters. 0 means none, -1 means no limit.
max-image-preview: [setting]	none, standard or large.
max-video-preview: [number]	Limit video previews to that many seconds. 0 allows a static image only, -1 is unlimited.
notranslate	Do not offer translation of the page in results.
noimageindex	Do not index images on the page.
unavailable_after: [date]	Stop showing the page after the date. RFC 822, RFC 850 and ISO 8601 formats are accepted.

Other search engines document their own sets. Check each vendor before relying on a directive outside this list.

When to use it

• Files with no HTML head. PDFs, Word and Excel downloads, images, and video. A meta tag cannot exist in them.
• Staging, preview and internal hosts. Send noindex, nofollow for every response from the host. It is quicker to apply than editing templates.
• Pagination, filter and search-result URLs that you want crawled for links but not indexed: noindex, follow.
• Expiring content. unavailable_after: 31 Dec 2027 23:59:59 GMT for a promotion page.

If the content is confidential, use authentication. noindex is a request to crawlers, not access control.

Configuration

nginx

# Every PDF and Word document on the site
location ~* \.(pdf|docx?|xlsx?)$ {
    add_header X-Robots-Tag "noindex, nofollow" always;
    try_files $uri =404;
}

# Whole staging host
server {
    server_name staging.example.com;
    add_header X-Robots-Tag "noindex, nofollow" always;
}

Two nginx behaviours cause most “it works on one URL and not another” bugs. Without always, add_header only applies to 200, 201, 204, 206 and the 3xx codes, so a 404 or 500 page goes out without it. And add_header inherits from the enclosing level only if the current level has no add_header of its own: put a Cache-Control add_header in a location and the server-level X-Robots-Tag disappears for that location. nginx 1.29.3 added add_header_inherit merge to change that. On older versions, repeat the header in each block.

Apache

<FilesMatch "\.(pdf|docx?|xlsx?)$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

This is Google’s own example pattern for PDFs and requires mod_headers. Use Header always set if error responses need it too.

Cloudflare

For a site proxied through Cloudflare, add a Response Header Transform Rule (Rules, Transform Rules, Modify Response Header): set a static header named X-Robots-Tag with value noindex, nofollow, matched by a custom filter expression such as:

ends_with(http.request.uri.path, ".pdf") or http.host eq "staging.example.com"

For Cloudflare Pages static assets, use a _headers file in the build output. Cloudflare documents X-Robots-Tag: noindex for preview deployments as a use case:

https://:project.pages.dev/*
  X-Robots-Tag: noindex

_headers rules do not apply to responses generated by Pages Functions. Set the header in the Function code for those.

Express

app.use('/downloads', (req, res, next) => {
  res.set('X-Robots-Tag', 'noindex, nofollow')
  next()
})

The robots.txt trap

robots.txt controls crawling. X-Robots-Tag is read from the response of a crawl. If a URL is disallowed, the crawler never makes the request, so it never sees the header. Google’s documentation says it plainly: if the page is blocked by robots.txt the crawler never sees the noindex rule and the page can still appear in search results. Google also does not support a noindex line inside robots.txt.

So the sequence for removing something is:

1. Remove the Disallow rule for that path.
2. Serve X-Robots-Tag: noindex on the URL.
3. Request a recrawl in Search Console’s URL Inspection tool, or wait.
4. Only after it has dropped out, if you want to stop crawling, add the Disallow back.

Verify

curl -sI https://example.com/files/report.pdf | grep -i x-robots-tag

Run it against the canonical, final URL. Then run it against a 404 path on the same host to see whether the header survives error responses. Search Console’s URL Inspection shows whether Google saw noindex on its last fetch.

Related

• Content-Type, the header that tells crawlers a URL is a PDF or an image in the first place
• Cache-Control: crawlers recrawl on their own schedule, but a long max-age at your CDN can keep serving an old header