# Content Signals and AI rules in SaaS robots.txt

42 of the 200 SaaS products we checked (21%) tell AI what it may do with their content in robots.txt, all with Cloudflare's Content-Signal line. None uses the IETF's Content-Usage line.

Drives: 1 Oct to 1 Nov 2026 · 200 products · Step: [Find](https://agentdrives.com/access/find.md) · Subject: robots.txt · Counted 1 Nov 2026. Source: https://agentdrives.com/findings/content-signals?via=md · CC BY 4.0

## Two lines for AI preferences

- `Content-Signal`: Cloudflare's line. Each label is yes or no.
  - `search`: building a search index
  - `ai-input`: feeding content into AI models for real-time answers
  - `ai-train`: training or fine-tuning AI models
- `Content-Usage`: The IETF's draft, vocab-08. Labels train-ai, ai-use, search, each y or n.

Cloudflare's managed robots.txt writes:

```
Content-signal: search=yes, ai-train=no, use=reference
```

## What we found

| Label | Yes | No | Not said |
|---|---|---|---|
| search | 42 | 0 | 0 |
| ai-input | 40 | 0 | 2 |
| ai-train | 30 | 12 | 0 |

- 12 of the 42 Content-Signal lines say no to AI training (ai-train=no), and 30 say yes.
- 0 of the 42 say no to using the content in AI answers (ai-input=no).
- All 42 say yes to search.
- At 20 of 201 products, robots.txt closes the sign-up page to AI agents.

## For agent builders

- Read Content-Signal and Content-Usage lines when you read robots.txt. They sit beside Allow and Disallow, not in place of them.
- Allow and Disallow still decide what you may fetch. The preference lines say what the owner wants done with what you fetched.
- ai-input is the label for an agent that reads a page to answer a question. Check it before you quote or summarise a page.
- The two lines use different labels: ai-train and ai-input in Content-Signal, train-ai and ai-use in Content-Usage. Don't mix them.

## How we tested

- Drives from 1 Oct to 1 Nov 2026, 201 products.
- At 200 of them, the drive read the robots.txt at the product's own domain.
- A product counts when a Content-Signal or Content-Usage line appears in that file.

## Limits

- Each product as it was on the day of its newest drive.
- Only the product's own domain. A docs or app host can carry a different robots.txt.
- We read the start of each file, so a line far down a long file can be missed.

## Sources

- [Every number on this page, with its definition](https://agentdrives.com/data/facts.md?via=md#signal.read)
- [robots.txt for AI agents](https://agentdrives.com/standards/robots-txt-ai.md)
- [Cloudflare: Content Signals](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/index.md)
- [IETF: AI preferences vocabulary, draft 08](https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt)
- [How we drive](https://agentdrives.com/method.md?via=md)