AgentDrives

Content Signals and AI rules in SaaS robots.txt

42 of the 200 SaaS products we checked (21%) tell AI what it may do with their content in robots.txt, all with Cloudflare's Content-Signal line. None uses the IETF's Content-Usage line.

Drives 1 Oct to 1 Nov 2026200 productsStep FindSubject robots.txtMarkdown

What SaaS robots.txt files tell AI. A Content-Signal line: 42 of 200, Says ai-train=no: 12 of 200, Says ai-input=no: 0 of 200, A Content-Usage line: 0 of 200.

Share this card: Wide PNG · Tall PNG

Two lines for AI preferences

Content-Signal
Cloudflare's line. Each label is yes or no.
  • search building a search index
  • ai-input feeding content into AI models for real-time answers
  • ai-train training or fine-tuning AI models
Content-Usage
The IETF's draft, vocab-08. Labels train-ai, ai-use, search, each y or n.
Cloudflare's managed robots.txt writes
Content-signal: search=yes, ai-train=no, use=reference

What we found.

12 of the 42 Content-Signal lines say no to AI training (ai-train=no), and 30 say yes. 0 of the 42 say no to using the content in AI answers (ai-input=no). All 42 say yes to search. At 20 of 201 products, robots.txt closes the sign-up page to AI agents.

LabelYesNoNot said
search4200
ai-input4002
ai-train30120

For agent builders.

  • Read Content-Signal and Content-Usage lines when you read robots.txt. They sit beside Allow and Disallow, not in place of them.
  • Allow and Disallow still decide what you may fetch. The preference lines say what the owner wants done with what you fetched.
  • ai-input is the label for an agent that reads a page to answer a question. Check it before you quote or summarise a page.
  • The two lines use different labels: ai-train and ai-input in Content-Signal, train-ai and ai-use in Content-Usage. Don't mix them.

Method and limits.

How we tested

  • Drives from 1 Oct to 1 Nov 2026, 201 products.
  • At 200 of them, the drive read the robots.txt at the product's own domain.
  • A product counts when a Content-Signal or Content-Usage line appears in that file.

Limits

  • Each product as it was on the day of its newest drive.
  • Only the product's own domain. A docs or app host can carry a different robots.txt.
  • We read the start of each file, so a line far down a long file can be missed.

Cite this

AgentDrives, “Content Signals and AI rules in SaaS robots.txt”, 1 November 2026, https://agentdrives.com/findings/content-signals.
@misc{agentdrives_content_signals_2026,
  author = {{AgentDrives}},
  title = {Content Signals and AI rules in SaaS robots.txt},
  year = {2026},
  month = nov,
  url = {https://agentdrives.com/findings/content-signals},
  note = {Agent test drive, CC BY 4.0},
}
CC BY 4.0, with credit

See what an agent sees in yours.

Free: what an agent finds, any wall in its way and how many things to fix. The Audit ($499) gives you every fix.