# robots.txt for AI agents

The file at a site's root that tells crawlers which pages to stay off. Some sites add rules that name AI agents.

Also called robots.txt.

Source: https://agentdrives.com/glossary/robots-txt · as data: https://agentdrives.com/glossary/robots-txt.json · licence CC BY 4.0

## In our drives

Our agent reads a site's robots.txt before it opens a page and stays off a page the rules close to it. A sign-up page closed this way is shown as such on the product's page.

## Products whose robots.txt asks crawlers to stay off the sign-up page

14 of the 151 products we drove.

- [ClickHouse Cloud](https://agentdrives.com/tools/clickhouse.md?via=md): L4 Runs on its own
- [Convex](https://agentdrives.com/tools/convex.md?via=md): L4 Runs on its own
- [SignWell](https://agentdrives.com/tools/signwell.md?via=md): L4 Runs on its own
- [Mattermost](https://agentdrives.com/tools/mattermost.md?via=md): L3 Does the work
- [PlanetScale](https://agentdrives.com/tools/planetscale.md?via=md): L2 Gets in
- [Akiflow](https://agentdrives.com/tools/akiflow.md?via=md): L1 Needs a person
- [Browse AI](https://agentdrives.com/tools/browse.md?via=md): L1 Needs a person
- [ClickUp](https://agentdrives.com/tools/clickup.md?via=md): L1 Needs a person

## Related terms

- [Wall](https://agentdrives.com/glossary/wall.md): Anything that stops an AI agent at a product's door: a bot check, a phone number at sign-up, a card before first use, a sign-up that only works in a browser.
- [Signed agents](https://agentdrives.com/glossary/signed-agents.md): AI agents that sign each request with a key the site can check, so the site knows who sent it. A site can let signed agents through a bot check that stops everyone else.

Every term: https://agentdrives.com/glossary.md