# robots.txt and AI agents: which sign-up pages it closes 15 of 151 products we drove close their sign-up page to agents in robots.txt. How the rules work, and how to keep sign-up open to agents. Source: https://agentdrives.com/standards/robots-txt-ai · Updated 2026-10-24 · as data: https://agentdrives.com/standards/robots-txt-ai.json · licence CC BY 4.0 robots.txt is the file where a site tells crawlers and agents which pages to stay away from. Some products use it to close their sign-up page to agents. ## The numbers From our drives of 151 products, 1–24 Oct 2026 (https://agentdrives.com/data/facts?via=md): - robots.txt closes the sign-up page: 15 of 151 (https://agentdrives.com/data/facts?via=md#who.robots_closed) - robots.txt kept our agent out of sign-up: 1 of 151 (https://agentdrives.com/data/facts?via=md#who.robots_stopped) ## What it is robots.txt is the Robots Exclusion Protocol, a standard since September 2022 as RFC 9309. [1] The file lives at `/robots.txt`, in lower case, at the top of the site. [1] It holds groups of rules, one per crawler name, matched without regard to case. [1] Within a group the most specific path wins. [1] The IETF is drafting a way to say what content may be used for, with the categories AI Training, AI Use, Search [2]. It can go in robots.txt as a `Content-Usage` line. [3] The text is a working draft. [2] Our agent keeps to robots.txt. [Our agent's page](https://agent.agentdrives.com/bot) says how. ## What we found 15 of 151 products we drove close their sign-up page to agents in robots.txt. At 1, that rule was what kept our agent out of sign-up. ## How to keep sign-up open to agents 1. **Read your file as an agent would.** Find the group that matches every crawler (`User-agent: *`) and any group for AI agents, and check whether it covers `/signup`, `/register` or your key page. 2. **Allow the pages an agent needs to start.** Add an `Allow` line for sign-up and login. The longer, more specific path wins over a broad `Disallow`. [1] 3. **Keep the file answering.** If robots.txt fails with a server error, a crawler must assume everything is closed. [1] ``` User-agent: * Disallow: /app/ Allow: /signup Allow: /login ``` ## Check it yourself 1. **Fetch it from a server.** Look for a `Disallow` that covers your sign-up path. ``` curl -s https://example.com/robots.txt ``` 2. **See it on your product's page.** Each product page here says whether robots.txt closes the sign-up page. ## Products whose robots.txt closes the sign-up page to agents - [Akiflow](https://agentdrives.com/tools/akiflow.md?via=md): L1 Needs a person - [Browse AI](https://agentdrives.com/tools/browse.md?via=md): L1 Needs a person - [ClickHouse Cloud](https://agentdrives.com/tools/clickhouse.md?via=md): L4 Runs on its own - [ClickUp](https://agentdrives.com/tools/clickup.md?via=md): L1 Needs a person - [Close](https://agentdrives.com/tools/close.md?via=md): L1 Needs a person - [Convex](https://agentdrives.com/tools/convex.md?via=md): L4 Runs on its own - [Figma](https://agentdrives.com/tools/figma.md?via=md): L1 Needs a person - [Help Scout](https://agentdrives.com/tools/helpscout.md?via=md): L1 Needs a person - [HubSpot](https://agentdrives.com/tools/hubspot.md?via=md): L1 Needs a person - [Mattermost](https://agentdrives.com/tools/mattermost.md?via=md): L3 Does the work - [Plain](https://agentdrives.com/tools/plain.md?via=md): L1 Needs a person - [PlanetScale](https://agentdrives.com/tools/planetscale.md?via=md): L2 Gets in - [SendGrid](https://agentdrives.com/tools/sendgrid.md?via=md): L1 Needs a person - [SignWell](https://agentdrives.com/tools/signwell.md?via=md): L4 Runs on its own - [Smartsheet](https://agentdrives.com/tools/smartsheet.md?via=md): L1 Needs a person ## Questions **Do agents have to follow robots.txt?** It is a request, not a lock. Our agent follows it. A site that wants agents out of a page should also check them at the edge. **Should I block AI crawlers?** That is your choice. Blocking training crawlers and letting agents sign up can go together: name each one in its own group. ## Sources 1. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.txt) 2. [IETF AI Preferences](https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt) 3. [IETF AI Preferences](https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.txt) ## More standards - [MCP](https://agentdrives.com/standards/mcp.md): A server that turns a product's actions into tools an agent can call. - [llms.txt](https://agentdrives.com/standards/llms-txt.md): A Markdown map of a product's docs, written for language models. - [OpenAPI](https://agentdrives.com/standards/openapi.md): A machine-readable description of every route of an API. - [Web Bot Auth](https://agentdrives.com/standards/web-bot-auth.md): An agent signs each request; a site's bot check can verify who sent it. - [Signed agents](https://agentdrives.com/standards/signed-agents.md): Agents a bot-check vendor has verified, which a site can let through. - [x402](https://agentdrives.com/standards/x402.md): A 402 answer an agent pays in a stablecoin, request by request. - [MPP](https://agentdrives.com/standards/mpp.md): A 402 payment challenge an agent pays by card or stablecoin.