robots.txt: where agents may go.

What it is

robots.txt is the Robots Exclusion Protocol, a standard since September 2022 as RFC 9309.¹ The file lives at /robots.txt, in lower case, at the top of the site.¹ It holds groups of rules, one per crawler name, matched without regard to case.¹ Within a group the most specific path wins.¹

The IETF is drafting a way to say what content may be used for, with the categories AI Training, AI Use, Search². It can go in robots.txt as a Content-Usage line.³ The text is a working draft.²

Our agent keeps to robots.txt. Our agent's page says how.

What we found

15 of 151 products we drove close their sign-up page to agents in robots.txt. At 1, that rule was what kept our agent out of sign-up.

How to keep sign-up open to agents

1. 1

   Read your file as an agent would.

   Find the group that matches every crawler (User-agent: *) and any group for AI agents, and check whether it covers /signup, /register or your key page.

2. 2

   Allow the pages an agent needs to start.

   Add an Allow line for sign-up and login. The longer, more specific path wins over a broad Disallow.¹

3. 3

   Keep the file answering.

   If robots.txt fails with a server error, a crawler must assume everything is closed.¹

User-agent: *
Disallow: /app/
Allow: /signup
Allow: /login

Check it yourself

1. Fetch it from a server.

   Look for a Disallow that covers your sign-up path.

   curl -s https://example.com/robots.txt

2. See it on your product's page.

   Each product page here says whether robots.txt closes the sign-up page.