What AI crawlers actually do on a site

Who actually came

One line of the log is one request. The counts below take one match per line, which matters more than it sounds: a crawler's User-Agent usually contains its own name twice, once as the name and once inside the contact URL, so counting occurrences instead of lines inflates every figure by roughly two. That mistake is easy to make and we made it ourselves on the first pass.

Finding one: llms.txt was requested zero times

Not rarely. Zero, out of 1,425 requests, over fifteen days, on a site where the file exists, is valid, is linked and is listed. This matches what a large study found across roughly 300,000 domains: no measurable link between having llms.txt and being cited by AI. Our log cannot prove that finding, one site never can, but it is a data point from the field rather than from the specification, and it points the same way.

This is not an argument for deleting the file. It costs nothing, it is human-readable, and some agents may read it when they choose to look. It is an argument against selling it, or buying it, as the thing that gets you cited.

Finding two: the crawler that feeds ChatGPT Search is a small minority

Strip out Meta, whose crawler feeds Meta and none of the assistants people actually ask, and 904 requests remain. Of those, OAI-SearchBot accounts for 52, or 5.8 percent. The rest is ClaudeBot and GPTBot, and GPTBot is OpenAI's training crawler, not its search crawler. OpenAI documents them as separate agents controlled separately in robots.txt.

The practical consequence is a common and expensive mistake: blocking GPTBot to opt out of training also feels like blocking ChatGPT, but it is not the crawler that decides whether ChatGPT Search can cite you. Those are two different doors, and most sites close both while meaning to close one.

Finding three: crawlers scout far more than they read

Of the 904 requests from assistant crawlers, 305, or 34 percent, went to just two files: robots.txt and sitemap.xml. The single most requested content page was the homepage, at 17 requests in fifteen days. Every other page sat at three or four. They walk the whole site, and they walk it shallowly.

That reframes what to optimise. If a third of crawler attention goes to robots.txt and the sitemap, then those two files are not housekeeping, they are the interface. A sitemap that omits a page, or a soft 404 that returns 200 for every wrong address, wastes the part of the budget the crawler actually spends.

Finding four: allowing a crawler does not summon it

PerplexityBot, Google-Extended, CCBot, Applebot-Extended and Bingbot never appeared once, although robots.txt explicitly allows all of them. Access is a necessary condition and not a sufficient one. A crawler has to learn the site exists, which happens through sitemaps, through indexes such as Bing's via IndexNow, and through being mentioned somewhere it already crawls.

Why this is a different job from SEO

The overlap between ranking in Google and being cited by an assistant is much thinner than it looks. Published analyses put 83 percent of citations in AI Overviews on pages outside the organic top ten, and 28.3 percent of the pages ChatGPT cites most are not visible in Google for the same query at all. Fewer than one in ten sources cited by ChatGPT, Gemini and Copilot sit in Google's top ten for that query.

Read in one direction that is discouraging: a good Google position does not carry over. Read in the other, it is the entire opportunity, because it means an assistant can cite you before you rank. Princeton-backed work puts the lift from GEO techniques at up to 40 percent, and the single technique with the most directly measured effect is citing your sources inside your own claims, at up to 115 percent for pages sitting fifth in search.

What is actually measurable, and how

The awkward part of this field is that the most valuable visibility produces no visit at all. The assistant paraphrases the answer, the person reads it, and arrives a week later typing your name directly. Analytics will file that as direct traffic. So one number is never enough; measure in three independent ways.

• Server logs. Which AI crawlers come, how often, and which pages they take. This is the only fully objective signal, it needs no tools, and it is where every number in this article came from. Watch the share of content pages and the share of the search crawler, not the total.
• Control prompts. A fixed list of questions asked in a fresh session across several assistants, on a fixed weekly schedule, written into a log. Record not only whether you were named but who was named instead: that list of competitors cannot be obtained any other way.
• The form. One field asking how the person heard about you, with an explicit option for an AI assistant, recorded alongside the technical referrer. Those two disagree precisely in the interesting case, and the disagreement is the measurement.

What this article deliberately does not contain: results. These are baseline numbers, taken before the changes they will be compared against had time to take effect. Publishing an effect we have not measured yet would be exactly the dishonesty this piece argues against. The follow-up measurement is due in a month, on the same site, by the same method.