We Answered 14 Million AI Crawler Requests and Never Once Said Slow Down
14,396,236 AI crawler requests in ten days. Zero 429 responses. The new bot trust scores grade whether a crawler slows down, and we never once asked one to.
Between October 14 and October 23 our servers answered 14,396,236 requests from AI crawlers.
We said 200 OK to 14,322,524 of them.
The number of times we told a crawler to slow down was zero. Not a small number. Zero. There is no HTTP 429 anywhere in that window.
I only went looking because of two things that happened this week. On October 16 Cloudflare started grading approved bots on whether they obey robots.txt and whether they ease off after a 429. On October 22 Gary Illyes of Google filed an IETF draft that writes the same conduct down as crawler best practice.
Both of those grade a conversation.
I checked our side of the conversation and found it empty.
What Cloudflare Started Grading on October 16
Cloudflare already runs a Verified bots list. Getting on it was a one-time check of who you are.
The post published on October 16 adds what happens after. Cloudflare calls it "a system to evaluate and report when Verified bot behavior starts to deviate from this rule."
It grades two things.
First, robots.txt. Cloudflare looks at "whether a bot's observed traffic is consistent with those published preferences over time," and says it watches for repeated patterns rather than one-off slips.
Second, the 429. When a server answers Too Many Requests, Cloudflare asks "whether the bot's traffic adapts after receiving that signal. Does it slow down, pause, or does it continue sending requests?"
Right now the grading is internal. Cloudflare says the scores will go public on Radar later, and that customers will eventually pick their own trust threshold for how Verified bot traffic gets handled.
So the grade is not live for site owners yet. The data it will be built from is being collected now, from servers like ours.
Gary Illyes Wrote the Crawler Rulebook Down, and Anthropic Filed a Rival Format the Same Day
Three drafts landed in the IETF Web Bot Auth working group on October 22. All three are about proving who a bot is and how it should behave.
The first is Crawler best practices, by Gary Illyes of Google, Mirja Kuhlewind of Ericsson and AJ Kohn. Its abstract is one sentence long. Its list is six items.
- Obey the robots exclusion protocol, including the X-Robots-Tag header
- Identify yourself with a user agent that links to your own documentation
- Do not disrupt the server, which means rate limiting and back-out logic
- Honour HTTP caching directives so you stop re-downloading the same page
- Publish your crawling IP ranges in a standard JSON format
- Document what you do with the data, how to block you, and how to reach you
The draft is careful to say these are voluntary and already common.
The other two drafts are the interesting part, because they do the same job twice.
Illyes also filed JAFAR, a JSON format for the company running a bot to publish the addresses it crawls from. Kevin Kelley of Anthropic filed SAIL, which does the same thing in TOML at a well-known address, with optional promises that a service sends from nowhere else and that nobody else sends from those addresses.
Same working group. Same day. Same problem. Two formats.
That is worth watching rather than acting on. Nobody should build against either file until one of them wins.
Our Own 14.4 Million Requests, Sorted by What We Answered
Here is the whole portfolio for those ten days, every request our log ingestion classified as an AI crawler.
- 200 OK: 14,322,524
- 301 moved: 30,147
- 404 not found: 24,975
- 403 forbidden: 11,090
- 206 partial content: 2,763
- 401 unauthorised: 2,523
- 500 server error: 2,056
- 304 not modified: 109
- 503 unavailable: 27
- 429 too many requests: 0
99.49 percent of it is a flat yes.
The 304 line is the one that stopped me. A 304 means a crawler asked whether a page had changed since last time, and we told it no, do not bother downloading it again. Out of 14.4 million requests that happened 109 times.
Why Our Own Pages Tell Crawlers Not to Cache Them
I fetched the headers to find out why.
The home page of this site answers with Cache-Control: no-store, no-cache, must-revalidate and an expiry date in November 1981.
That is not a decision anybody made. Line 8 of our front controller calls session_start(), and PHP sends those headers along with a session. Every article, every glossary term and every case study on RealSEOLife.com has been telling crawlers to never keep a copy and to ask again every single time.
The guide sites are different and no better. over50.bellyupcharleston.com and www.aisymantix.com send no caching information at all. No Cache-Control, no ETag, no Last-Modified. A crawler has nothing to ask about, so it downloads the page again.
One part of the setup does it properly. https://www.realseolife.com/ai/catalog.json carries a real Last-Modified and a one hour max-age, because Apache serves that file directly and nothing starts a session in front of it.
So the only cacheable part of our sites is the part we built for AI to read, and we got that right by accident.
The ClaudeBot Burst We Can Now Predict
The reason the 429 matters here is that we already have one crawler behaviour we can see coming.
Charleston went live with six guide sites. On October 6 ClaudeBot made exactly two requests to each one and left.
On October 14 it came back and walked five of the six in a single day. The over-50 guide alone took 92,352 filter URL requests and 674 mebibytes.
Then it stopped. Zero filter requests on every one of the next eight days, with a steady 4 to 26 ordinary requests a day.
Anthropic's own crawler page says it aims "for minimal disruption by being thoughtful about how quickly we crawl the same domains and respecting Crawl-delay where appropriate." The robots.txt on that Charleston site is three lines. User-agent star, Allow slash, and a sitemap address. There is no Crawl-delay in it, so there was nothing to respect.
The thirty original guide sites across Jacksonville, Orlando, St Augustine, Charlotte and Tampa have had zero ClaudeBot filter requests since October 6. Eighteen days, not one.
Last week I wrote that this looked like a crawler enumerating a site on arrival. That was wrong in one detail, and the detail matters. Charleston arrived on October 6 and was enumerated on October 14. The trigger is not the first request. It is the first deep crawl, whenever that happens to land.
And the size of that one day is not random. It tracks the number of filter links we publish on the page.
Every Address It Asked For Was One We Published
This was the question I actually needed answered, because it decides what to build.
If a crawler is making up filter combinations, taking the links off the page changes nothing and only a server rule will help. If it is following links, taking the links off the page ends it.
On that single day the over-50 guide received 92,405 ClaudeBot requests at 91,219 different addresses. So it was exploring, not re-asking for the same URLs.
I pulled 4,000 of those addresses and broke them into their 131 separate filter settings. Then I checked each one against the page as it actually renders, the home page plus one filtered view.
All 131 were there. Not one invented value.
The site has six filter sets behind those links.
- 6 dietary options
- 34 discovery tags
- 52 experience tags
- 27 neighborhoods
- 16 venue types
- 2 sort orders
Multiply those out and the full combination space is 18,542,580 addresses. ClaudeBot touched 0.49 percent of it, and every address it asked for was one we handed it.
So the links are the cause. That is a much cheaper problem than the alternative.
Three Changes We Are Making
Three changes
- 1. Send a cache signal. Every page that renders through PHP needs a real
Last-ModifiedorETag, which means getting the session headers off pages that do not need a session. A crawler cannot be graded on honouring a cache directive we never send. - 2. Answer the filter flood with a 429 and a
Retry-Afterinstead of a 200. One site first, siblings untouched, so we can read whether the request rate actually falls and which vendors ease off fastest. - 3. Move the filter grid off separate addresses. The link check says that works, and the filter link counts give us a number to predict against: cut the published links and the one-day burst should shrink with them.
The Scorecard Needs Two Players
The week reads like a rulebook arriving. Cloudflare grading conduct. An IETF draft listing the conduct. Two rival formats for proving who a bot is. Google restating that robots.txt controls whether a crawler may fetch a page and says nothing about what happens to the page afterwards.
All of it points the same way. Identity and conduct for crawlers are being settled outside your website, by networks and working groups, and the raw material will be server logs.
Which makes the quiet part of this week's numbers the useful part. A 429 that was never sent. A Last-Modified that was never written. 109 polite exchanges out of 14.4 million.
We have spent a year reading what the crawlers do to us. It turns out we never said much back.
See AI crawler rate limits Tested on Live Websites
Follow the related case studies and experiments to see how these ideas work on live sites.