Fifteen AI Crawlers Walked Into My Server. It Was One Machine.
One address claimed to be Amazon, OpenAI, Anthropic, Perplexity and seven other companies in a single day. Then it sent a DELETE request. Real crawlers do not do that.
The crawler name in your server log is a claim. It is not an identity.
I have known that in the abstract for years. This week I finally measured what it costs, and the number was worse than I expected.
Across my portfolio, over the 28 days ending 2026-09-25, declared AI crawlers sent 318,345 requests. Almost all of them were ordinary page reads. But 1,065 were something else: POST requests, and a handful of DELETE requests.
That small pile is where the interesting thing was hiding. Of those 1,065 requests, 902 came from just fifteen addresses. Each of those fifteen addresses claimed to be nine to eleven different companies' crawlers on the same single day.
One machine. Fifteen name tags.
There are four checks further down that will tell you whether the same thing is sitting in your own bot numbers. You can run all of them tonight on a plain access log. The first one takes about a minute.
I Went Looking to Confirm Something Else
On 2026-09-22, Help Net Security reported an Akamai finding that stuck with me. Verified AI crawlers, ChatGPT among them, have started sending POST requests instead of only reading pages. POST means doing something: a login, a cart addition, a checkout.
In Akamai's 30-day look at its customer traffic, ecommerce made up 44.8 percent of those AI bot POST transactions. Travel hit 30 percent in one month. Model Context Protocol traffic was 4.1 percent.
That is a genuinely big idea. A crawler that reads is an audience. A crawler that buys is a customer.
So I went to check my own logs, fully expecting to find a small version of the same curve.
What I actually found was 1,065 non-GET requests, and 902 of them were a lie.
What Fifteen Costumes Looks Like in a Log
Here is the shape of it. I grouped every non-GET request from a declared AI crawler by the address that sent it, then counted how many distinct vendors each address claimed on a single day.
Fifteen addresses came back wearing nine or more. The busiest one claimed eleven vendors and fifteen separate crawler names, and sent 206 requests to a single site.
Every one of those fifteen addresses sat in the 34.x and 35.x ranges. That is Google Cloud. Anybody can rent an address there for a few dollars.
And the names they wore were not obscure. Amazonbot. GPTBot. ClaudeBot. Claude-User. Claude-SearchBot. PerplexityBot. meta-externalagent. GrokBot. DeepSeekBot. OAI-SearchBot. CCBot. YouBot. Bytespider. cohere-ai. ChatGPT-User.
Fifteen companies do not coordinate a sweep of one person's websites on a Friday from the same cloud region. One actor rotating a list does.
The DELETE Request Is the Tell
If you only take one thing from this article, take this one.
Ninety-one of those requests used the DELETE method. DELETE asks a server to remove something.
No search crawler sends DELETE. No AI training crawler sends DELETE. No retrieval fetcher sends DELETE. There is nothing on a content site for them to delete, and deleting is not how any of them work.
So when a request says it is Amazonbot and it asks my server to delete something, I do not have a puzzle. I have an answer. Something is probing for an unprotected write endpoint while wearing Amazon's name so my logs look boring.
It worked, too, in a sense. It sat in my bot counts as legitimate AI crawler activity until I looked at the method column.
Why This Breaks Every Crawler Number You Read This Month
Two days after the Akamai piece, Help Net Security covered DataDome's State of Bot and Agent Security Report 2026. DataDome looked at trillions of requests across more than 75,000 customer sites over the twelve months to June 2026, and separately ran ten simulated bots at 21,491 popular homepages.
About 65.3 percent of those sites detected none of the ten. Only 2.4 percent stopped or challenged all of them.
Put the two reports side by side and the problem is obvious. One says AI bot behaviour is changing. The other says two thirds of sites cannot tell a bot from a person in the first place.
Both can be true. And that is exactly the trouble, because if your instrument is the user agent string, you cannot tell which one you are looking at.
I am not saying Akamai is wrong. Akamai sees ecommerce and travel traffic at a scale I never will, and my portfolio is content sites, so a shopping-cart trend has no reason to appear here. What I am saying is narrower and harder to argue with: on my own logs, the traffic that looked like the new behaviour was overwhelmingly one actor in costume, and nothing in the log itself would have told me that.
The Fix Became a Standard Three Weeks Ago
Here is the part that made this week worth writing about.
On 2026-09-01, the bot identity protocol became draft-ietf-webbotauth-httpsig-protocol-00, an adopted working-group Internet-Draft at the IETF, titled HTTP Message Signatures for automated traffic. It replaced the individual submission that came before it, which is what adoption looks like on the datatracker.
The idea is simple enough to explain in one breath. A crawler signs its request with a private key. It names where its public key lives using a Signature-Agent header. My server fetches that key from a well-known location, /.well-known/http-message-signatures-directory, and checks the signature itself.
Think of it like the difference between someone saying they work for the gas company and someone handing you a badge you can scan.
And this is not a proposal nobody has touched. Google's own crawling changelog records that it added Web Bot Auth documentation on 2026-05-04, and Cloudflare publishes bot verification documentation for it. Two of the biggest companies running crawlers were already there before the working group adopted it.
One caveat worth keeping. A valid signature proves who signed the request. It does not prove the crawler behaves well, respects your robots file, or has any right to your content. It answers who, not whether.
The Same Week, My Own Test Fell Over
I would rather tell you this part than have somebody find it.
Two weeks ago I wrote up a hypothesis on this site: that the mix of AI crawlers hitting each of my properties reflects a real platform preference, and that the mix would therefore hold steady across weekly measurements with nothing changed on my end. I gave it a falsification rule. If the top three crawlers on any property reshuffled between measurements, the idea was dead.
This run was measurement two of three. Four of seven properties reshuffled. Dead.
But the reason it died is more useful than the test was.
The three properties that held their order were RealSEOLife.com, SEO.Krisada.com and AIWebsiteSystems.com. On all three, the top crawler outran second place by at least 3.64 times. The four that reshuffled had their leaders sitting within 1.71 times of each other.
That is not a story about platform preference. That is arithmetic. When two numbers are close, small week-to-week noise flips their order. When one number is four times the other, nothing flips.
I had not built a test of preference. I had built a test of margin width, and it would have cheerfully agreed with me on any portfolio whose crawler mix happened to be lopsided.
And the Example I Built It On Was a Single Week
It gets one layer worse, and this is the layer I actually want people to learn from.
The headline example behind that whole hypothesis was Claude-User: 587 requests on AISymantix.com over 28 days, against a single request on RealSEOLife.com. Same server, same owner, similar subject matter. It looked like a clear preference for one property.
So I broke the same 28 days into weeks.
Of those 587 requests, 586 landed inside one week, 2026-08-22 to 2026-08-28. The other three weeks were 0, 1 and 0.
It was never a preference. It was one burst. And because my measurement window was a rolling 28 days, that burst showed up as a standing pattern for four weeks, then as a dramatic collapse to almost nothing the moment it rolled out the back of the window.
A rolling window will do that to you every time. It turns one event into a trend, holds the illusion for exactly as long as the window is wide, and then hands you a second fake story on the way out.
Four Checks You Can Run on Your Own Logs Tonight
None of these need a warehouse. A raw access log and a text editor will do.
- 1. Group your non-GET bot requests by address, and count how many different companies each address claims in one day. Three or more from one address is not three companies.
- 2. Look at the method column before you report anything. A crawler sending DELETE, PUT or PATCH is not a crawler.
- 3. Break any 28-day crawler figure into four weekly figures before you call it a pattern. If it all sits in one week, it is an event.
- 4. Check whether the address belongs to the company the name claims. A request saying Amazonbot from a rented cloud address is answering its own question.
Receipts Beat Names
I have been saying for a while that there is no debating receipts. This week the receipts came for one of my own ideas, which is the only real test of whether you meant it.
The larger point is not about spoofing. It is that for the whole history of the web, we have measured automated traffic by asking it to introduce itself, and then we built reports, dashboards, scores and strategies on top of the answer.
That was always a soft foundation. It is getting softer as AI clients multiply and as the value of pretending to be one goes up.
The standard that fixes it now exists, two of the largest crawl companies already support it, and it needs a key pair and one file at a known location. So that is what goes on my list next: publish the directory, capture the header, and start reporting verified separately from claimed.
Until then, I will keep saying claimed when I mean claimed.
And I will keep checking the method column.
See Where These Ideas Get Tested
The case studies and experiments are where the ideas in these articles get tested on live sites.