Field Note
What a Crawler Claims Versus What It Can Prove
Before treating a traffic spike as interest, confirm the visitor is who it says it is. Two of ours were lying, and three honest ones could not prove anything.
The setup.
A crawler user agent is a claim, not an identity. Anything on the internet can send the string GPTBot.
Several AI vendors publish the IP ranges their crawlers operate from. That turns the claim into something checkable.
So we ran every crawler hitting the BellyUp ATL network against those published lists.
The finding.
The headline number held up. 46,727 of 46,829 GPTBot requests came from addresses inside the published OpenAI range list, spanning 41 of 43 addresses.
The more useful result was the middle of the table. Three vendors returned zero verified requests, and that proves nothing at all.
Meta, Amazon and Anthropic do not publish crawler IP ranges. Zero verified means there is no list to check against. It does not mean the traffic was fake, and network ownership backed all three up.
The actual impostors looked completely different. Two cloud addresses were cycling through crawler identities.
One claimed 14 separate vendors in a single day, among them Anthropic, OpenAI, Meta, Baidu and Cohere. It spent that day requesting WordPress login forms and environment files, and 219 of its 228 requests returned 404.
Neither address appeared in any publisher range.
| Crawler | Requests | IPs | Requests inside published range | Identity evidence |
|---|---|---|---|---|
| GPTBot | 46,829 | 43 | 46,727 | Publisher IP list |
| OAI-SearchBot | 357 | 55 | 249 | Publisher IP list |
| Googlebot | 212 | 61 | 192 | Publisher IP list |
| meta-externalagent | 2,678 | 71 | 0 | No published list. Network AS32934 Facebook |
| Amazonbot | 1,533 | 400 | 0 | No published list. Network AS14618 Amazon |
| ClaudeBot | 241 | 9 | 0 | No published list. Network AS16509 AWS |
| Date | Vendor identities claimed in one day | Requests | Requests returning 404 |
|---|---|---|---|
| September 6 | 14 | 228 | 219 |
| August 21 | 5 | 1,104 | 1,052 |
The change.
Nothing needed fixing, which is the point of checking.
Both impostor addresses were already classified as scanners and already excluded from the crawler totals. The verification pass confirmed the classification rather than correcting it.
What changed was the reporting. Roughly 1,300 of about 75,000 requests were impostors, so the surge was genuine and the real problem lived somewhere else entirely.
Whether the three unverifiable vendors ever publish address ranges. Until they do, their traffic carries an honest asterisk instead of a verification badge, and the totals say so out loud. Revisited whenever the published range lists refresh.
More real work, shown
Every field note is one finding from a live website, the data behind it, and the metric we are watching next.