Field Note
An AI Crawler Read 39,000 Pages of Nothing on Our Site
A verified OpenAI crawler pulled 43,309 requests from one restaurant network in a single evening. Almost none of it was content.
The setup.
BellyUp ATL is a six site restaurant discovery network for Atlanta. One shared venue dataset, five audience subdomains reading it through different lenses.
Server logs showed a sudden spike in AI crawler traffic. The obvious read was that the AI companies had finally found the site.
We checked before celebrating.
The finding.
The spike was real. It was also almost entirely self inflicted.
On September 9, GPTBot made 43,309 requests across 38,971 distinct URLs. That is 1.1 requests per URL, so it was one clean sweep rather than a stuck loop.
The URLs were the problem. Every subdomain hub offers four filter dimensions plus a sort control, and every filter combines with every other one.
Twelve venues on the bars subdomain generated 26,863 crawlable URLs. Across the network, crawlers found 40,404 of them.
Most of those URLs return nothing at all. We opened two of them, a distillery with a DJ in Lindbergh and a kid friendly nightclub in Decatur. Both returned the same empty result message wrapped in the same 22KB page.
Here is the part worth keeping. The canonical tag was already correct on every one of those URLs, pointing back at the unfiltered hub the whole time.
Canonical governs indexing. It does not govern crawling. GPTBot read the entire space regardless.
For scale, Search Console shows this network at zero clicks and 620 impressions over the same window. The machines were busy. Nobody else was.
| Subdomain | Venues | Filter URLs crawled | Filter requests | Real URLs crawled |
|---|---|---|---|---|
| bars | 12 | 26,863 | 30,808 | 113 |
| bbq | 8 | 7,890 | 8,903 | 71 |
| vegan | 8 | 6,213 | 7,207 | 106 |
| nightlife | 6 | 2,091 | 2,653 | 41 |
| family | 5 | 1,709 | 1,738 | 94 |
The change.
Two changes, both on the subdomains only. The base site reads no filter parameters, so it was left alone.
- robots.txt now disallows the query string space on all five subdomains.
- Filtered and sorted views now carry a noindex, follow robots tag, so anything already discovered drops out of the index while venue links still pass through.
Unfiltered hubs, neighborhood pages and guide pages were each checked and carry no robots tag at all.
Whether crawl volume moves onto real pages. Across the retained window the split was roughly 52,000 requests to filter URLs against 13,000 to real ones. A recheck in late September shows whether blocking the filter space redirected that budget onto venue and neighborhood pages, or simply removed it.
More real work, shown
Every field note is one finding from a live website, the data behind it, and the metric we are watching next.