Experiment
What Happens After We Close an AI Crawler Trap?
A live before-and-after experiment measuring whether crawler requests move from 40,404 faceted URL combinations toward real pages and machine-readable venue resources after the discovery structure changes.
When the query-string crawl space is closed and the useful machine-readable resources become explicitly discoverable, returning verified crawlers will spend a substantially smaller share of requests on filter URLs and a larger share on base pages, venue pages, catalogs, AI instructions, and venue data. A request redistribution would support the influence of structure on discovery. It would not prove indexing, citation, training use, ranking influence, or human demand.
What is being tracked.
How the test is structured.
The before state is Digital Karma event 595. Across the retained period, the BellyUp ATL network logged 46,829 GPTBot-classified requests. Of those, 46,727 came from 41 of 43 source IP addresses inside OpenAI's published GPTBot ranges.
On September 9, 2026, GPTBot made 43,309 requests across seven logged hosts. The six live properties produced 38,971 distinct requested URL strings after identical route strings across hosts were counted once. The faceted system exposed 40,404 combinations from only 31 venues, including 26,863 combinations from twelve venues on the bars property.
The September 11 intervention closed the query-string crawl space in robots.txt, added noindex,follow to filtered states, corrected a contradictory API rule, added AI instructions to the five live category properties, exposed all 31 venue records through working catalog links, and published an honest well-known AI discovery record.
The server logs, GSC tables, and dated intervention event will be read on the same timeline. The first checkpoint is planned for roughly September 25, while the raw request evidence remains inside the 35-day retention window.
What has happened so far.
Baseline recorded September 11, 2026.
- 43,309 GPTBot requests on September 9
- 40,404 exposed filter combinations
- 46,504 filter HTML requests across the retained period
- 2 venue JSON requests
- 2 AI foundation file requests
- 1 AI instructions file request
- roughly 620 Google impressions and 0 clicks on the main site's Search Console view at analysis time
The intervention is live. No post-change redistribution conclusion has been written.
What would support or challenge it.
Status: active.
If verified GPTBot returns and the request share moves away from filter URLs toward useful human pages and machine resources, the hypothesis gains support. If old filter URLs continue dominating, the crawler may be working from previously learned inventory or the closure may be insufficient. If the crawler does not return during the first window, the checkpoint is inconclusive.
Other crawler families will be measured separately because one bot's behavior does not establish a general rule. Search impressions, indexing evidence, AI citations, and human activity remain separate outcomes.
We do not architect the conclusion. We wait for the receipt.
Keep Following the Tests
Move from this open thread back into the full experiment library.