Experiment
One Filter Per Link: Cutting The Published Link Graph
A live test of whether AI crawlers walking millions of filter addresses are following links we publish or composing the combinations themselves. On 2026-11-01 every filter link on 119 guide subdomains was cut to carry a single setting instead of every setting already switched on. Declared AI crawler requests to those hosts fell 96.2 percent while the untreated remainder of the portfolio held flat.
The filter addresses these crawlers request are addresses reachable by following links we publish, not combinations the crawlers construct. If that is right, removing the multi-setting links from the rendered page ends the crawl. If the crawlers are composing the values themselves, removing links changes nothing and only a server-side rule will work. The two outcomes point at different fixes, which is why the question was worth testing rather than assuming.
What this test is watching.
Our guide sites let people filter venues by things like neighborhood and food type. Each filter used to be a link that kept your other choices switched on, which quietly created millions of different addresses for AI bots to fetch. We changed every filter link to carry one choice only, and we are watching whether the bot traffic stays away and whether Google visibility holds.
How the test is structured.
The treated group. 119 BellyUp! lens subdomains, every one built from the same shared hub template. Each publishes a filter bar with six groups: neighborhood, food type, experience, dietary, discovery and sort.
The defect. The old link helper merged the currently active filters into every link it drew. On an unfiltered home page there was nothing active, so every link carried one setting and the page looked correct. One click in, every link carried two settings. Two clicks in, three. The naive combination space on one measured host was 16,823,268 addresses.
The change. The link helper now builds each link from the chosen setting only. A group's All link became the bare page address, and a small inline script merges a click with whatever is already in the browser address bar so that people can still stack up to three filters plus a sort, exactly as before. No robots.txt rule was added, no page was hidden, no rate limit was applied.
The treatment timestamp. Taken from the file modification times on the server, not from the session date. All 119 templates were rewritten between 02:49:30 and 02:50:37 UTC on 2026-11-01, a 67-second rollout. The pre-change templates were kept in a dated backup directory on the server and were read back during measurement to confirm the old behaviour from the code itself.
The comparison group. Every other portfolio docroot reporting logs into the warehouse over the same days, untouched by the change. This group is what makes the result readable at all. A crawl volume change on its own means nothing.
What is measured. Daily requests per declared AI crawler fingerprint from the warehouse, split by treated and untreated hosts, reading the seven complete days to 2026-10-31 against the five complete days from 2026-11-02. 2026-11-01 is excluded from both as the treatment day. Alongside the crawler series: summed daily Google impressions and clicks for the treated hosts, and daily requests from the retrieval-side fetchers.
What has happened so far.
2026-11-07 checkpoint. The crawl ended.
Declared AI crawler requests to the 119 treated hosts fell from 1,263,440 a day across the seven complete days to 2026-10-31, to 48,207 a day across the five complete days from 2026-11-02. That is down 96.2 percent.
The untreated remainder of the portfolio went the other way over the same days, from 10,353 requests a day to 10,616. Up 2.5 percent.
Requests a day per crawler, seven complete days to 2026-10-31 against five complete days from 2026-11-02:
- GPTBot, OpenAI, training: 654,029 to 461. Down 99.93 percent.
- meta-externalagent, Meta, training: 376,722 to 436. Down 99.88 percent.
- Amazonbot, Amazon, training: 226,669 to 42,620. Down 81.2 percent.
- ClaudeBot, Anthropic, training: 2,199 to 639. Down 70.9 percent.
- OAI-SearchBot, OpenAI, AI search: 1,986 to 606. Down 69.5 percent.
- Bytespider, ByteDance, training: 1,744 to 2,331. Up 33.7 percent.
- ChatGPT-User, OpenAI, user retrieval: 54 to 36. Down 34.1 percent.
- PerplexityBot, Perplexity, AI search: 17 to 14. Down 17.9 percent.
The four heaviest crawlers accounted for essentially all of the fall. The three that go and fetch a page for a person moved by single-digit request counts.
The vendors stopped at different speeds. GPTBot read 173,172 requests on 2026-10-31 and 383,487 on the treatment day, then 358 the following day and nothing material afterwards. It drained what it was holding inside one day. Meta's crawler went from 3,515 to 10 immediately. Amazonbot took six days to reach the floor, roughly halving each day: 184,500, then 129,643, 96,962, 60,401, 28,436, 24,977 and 2,326.
The treatment was verified in rendered HTML, not assumed. Three live views of over50.bellyupcharleston.com were fetched on 2026-11-06 and parsed: unfiltered, one setting deep and two settings deep. All three returned HTTP 200 and all three publish 143 distinct filter links, every one carrying exactly one setting and none carrying more. Before the change, the two-setting view would have published 143 links each carrying three settings.
The boring explanation was ruled out first. If log ingestion had broken on 2026-11-02 the whole series would have collapsed and the result would be an artifact. It did not break. Sites reporting logs stayed between 267 and 276 a day right through the window, human-class requests rose rather than fell, and 2026-11-06 was checked hour by hour and carries all 24 hours.
Visibility baseline, not yet a result. Summed Google impressions across the treated hosts averaged 2,318 a day over the seven complete days to 2026-10-31, with clicks between 9 and 20. The three days after the change read 2,096, 2,133 and 1,977, an average of 2,069. Search Console lags two to three days, so those are exactly the days most likely to be incomplete. They are recorded as a baseline. The first conclusive reading is after 2026-11-14.
What would support or challenge it.
The hypothesis holds for multi-setting addresses, which is the scope that was actually tested. The links went and the crawl ended, on 119 hosts at once, against a comparison group that did not move. A server-side rule was never needed and no page was hidden to achieve it.
Two honest limits belong on the record. First, the volume was already falling before the change, from 1,974,216 requests on 2026-10-25 down to 879,370 on 2026-10-30, so part of what happened here was a burst that was ending anyway. What the change did was take it to the floor and hold it there, which the 2.5 percent rise in the untreated group over the same days supports. Second, the distinct-address census per host per crawler per week that this test originally asked for has still not been run. It timed out twice against a 46 GB raw log table. The intervention answered the question more directly than the count would have, but the count remains unrun rather than quietly dropped.
The open question moves to what it cost. Removing 1.2 million requests a day is only good news if those requests were buying nothing. The early shape is reassuring, because the crawlers that collapsed are the ones that read everything and send nothing back, while the fetchers that go and get a page because somebody asked a question barely moved at all. The bet on the record is that over the 28 days from 2026-11-01 the treated hosts hold their impressions and the answer-side fetchers do not fall. If summed impressions sit more than 20 percent under the 2,318 a day baseline for seven consecutive complete days, that bet was wrong.
Keep Following the Tests
Move from this open thread back into the full experiment library.