Experiment
Did Ending The Crawl Cost Any Visibility
On November 1 we stopped 1.2 million AI crawler requests a day by cutting every filter link down to a single setting. This test asks the only question that has money attached: did any of that crawling buy us visibility we are now losing? It reads Google impressions and clicks across the treated sites every week for four weeks, beside the requests still arriving from the fetchers that feed AI answers rather than AI training.
The 1.2 million requests a day that stopped on 2026-11-01 were buying nothing on the visibility side, so over the 28 days after the change the treated sites will hold their Google impressions inside the band the untreated weeks already produced, and requests from the retrieval-side fetchers to those sites will not fall by more than half once filter addresses are separated from content addresses.
What this test is watching.
Our guide sites let people filter venues by things like neighborhood and price. Until November 1 every one of those filter links carried all the settings already switched on, so each extra click multiplied the addresses the site published, and AI crawlers walked millions of them. We cut each link down to one setting and the crawling stopped. The worry underneath that result is simple. If more than a million requests a day were reading our pages, maybe some of them were worth something, and we just threw it away. So we wrote down the losing number before we looked: if Google impressions across these sites drop more than 20 percent for seven days running, the change cost us real visibility and we take it back.
How the test is structured.
The treated group. Every BellyUp! host in the warehouse, 154 of them, covering ten city hubs and their lens subdomains. The treatment is the single-setting filter link helper applied to 119 lens subdomains on 2026-11-01 at 02:50 UTC, recorded in the companion test one-filter-per-link-stopped-the-crawl.
What is measured. Summed daily impressions and clicks from the warehouse Search Console page table, read weekly. Beside that, daily requests per crawler fingerprint from the raw request log, split into total requests and requests to addresses carrying no query string. That split is the part most measurements skip and it turned out to decide the result.
The losing condition, written before the first reading. The test fails if summed daily impressions fall below 1,855 a day, which is 20 percent under the 2,318 a day measured across the seven complete days to 2026-10-31, on seven consecutive complete Search Console days. It fails more usefully if requests from the four retrieval-side fetchers, OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-User, fall by more than half against their pre-treatment rate, which would mean the removed links were feeding the answer side and not only the training side.
What would rewrite it rather than close it. If impressions hold but clicks fall, the loss is in how results are presented rather than in discovery, and that needs its own series.
Known limits. Search Console lags two to three days, so the newest complete day always trails the run date. The groups are not randomised. Impressions are not rankings. Clicks run between 8 and 25 a day across the whole window, which is too small to carry a conversion claim either way.
What has happened so far.
2026-11-14 checkpoint, reading one of four. Both legs hold.
Impressions across the treated hosts ran 2,318 a day across the seven complete Search Console days to 2026-10-31, and 2,231 a day across the ten complete days from 2026-11-01 to 2026-11-10. A difference of 3.8 percent.
Week against week it is closer still: 16,224 impressions and 100 clicks in the seven complete days to 2026-10-31, against 16,104 impressions and 85 clicks in the seven complete days to 2026-11-10. Down 0.7 percent on impressions.
The weakest single day in the whole post-treatment window is 1,977 on 2026-11-03, which is 6.6 percent above the losing number. Two of the ten days, 2,752 on 2026-11-06 and 2,745 on 2026-11-07, are the highest readings in the month.
The daily series after the treatment: 2,096, 2,133, 1,977, 2,148, 2,194, 2,752, 2,745, 2,138, 2,106 and 2,021.
The crawler leg needed a correction before it could be read.
The losing condition names four retrieval-side fetchers and never says which four, which is a defect in how it was written rather than in the data. Read on total requests, OAI-SearchBot alone would have failed the test. Read on requests to real content addresses, nothing does.
OAI-SearchBot went from 13,924 requests in the pre week to 4,713 in the post week, down 66.2 percent. But 9,176 of those 13,924 requests carried a filter query string, and on 2026-10-31 its query-string share hit 92.8 percent. Its requests to addresses with no query string went from 4,748 to 3,264, down 31.3 percent.
ChatGPT-User went from 431 requests to 339, with content requests down 20.3 percent. PerplexityBot went from 135 to 934, with content requests up 528 percent from a small base, driven by one 823-request day on 2026-11-08. Claude-User went from 7 requests to 16.
So the metric is now stated properly: the retrieval leg is read on requests to addresses with no query string, across those four named fingerprints.
A side result that arrived by accident.
Eighteen new sites were cloned from the same code on 2026-11-05 through 2026-11-07, after the link change. The one that publishes a filter grid, 14 single-setting addresses, received exactly 14 distinct filter addresses from Amazonbot, 14 from GPTBot and 11 from ClaudeBot, at one request each. The other 17 publish no filter links and received no filter requests at all.
That is not part of this test, but it answers the question a reader of it would ask next: a site built this way from the start is not crawled hard either.
The crawl floor also kept falling. The 48,207 requests a day reported on 2026-11-07 was Amazonbot still working through its queue. Across the seven complete days to 2026-11-12 the same hosts took 4,283 requests a day, down 99.66 percent from 1,263,440.
Next reading 2026-11-21. Decision 2026-11-29.
What would support or challenge it.
Nothing is concluded yet, on purpose. This is reading one of four inside a 28-day window that runs to 2026-11-29, and the ledger this test reports into has twice refused to move a finish line after one good reading.
What can be said today is that the first reading went the way the hypothesis predicted on both legs, and that the one number which looked like a failure stopped looking like one as soon as filter addresses were separated from content addresses. A search-side fetcher spending two thirds of its visits on filtered duplicates of the same venue list was not visibility. It was waste, and it was waste we published.
If the impressions series is still inside its band on 2026-11-29, the practical result is a build rule rather than a repair: publish one setting per filter link from the first day a site exists.
Keep Following the Tests
Move from this open thread back into the full experiment library.