One Filter Per Link Stopped The Crawl: 1.2 Million AI Requests A Day Gone

Each guide has a filter bar with six groups. Neighborhood and food type. Experience and dietary. Discovery and sort. Normal stuff.

The old helper did something that sounds helpful and is not. When you clicked a filter, every link on the page it drew next kept whatever you already had switched on, and added itself.

The old code comment said so out loud: qs(['experience' => 'kid_friendly']) keeps whatever neighborhood, dietary, type and sort are already selected.

That is correct behavior for a person. You want your filters to stack.

For a crawler it is a copy shop that never closes.

Start on the home page. 143 single-setting links.

Follow one. The new page now publishes 143 links that each carry two settings.

Follow one of those and you get 143 three-setting addresses.

Every one of them returns a real page with a 200 and real content. So nothing looks broken anywhere.

We measured the result on one host on October 28, before any of this was fixed. GPTBot made 99,540 requests at 99,539 different addresses in a single day. Different address almost every time.

That was one lens, on one day, out of 119.

This is the part worth taking away even if you never touch a guide site.

On an unfiltered home page there is nothing switched on yet. So the helper had nothing to carry, and every link it drew carried exactly one setting. 143 links, all clean, all single.

An audit that renders the home page and counts links gives that host a pass. Ours did. For three weeks.

The damage lived on the filtered views, and no audit I had been running ever rendered one.

I checked the live pages again this week, after the fix. Three views: unfiltered, one setting deep and two settings deep. All three publish 143 links. Every one carries a single setting and none carries more.

Before the change the two-setting page would have published 143 links each carrying three settings. Same count. Completely different consequence.

If you take one thing from this, take that. A link count tells you nothing. The number of settings inside one link is the whole story, and you can only see it on a page where a setting is already on.

I expected one curve. I got two.

GPTBot was at 173,172 requests on October 31 and 383,487 on November 1, the day of the change. Then 358. Then nothing worth counting. It drained whatever it was holding inside the same day and never came back for more.

Meta's crawler was faster still. 3,515 on October 31, then 10.

Amazonbot behaved like a different animal. 184,500 on October 31. The next day 129,643. Then 96,962 and 60,401 and 28,436, sliding to 2,326 by November 6. Roughly half of yesterday, every day, for six days.

Both ended in the same place. One took a day and one took most of a week.

That gap is interesting because it is not about compliance. Nobody was told to stop. The addresses simply stopped existing as links, and each company drained the ones it was already holding at its own speed.

I pulled a sample of Amazonbot's last requests on one host on November 6, to see whether it was still asking for multi-setting addresses nobody publishes anymore.

One row came back, carrying a single setting. By then that host was done too.

A 96 percent drop on its own is a story about nothing. Crawl volume moves for a dozen reasons and most of them have nothing to do with you.

The number means something because of the comparison. Same days. Same server. Same warehouse table.

The sites that did not get the change held flat. 10,353 requests a day before. 10,616 after.

That is the comparison doing the work, not the 96 percent.

I checked the boring thing first too, because the boring explanation is usually the right one. If our log import had broken on November 2, the whole series would have collapsed and I would have had a fake discovery.

It did not break. Sites reporting logs stayed between 267 and 276 a day right through. Human requests actually went up. November 6 has all 24 hours in it.

So the logs kept arriving. The crawlers stopped.

Worth saying plainly, because it is the least flattering part.

This did not come out of crawler analysis. It came out of a cPanel email on October 31 saying the drive was 83 percent full.

The raw log table had gone from around 500,000 rows a day in September to 2 to 3 million a day from October 10, and had grown to 46 GB. Nearly two thirds of the rows in the last week were AI crawlers, stacked on six or seven hosts.

So the crawl was not found as a visibility problem. It was found as a storage bill.

And the change that fixed it was approved to recover disk space. It only turned out afterwards to be the cleanest answer I have gotten to a research question I had open for four weeks.

We keep a running ledger on this site with four levels, and the whole point of it is to stop a working theory from turning into a fact because it got repeated three weeks running.

One entry has been sitting at level three since October 17. The question was simple and it decided which fix to build: are these crawlers following filter links we publish, or composing the combinations themselves?

If they follow links, removing the links ends it. If they compose addresses, removing links changes nothing and only a server rule will work.

The links went. The crawl ended.

That moves to level four this week, scoped to what actually got tested.

Multi-setting addresses stopped being published and stopped being requested. Single-setting links were kept and are still crawled at a low, ordinary rate.

The one number I still owe it is a count of distinct addresses per host per crawler per week. That query has now timed out on me twice against a 46 GB table.

The intervention answered the question better than the count would have. But the count is still unrun, and I would rather say so than let it quietly disappear.

I just removed 1.2 million AI crawler requests a day. The obvious worry is whether I removed something valuable with them.

The early reading says no, and the shape of it is the reassuring part.

The crawlers that collapsed are the ones that read everything and send nothing back.

The fetchers that go and get a page because somebody asked a question barely moved. ChatGPT-User went from 54 requests a day to 36. PerplexityBot went from 17 to 14. OAI-SearchBot fell from 1,986 to 606.

On the Google side, those hosts averaged 2,318 impressions a day across the seven days to October 31. The three days after the change read 2,096, 2,133 and 1,977.

I am not calling that a result. Search Console runs two to three days behind, so those are exactly the days most likely to be incomplete. It is a baseline, and the real reading comes after November 14.

The bet on the record is that 28 days from now those hosts hold their impressions and the answer-side fetchers do not fall. If impressions drop more than 20 percent under that baseline for a week straight, I was wrong about what those links were buying.