SEO Life This Week
SEO Life This Week: GPTBot Stopped. Meta's Crawler Kept Going.
The same robots.txt line took GPTBot from 63,649 filter requests to zero and Meta's crawler from 937 to 10,566. Then our own build deleted the rule from thirty sites.
Audio Summary
Listen to This Week's Summary
This two-minute recording covers the crawler split, the build that erased the robots.txt repair, and the Real SEO™ opening checkpoint.
Read by Kodi, AI co-architect
Read transcript
This week showed a clear difference in how AI crawlers follow robots.txt rules.
On BellyUp! Atlanta, GPTBot made more than sixty-three thousand filter requests before the new rule. After the rule was added, GPTBot dropped to zero. OAI SearchBot also dropped to zero, and Amazonbot dropped by more than ninety percent.
Meta did the opposite. Its crawler went from fewer than one thousand filter requests to more than ten thousand. So, the exact same rule worked very differently depending on the crawler.
Then another problem appeared.
The robots rule worked for twenty-one days, but the website build system later recreated the robots file and accidentally deleted the rule from thirty BellyUp! sites.
That means the real problem was not just fixing robots.txt. The fix also had to be added to the system that creates the file. Otherwise, the next website build could erase the repair.
The data also showed an important warning.
Stopping bad filter crawling did not prove that the bots started crawling useful pages instead. Clean page requests increased, but crawler activity across the whole portfolio also increased heavily during the same period. There is not enough evidence yet to say the robots rule caused the improvement.
A server request only proves that a crawler reached a URL. It does not prove the page was indexed, used by an AI system, cited in an answer, or seen by a person.
The Real SEO campaign also reached its first checkpoint.
Clicks for the exact search terms increased from six to ten, but impressions dropped from one hundred thirty-nine to twenty-two. Average ranking also became somewhat worse.
The Articles hub received only five impressions and no clicks. It missed all four of its opening goals. It is still too early to give the campaign credit for changes that may have started before it was published.
The biggest lesson from the week is simple.
Finding a problem is not enough. Writing about the problem is not enough. Even making the correct repair is not enough if the system later removes it.
A fix only becomes real when the system keeps the fix in place and the next round of data proves that it stayed fixed.
What the week says.
Crawler compliance was different for each vendor, and a correct hand edit lasted only until the generator rebuilt the file.
The same robots.txt line stopped GPTBot but not Meta's crawler
Across BellyUp! Atlanta, GPTBot filter requests fell from 63,649 to zero after a query-string rule was added to robots.txt. OAI-SearchBot also fell to zero, Amazonbot dropped 92 percent, and Meta's crawler rose from 937 to 10,566.
The comparison uses August 28 to September 11 before the rule and September 12 to October 2 after it. The dates isolate one crawl-control change, but they do not remove every outside influence.
| Crawler | Before the rule | After the rule | Measured change |
|---|---|---|---|
| GPTBot | 63,649 | 0 | Down 100% |
| OAI-SearchBot | 19 | 0 | Down 100% |
| Amazonbot | 545 | 45 | Down 92% |
| Meta external agent | 937 | 10,566 | Up 11.3 times |
Before is August 28 through September 11. After is September 12 through October 2, beginning the day after the query-string rule changed.
| Metric | September 14 to 20 baseline | September 24 to 30 checkpoint | Movement |
|---|---|---|---|
| Exact-query impressions | 139 | 22 | Down 84.2% |
| Exact-query clicks | 6 | 10 | Up 66.7% |
| Exact-query CTR | 4.32% | 45.45% | Up 41.13 points |
| Exact-query weighted position | 17.07 | 21.55 | Worse by 4.48 |
| Articles hub impressions | 12 | 5 | Down 58.3% |
| Articles hub clicks | 2 | 0 | Down 2 |
| Articles hub weighted position | 30.83 | 16.40 | Better by 14.43 |
| Registered thresholds reached | Not scored | 0 of 4 | Opening round missed |
The campaign published September 23. A separate Real SEO™ definition update published September 24, so this checkpoint measures the combined dated intervention and does not assign causation to one page.
Our build deleted the crawl rule from thirty sites
The rule survived three weekly backups and held GPTBot at zero for twenty-one days. On October 2 the sitemap generator rebuilt robots.txt from a fixed string and erased the rule from all thirty BellyUp! category subdomains without warning.
That turns the real correction from another hand edit into a build change. The open test will watch whether filter requests return now that the rule is gone.
Fewer filter requests did not prove more useful crawling
Clean URL requests on Atlanta roughly doubled, but portfolio GPTBot activity rose about twentyfold over the same period. Two candidate controls also had no before window, so we cannot credit the robots rule with moving crawl activity to real pages.
A server request only proves that a request reached a URL. It does not prove that the page was indexed, cited, used for training or read by a person.
The Real SEO™ Articles hub missed all four opening thresholds
The first complete seven-day window after publication brought 22 exact-query impressions and 10 clicks sitewide. The opening baseline had 139 impressions and 6 clicks, so clicks rose while impressions fell 84 percent and weighted position worsened from 17.07 to 21.55.
The Articles hub itself had 5 impressions, no clicks, and a weighted position of 16.40. It reached none of the four registered opening thresholds, and the campaign cannot be credited for movement that started before publication.
Krisada's read and Kodi's AI read.
I diagnosed the filter trap on September 9, published it, and then launched fifteen more sites with the same defect. The crawl rule worked. The process failed because the fix never reached the generator. Writing the article was not the fix.
The crawler split and the Real SEO™ checkpoint point in the same direction. Labels and totals are easy. Conditions and dates are what make them mean something. A correction only counts when the build preserves it and the next measurement can prove it stayed fixed.
Keep up with SEO Life
Every Saturday issue freezes the data, records what changed, and names the metric worth watching next.