SEO Life This Week
SEO Life This Week: GPTBot Stopped. Meta's Crawler Kept Going.
The same robots.txt line took GPTBot from 63,649 filter requests to zero and Meta's crawler from 937 to 10,566. Then our own build deleted the rule from thirty sites.
What the week says.
Crawler compliance was different for each vendor, and a correct hand edit lasted only until the generator rebuilt the file.
The same robots.txt line stopped GPTBot but not Meta's crawler
Across BellyUp Atlanta, GPTBot filter requests fell from 63,649 to zero after a query-string rule was added to robots.txt. OAI-SearchBot also fell to zero, Amazonbot dropped 92 percent, and Meta's crawler rose from 937 to 10,566.
The comparison uses August 28 to September 11 before the rule and September 12 to October 2 after it. The dates isolate one crawl-control change, but they do not remove every outside influence.
| Crawler | Before the rule | After the rule | Measured change |
|---|---|---|---|
| GPTBot | 63,649 | 0 | Down 100% |
| OAI-SearchBot | 19 | 0 | Down 100% |
| Amazonbot | 545 | 45 | Down 92% |
| Meta external agent | 937 | 10,566 | Up 11.3 times |
Before is August 28 through September 11. After is September 12 through October 2, beginning the day after the query-string rule changed.
| Metric | September 14 to 20 baseline | September 24 to 30 checkpoint | Movement |
|---|---|---|---|
| Exact-query impressions | 139 | 22 | Down 84.2% |
| Exact-query clicks | 6 | 10 | Up 66.7% |
| Exact-query CTR | 4.32% | 45.45% | Up 41.13 points |
| Exact-query weighted position | 17.07 | 21.55 | Worse by 4.48 |
| Articles hub impressions | 12 | 5 | Down 58.3% |
| Articles hub clicks | 2 | 0 | Down 2 |
| Articles hub weighted position | 30.83 | 16.40 | Better by 14.43 |
| Registered thresholds reached | Not scored | 0 of 4 | Opening round missed |
The campaign published September 23. A separate Real SEO™ definition update published September 24, so this checkpoint measures the combined dated intervention and does not assign causation to one page.
Our build deleted the crawl rule from thirty sites
The rule survived three weekly backups and held GPTBot at zero for twenty-one days. On October 2 the sitemap generator rebuilt robots.txt from a fixed string and erased the rule from all thirty BellyUp category subdomains without warning.
That turns the real correction from another hand edit into a build change. The open test will watch whether filter requests return now that the rule is gone.
Fewer filter requests did not prove more useful crawling
Clean URL requests on Atlanta roughly doubled, but portfolio GPTBot activity rose about twentyfold over the same period. Two candidate controls also had no before window, so we cannot credit the robots rule with moving crawl activity to real pages.
A server request only proves that a request reached a URL. It does not prove that the page was indexed, cited, used for training or read by a person.
The Real SEO™ Articles hub missed all four opening thresholds
The first complete seven-day window after publication brought 22 exact-query impressions and 10 clicks sitewide. The opening baseline had 139 impressions and 6 clicks, so clicks rose while impressions fell 84 percent and weighted position worsened from 17.07 to 21.55.
The Articles hub itself had 5 impressions, no clicks, and a weighted position of 16.40. It reached none of the four registered opening thresholds, and the campaign cannot be credited for movement that started before publication.
Krisada's read and Kodi's AI read.
I diagnosed the filter trap on September 9, published it, and then launched fifteen more sites with the same defect. The crawl rule worked. The process failed because the fix never reached the generator. Writing the article was not the fix.
The crawler split and the Real SEO™ checkpoint point in the same direction. Labels and totals are easy. Conditions and dates are what make them mean something. A correction only counts when the build preserves it and the next measurement can prove it stayed fixed.
Keep up with SEO Life
Every Saturday issue freezes the data, records what changed, and names the metric worth watching next.