SEO Life This Week

SEO Life This Week: GPTBot Stopped. Meta's Crawler Kept Going.

The same robots.txt line took GPTBot from 63,649 filter requests to zero and Meta's crawler from 937 to 10,566. Then our own build deleted the rule from thirty sites.

Issue 3 Krisada Eaton with Kodi 11 views
Lead Finding

What the week says.

Crawler compliance was different for each vendor, and a correct hand edit lasted only until the generator rebuilt the file.

What Moved

The same robots.txt line stopped GPTBot but not Meta's crawler

Across BellyUp Atlanta, GPTBot filter requests fell from 63,649 to zero after a query-string rule was added to robots.txt. OAI-SearchBot also fell to zero, Amazonbot dropped 92 percent, and Meta's crawler rose from 937 to 10,566.

The comparison uses August 28 to September 11 before the rule and September 12 to October 2 after it. The dates isolate one crawl-control change, but they do not remove every outside influence.

CrawlerBefore the ruleAfter the ruleMeasured change
GPTBot63,6490Down 100%
OAI-SearchBot190Down 100%
Amazonbot54545Down 92%
Meta external agent93710,566Up 11.3 times

Before is August 28 through September 11. After is September 12 through October 2, beginning the day after the query-string rule changed.

MetricSeptember 14 to 20 baselineSeptember 24 to 30 checkpointMovement
Exact-query impressions13922Down 84.2%
Exact-query clicks610Up 66.7%
Exact-query CTR4.32%45.45%Up 41.13 points
Exact-query weighted position17.0721.55Worse by 4.48
Articles hub impressions125Down 58.3%
Articles hub clicks20Down 2
Articles hub weighted position30.8316.40Better by 14.43
Registered thresholds reachedNot scored0 of 4Opening round missed

The campaign published September 23. A separate Real SEO™ definition update published September 24, so this checkpoint measures the combined dated intervention and does not assign causation to one page.

What Changed

Our build deleted the crawl rule from thirty sites

The rule survived three weekly backups and held GPTBot at zero for twenty-one days. On October 2 the sitemap generator rebuilt robots.txt from a fixed string and erased the rule from all thirty BellyUp category subdomains without warning.

That turns the real correction from another hand edit into a build change. The open test will watch whether filter requests return now that the rule is gone.

What Is Not Proven

Fewer filter requests did not prove more useful crawling

Clean URL requests on Atlanta roughly doubled, but portfolio GPTBot activity rose about twentyfold over the same period. Two candidate controls also had no before window, so we cannot credit the robots rule with moving crawl activity to real pages.

A server request only proves that a request reached a URL. It does not prove that the page was indexed, cited, used for training or read by a person.

Case Study Update

The Real SEO™ Articles hub missed all four opening thresholds

The first complete seven-day window after publication brought 22 exact-query impressions and 10 clicks sitewide. The opening baseline had 139 impressions and 6 clicks, so clicks rose while impressions fell 84 percent and weighted position worsened from 17.07 to 21.55.

The Articles hub itself had 5 impressions, no clicks, and a weighted position of 16.40. It reached none of the four registered opening thresholds, and the campaign cannot be credited for movement that started before publication.

Two Reads

Krisada's read and Kodi's AI read.

Krisada's Read

I diagnosed the filter trap on September 9, published it, and then launched fifteen more sites with the same defect. The crawl rule worked. The process failed because the fix never reached the generator. Writing the article was not the fix.

Kodi's Read

The crawler split and the Real SEO™ checkpoint point in the same direction. Labels and totals are easy. Conditions and dates are what make them mean something. A correction only counts when the build preserves it and the next measurement can prove it stayed fixed.

Saturday Review

Keep up with SEO Life

Every Saturday issue freezes the data, records what changed, and names the metric worth watching next.