Research Methodology

Once the filter trap was obvious, the rest of the request pattern started making more sense.

The site had several different discovery problems.

• robots.txt blocked an API location that the AI instructions file encouraged machines to visit
• five live category properties had no useful machine-readable instructions file
• the venue data had no working catalog that exposed all 31 records
• agents made 110 requests for expected well-known resources that did not exist

Some doors were missing.

One door had contradictory signs.

Another enormous door opened onto 40,404 mostly useless rooms.

The crawler trap makes one thing clear: architecture changes what becomes easy to discover.

That is direction, not command.

We cannot make GPTBot return. We cannot make it request a catalog. We cannot make OpenAI index, use, cite, or value anything it retrieves.

What we can do is stop creating tens of thousands of low-value choices and make the useful choices easier to find.

Then the server logs can tell us whether the machine behavior changes.

The before state is preserved in the Digital Karma event record. Then we changed five things across the six live domains.

• closed the query-string crawl space in robots.txt
• added noindex,follow to filtered and sorted page states
• corrected the API contradiction so the published venue feed is allowed
• added a real AI instructions file to each live category property
• rebuilt the machine catalog so all 31 venue records have a working data URL
• added an honest well-known AI discovery record without claiming an MCP server or agent exists

That last restraint matters. Bots requested MCP and agent-card resources, but the site does not run either system. A clean statement of what exists is more useful than advertising a door that opens onto nothing.

The accident was not the experiment.

The bot behavior was already happening. What changed is that we built enough instrumentation to see it.

The experiment begins now because the before state exists, the intervention has a date, and the next request can be compared against both.

The first checkpoint is roughly two weeks after the September 11 change. Raw request retention is 35 days, so this is not something to remember later and reconstruct from a vague impression. The evidence window has a clock on it.

If GPTBot returns and query-string requests fall while useful pages and machine resources receive a larger share, the redistribution hypothesis gains support.

If the bot returns and keeps hammering the old URLs, we learn that closing the current gate does not erase previously learned inventory.

If the crawler does not return during the first checkpoint window, the result is inconclusive. Silence is not proof that the new structure failed.

If the datasets remain untouched, that is also a result. It would tell us that making a resource available and making it discoverable are still not the same thing.

The proposition is not that we know how to control AI crawlers.

We do not.

The useful proposition is that we can see signals most websites do not measure, build deliberate discovery structure, and observe what machines actually do afterward.

That is a smaller claim.

It is also one we can prove or disprove with a server log.