Field Note
We Published a Dataset for Machines. They Read It Five Times.
A JSON record for every venue, a category taxonomy, a federation manifest and an llm.txt. In 25 days machines read the whole layer five times.
The setup.
BellyUp ATL publishes more than pages. There is a JSON record for every venue, a discovery category taxonomy, a federation manifest, and an llm.txt written specifically for AI systems.
The crawlers showed up in real volume. So we measured which of it they actually opened.
The finding.
They read the HTML and skipped the data.
Filter pages absorbed 46,504 successful requests. The venue JSON records got two. The federation files got two. llm.txt got one.
Three things were blocking it. All three were ours.
- robots.txt disallowed the API directory while llm.txt instructed crawlers to fetch a file inside it. The site was publishing two opposite instructions.
- The venue data directory has indexes disabled and returns 404, and nothing listed the 31 individual record URLs. Machines had no way to enumerate what existed.
- The five subdomains, which absorb roughly 95 percent of all crawl, carried no llm.txt at all. Every request for one returned 404.
There was one more signal we nearly filed as noise. 110 requests hit the well-known directory and every single one returned 404.
Some of that was ordinary scanner junk. The rest was agents, including Claude and an agent verification service, hunting for a machine interface and finding a locked door.
| Surface | Requests in 25 days |
|---|---|
| Filter HTML pages | 46,504 |
| well-known requests (every one a 404) | 110 |
| Discovery taxonomy JSON | 8 |
| Venue JSON records | 2 |
| AI federation files | 2 |
| llm.txt | 1 |
The change.
Four fixes, all generated by the build script rather than hand written, so they cannot drift.
- robots.txt now explicitly allows the venue API file it used to block.
- The AI catalog now lists a direct JSON URL for all 31 venue records.
- All five subdomains now serve their own llm.txt pointing back at the shared dataset.
- The well-known directory now serves a security contact file and an agent discovery pointer. It also turned out to be unreachable by configuration, so that had to be opened before anything in it could work.
One deliberate omission. The agents were asking for MCP and agent card files, and this site runs neither an MCP server nor an agent.
Publishing a card for an endpoint nothing can connect to is worse for an agent than a clean 404. The discovery pointer states plainly what the site is and what it is not.
Requests to the JSON layer. The baseline is five reads in 25 days against 46,504 HTML requests. If the catalog listing and the subdomain pointers do their job, venue record reads become a visible number instead of a rounding error. Checked in late September.
More real work, shown
Every field note is one finding from a live website, the data behind it, and the metric we are watching next.