This Week at Real SEO Life: The Robots Tripled Up and One Site Jumped 31 Spots

A crawler is a robot program. It visits web pages, reads them, and files what it read. OpenAI runs some. So do Google, Meta, Amazon, Anthropic, Perplexity and about a dozen others.

What those robots read is what the AI chat tools use later, when somebody types a question. So they matter for one simple reason. If no robot ever comes to your site, no chat tool learns your site exists. You can write the best page on the internet and it will sit there alone.

That is why we count robot visits. It is the earliest sign that anyone out there, person or machine, knows you are here.

Here is how many robot visits a day our seven main sites got, week by week:

➤ Week of September 7: 683 a day ➤ Week of September 14: 1,217 a day ➤ Week of September 21: 2,249 a day ➤ Week of September 28: 2,684 a day ➤ October 5 to 9: 2,004 a day

That last number is almost three times where we started in early September.

One note on counting. That last stretch is five days, not seven, because our data stops October 9. So I counted per day. A five day week next to a seven day week always looks like a drop even when nothing happened. More on that later, because I fell for it.

Why this matters to you: robot visits are the cheapest thing you can measure and almost nobody looks. Your web host logs every visit. The robots are in there right now, by name. Go look.

In the last 28 days, RealSEOLife.com was visited by 12 different AI robots. Here are the five biggest:

➤ Meta's robot: 23,224 visits ➤ OpenAI's GPTBot: 5,366 visits ➤ Anthropic's ClaudeBot: 3,528 visits ➤ Amazon's robot: 2,282 visits ➤ Perplexity's robot: 1,146 visits

That is 37,688 visits total, and Meta alone is about 6 out of every 10. I have no idea why Meta likes us this much. I just know it does.

Here is the part I like. We are a small site and we are not famous. Every one of those companies still sent a robot. You do not have to be big to get read. You have to be readable.

AISymantix.com is our site about measuring AI visibility. In late August its average spot in Google was about 60th. Now it is about 29th.

Quick explainer, because this number fools people. Google gives you an average spot across every search you showed up in. Lower is better. Number 1 is the top of the page.

My first instinct was to be suspicious, and here is why. Average position can improve for a boring reason. If you stop showing up for a bunch of searches where you were sitting 80th, your average gets better even if not one page moved an inch. Picture a class where the lowest scoring kids transfer out. The class average goes up. Nobody studied harder.

So I went and looked at the individual searches instead of the average. Then I stopped being suspicious:

➤ "searchmetrics seo visibility": 73rd to 12th ➤ "searchmetrics visibility": 62nd to 20th ➤ "searchmetrics seo visibility score": 24th to 8th ➤ "search visibility score": 53rd to 33rd ➤ "search visibility": 57th to 48th

Twelve of the fifteen searches we track on that site improved. Those pages really moved. Eighth place for anything is a good day around here. One brand new search showed up too, "what does ai search visibility mean for seo", straight in at about 31st.

And here is the twist. Two searches did drop off the list completely, and one of them we had been sitting 27th for. Losing a good spot drags your average the wrong way. So the real climb is a little bigger than the average makes it look, not smaller.

Why this matters to you: average position can move for a real reason or a boring one, and the number itself will not tell you which. Ten minutes looking at your individual searches will. Do not assume in either direction. I almost talked myself out of good news.

For most of August, RealSEOLife.com got 0 or 1 clicks a week from Google. Then:

➤ Week of September 14: 8 clicks ➤ Week of September 21: 12 clicks

Twelve clicks. I am telling you a number that small on purpose.

Our best search is "real seo". Twenty two clicks from it, sitting about 21st. That is page three. Page three is not good. It is also not nothing.

Why this matters to you: every chart in every SEO case study started here, at almost zero. Nobody publishes this part. We will, every week, up or down.

Now the surprise, and it is the most useful thing here.

We keep logs for about 180 websites. Across all of them, robot visits went from 266,465 in 28 days to 2,737,765 in the next 28. Ten times more.

Almost none of it came to the seven sites we write on. Four robots did nearly all of it: OpenAI's, Meta's, Amazon's and Anthropic's.

So where did two and a half million visits go?

Our city food and nightlife guides. Those pages have filter buttons. Price, neighborhood, open now. Click a different mix of buttons and the web address changes, which means every mix looks like a brand new page to a robot.

One of those sites has about 17 real pages. It has more than 82,000 possible button combinations. The robots went and tried them.

Being straight with you: this week's file does not break the two and a half million down site by site, so I cannot prove every last visit landed there. But we measured those guide sites directly over the past two weeks and the counts were in the hundreds of thousands each. That is where it points.

If your site has filter buttons, sort buttons, or a date picker, this might be happening to you right now.

Robots have limited time for your site. If they spend it on 82,000 button combinations, they are not spending it on the page you worked on all weekend.

We are not telling anyone to delete their filters. Filters are good for people, and people come first. We are going to try moving the button state into the part of a web address that never reaches the server, so humans keep every filter and robots stop seeing fake pages. We will report back.

Your own check takes five minutes. Open your server log. Count the robot visits with a question mark in the address. If that beats your number of real pages, you found it.

On September 24, Google added something to Search Console that is genuinely fun.

You can now see when somebody found you by searching with a picture instead of words. Google Lens, Circle to Search on Android, uploading a photo, or right clicking an image in Chrome and choosing "Search this image."

People point their phone at a thing and search for it. That has been happening for years and you could not see it. Now there is a filter for it in your reports.

Why this matters to you: if your pages have real photos of real things, turn that filter on this week. One caution. Google did not say whether those searches were counted before September 24, or whether there is any older data. So read the new numbers, but do not compare across that date yet.

Researchers published a test on October 4 that we liked a lot. They took seven AI agents, the kind that search the web to answer you, then rigged the search tool to come back empty and tried different ways of saying so.

When the tool said "no results, and here is why" in one sentence, the agents mostly admitted they did not know. One group went from admitting it about 23 percent of the time to about 97 percent. When the tool came back quiet instead, one agent made up sources.

A polite no with a reason works. Silence invites made up answers.

Why this matters to you: sooner or later your site turns a robot away. When it happens, say so in words on the page. A blank error screen teaches a robot nothing. One sentence teaches it something.

Fair warning. The test used a pretend search index, not real websites, so nobody has proven it on a live server yet.

Leaving this in, because you will make the same one.

I opened this site's Google numbers and saw impressions go 1,304, then 682, then 558, then 332. Four weeks straight down. My stomach dropped a little.

Then I checked the dates. That last number covers two days, not seven. Google's data only ran through October 6. Two days and 332 impressions is 166 a day. The week before was 80 a day. So it is more than double, and it is the best of the four. I had it backwards for ten minutes.

The rule is free: before you believe a number went down, count how many days it covers. The newest bar on any chart is almost always short, and short always looks like bad news.