[Writing](https://ruesandora.com/writing/) Report

# Why 11 of 30 sites could not be measured: firewalls and scanners

By [Rues](https://ruesandora.com/about) · Published 29 October 2026 · Updated 29 October 2026

Sources checked on 25 October 2026

Short answer

On 19 October 2026, 11 of the 30 Turkish sites in our first list could not be measured: 7 refused the automated request, 3 had too little coverage, 1 was unreachable. Most refusals were e-commerce and local services. We do not know why they refused, so we do not say bots were blocked. Unmeasured is not good or bad.

What happens when a scanner gets a 403 from your site? No score comes out. We hit this while measuring 30 Turkish sites, and more than a third of the first list could not be measured. This post covers those sites: what happened, what we do not know and how to test your own site.

The measurement ran on 19 October 2026. On the publication day the measurement is 10 days old. For the overall results see [AI readiness of 30 Turkish sites](https://ruesandora.com/writing/turkey-websites-ai-readiness). Only the unmeasured side is covered here. We do not name sites.

## What happened in the first list?

We picked 30 sites by hand across eight sectors and wrote the list down before scanning. We ran the Ruescan engine from a single address with an honest bot identity. One scan per site, with a 45-second pause between scans.

| Result | Sites |
| --- | --- |
| Measured | 19 |
| Not measured | 11 |

How the 11 split:

| Reason | Sites | Meaning |
| --- | --- | --- |
| Refused the automated request | 7 | The server returned 401, 403 or 429, or showed a bot-protection page |
| Low coverage, score provisional | 3 | The scan produced a result but very little could be measured. No score was given |
| Unreachable | 1 | The scan reported the site as unreachable, we did not record the detail |

By sector, the unmeasured were: local services 4 of 4, e-commerce 3 of 4, travel 2 of 3, health 1 of 3, public sector and education 1 of 4. Every news, banking and finance, and software site (12 sites) was measured.

## The backups did the same

We replaced empty slots with backup sites from the same sector. In total we tried 57 sites. 30 were measured and 27 were not.

| Reason | Sites |
| --- | --- |
| Refused the automated request | 17 |
| Low coverage, score provisional | 5 |
| Unreachable | 3 |
| "Try again later" answer (load) | 2 |

27 of 57 tries, 47%, gave no result. By sector: e-commerce 10 of 14, travel 8 of 11, local services 5 of 9, public sector and education 2 of 6, health 2 of 5. None of the 12 news, banking and finance, and software tries failed.

## We do not know why they could not be measured

A 403 from a site does not mean it blocks AI bots. The refusal could come from several places: the address the request came from, the bot identity header, the request rate or some other rule in a security layer. We did not measure which one it was. So we do not write "bot blocked".

We ran one more experiment. For the 8 sites where the full scan failed, we sent two requests, for `robots.txt` and `llms.txt` only. Just one of the eight returned robots.txt. Six returned 403 and one timed out. A single small file request did not get through either.

Even on the three sites Ruescan marked provisional, the second run differed. One public-sector site scored 58 in the first run and 36 in the second. Two sites that were unreachable in the first run gave a result in the second, but their coverage stayed low, so their scores remained provisional.

## Is an unmeasured site good or bad?

Neither. Ruescan keeps unmeasured signals out of the score, as a rule. Scoring an unmeasured site would mean guessing. We do not guess.

There is a cost. Results shift toward sites that accept the request. If a general table shows an "average score", 27 sites are missing from that average. That is why we wrote it as a limit in the 30-site post too.

## Is your site turning scanners away, and how would you know?

Send two requests to your own site and compare the answers. From a terminal:

```
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://your-site.example/
curl -s -o /dev/null -w "%{http_code}\n" -A "MyChecker/1.0 (+https://your-site.example/bot)" https://your-site.example/
```

If both answers match, the identity header gets no special treatment. If they differ, your security layer treats one identity differently. Then check these:

1. Request `https://your-site.example/robots.txt` the same two ways. It should return 200 as `text/plain`.
2. In the server log, look at which status codes bot-identified requests get. To separate real bots from fake identities, follow [verify AI bots in server logs](https://ruesandora.com/writing/verify-ai-bots-server-logs).
3. If you use a CDN or firewall, learn its default behavior for AI bots: [is your CDN blocking AI bots](https://ruesandora.com/writing/is-cdn-blocking-ai-bots).
4. Compare the rules in your robots.txt against the [robots.txt for AI bots](https://ruesandora.com/writing/robots-txt-for-ai-bots) guide.

These two commands do not prove whether AI bots can enter your site. They only show whether a script and a browser identity get the same answer.

## Frequently asked questions

### Why could the sites not be measured?

Most answered the automated request with an error code such as 403. We do not know the cause (address, identity header, rate or another rule).

### Does an unmeasured site block AI bots?

No, not necessarily. The answer to a scanner can differ from the answer to real bots. We could not verify that in this measurement.

### Which sectors were unmeasured most often?

In the first list, local services (4 of 4 sites) and e-commerce (3 of 4). The sample is small and sectors were classified by hand, so do not draw a general rule.

### Is an unmeasured site given a score?

No. Unmeasured signals stay out of the score, and a site where everything is unmeasured gets no score.

## Sources

- G8 pilot measurement, 19 October 2026, 57 site tries (internal record, raw output kept in-house).
- [AI readiness of 30 Turkish sites](https://ruesandora.com/writing/turkey-websites-ai-readiness), the general results of the same measurement.

## Want to talk about this for your brand?

Write about your project and what you want to do. The message goes straight to Rues.

[Open the contact form](https://ruesandora.com/#iletisim)

[Email info@ruesandora.com](mailto:info@ruesandora.com) [Telegram ruesandora](https://t.me/ruesandora) [X @ruesandora0](https://x.com/ruesandora0) [Instagram @ruesandora0](https://www.instagram.com/ruesandora0) [YouTube @ruesandora0](https://www.youtube.com/@ruesandora0)

Page: https://ruesandora.com/writing/why-11-sites-were-unmeasurable