Menu

Research · June 2026

Should you block AI crawlers? A robots.txt study of top Dutch sites

Should you block AI crawlers? Blocking a training bot such as GPTBot has no effect on your AI visibility, but blocking a search index bot makes you uncitable in AI search. Of the top 500 .nl domains with a readable robots.txt, 20.4% block at least one AI crawler: GPTBot is blocked by 17.6%, OAI-SearchBot by 9.2%, and Googlebot by 0.7%. AI companies' training bots are blocked far more often than the search index bots that decide whether a site shows up in answers from ChatGPT, Perplexity, and Google. A robots.txt measurement taken on June 17, 2026.

426
.nl domains measured
19
AI crawlers checked per site
20.4%
block at least one of them

AI assistants do not pull their answers from a single source. One crawler trains the model; another builds the search index the assistant cites from in real time. Sites mostly block the first, while it is the second that decides whether you appear in AI answers.

The measurement

The key figures

Denominator: 426 readable robots.txt files from the top 500 .nl domains.

20.4%

block at least one AI crawler

6.6%

block only a training bot and leave the search index bots open

13.1%

block an AI search index bot and cannot be cited in AI search

13.4%

block Google-Extended but not Googlebot

0.7%

block Googlebot, the only robots.txt route out of Google and AI Overviews

5.9%

have an llms.txt file

How it works

Does blocking AI crawlers affect your AI visibility?

It depends on the kind of crawler. There are three kinds, and only two of the three affect whether you can be found in AI.

Training bot

Collects text to train AI models.

GPTBot, ClaudeBot, CCBot

No effect on your AI visibility

Search index bot

Builds the index an assistant cites from in real time.

OAI-SearchBot, PerplexityBot, Claude-SearchBot

Decides whether you get cited

Live-fetch bot

Fetches a page at the moment a user asks the question.

ChatGPT-User, Perplexity-User

Decides whether you get cited

The core finding

GPTBot vs OAI-SearchBot: which one do sites block?

Mostly GPTBot. Training bots are shut out most often. The search index bots, which decide whether you get cited, are left open far more often.

Blocked most often: training bots

GPTBot
17.6%
CCBot
16.7%
ClaudeBot
14.8%

Drive your visibility: search index bots

OAI-SearchBot
9.2%
Claude-SearchBot
8.5%
Googlebot
0.7%
36/75

Of the 75 sites that block GPTBot, at least 36 leave OAI-SearchBot open. So roughly half of them remain citable in ChatGPT's search-based answers.

By crawler

How many sites block GPTBot? 17.6%, 25 times as often as Googlebot

Training botSearch index botLive-fetch botDeprecatedMixed
GPTBot
17.6%
CCBot
16.7%
Amazonbot
15%
ClaudeBot
14.8%
Bytespider
14.6%
Google-Extended
14.1%
anthropic-ai
12.7%
ChatGPT-User
12.4%
PerplexityBot
12.4%
Applebot-Extended
12.2%
Claude-Web
12%
Meta-ExternalFetcher
10.1%
Meta-ExternalAgent
9.9%
Perplexity-User
9.4%
OAI-SearchBot
9.2%
Claude-SearchBot
8.5%
Claude-User
8.5%
Applebot
3.8%
Googlebot
0.7%

The training bots sit at the top, the search index bots below them. GPTBot (17.6%) versus OAI-SearchBot (9.2%) and Claude-SearchBot (8.5%): that contrast is the heart of this study.

The pattern

Sites that block one AI crawler block 10 on average

A site that keeps out one AI crawler rarely stops at one. Broad blocklists get copied wholesale rather than weighed bot by bot.

10 or more
51 sites
exactly 1
16 sites
5 to 9
13 sites
2 to 4
9 sites
10.0

AI crawlers blocked on average per blocking site, median 11

The clearest example is NU.nl and AD, both owned by the publisherDPG Media, with a nearly identical robots.txt down to the same DPG copyright block. A shared template, in practice.

The examples

From NOS to de Volkskrant: who blocks what

NOS is the Dutch public broadcaster; de Volkskrant is one of the country's national newspapers.

Training bot blocked, search bot open

Blocks GPTBot, leaves OAI-SearchBot open, 36 domains

They keep the training bot out but remain citable in ChatGPT search.

  • NOS
  • RTL
  • Naturalis
  • NPO Radio 1
  • BNR
  • Ticketmaster
  • Tripadvisor
  • eBay
  • Thomann
  • Vinted
  • ZorgkaartNederland

Not citable in AI search

Blocks an AI search index bot, 56 domains

News media and broadcasters are heavily represented here, often a deliberate stance on copyright.

  • NRC
  • de Volkskrant
  • De Telegraaf
  • Trouw
  • Het Parool
  • het Financieele Dagblad
  • De Correspondent
  • De Groene Amsterdammer
  • NU.nl
  • AD
  • Brabants Dagblad
  • BN DeStem
  • De Gelderlander
  • De Limburger
  • Leeuwarder Courant
  • Dagblad van het Noorden
  • Omrop Fryslân
  • Omroep West
  • RTV Noord
  • Overheid.nl
  • DigiD
  • Gemeente Amsterdam

Google-Extended blocked, Googlebot open

Blocks Google-Extended but not Googlebot, 57 domains

As a result they still show up in AI Overviews, which draw on the regular search index.

  • NRC
  • De Telegraaf
  • Delpher
  • Amazon.nl
  • Gemeente Amsterdam
  • Tripadvisor
  • Vinted
  • ZorgkaartNederland

A selection from the measurement. The full list is in the research document (in Dutch).

Manual check

Four news brands up close

NOS, NU.nl, AD and RTL could not be reached in the automated crawl, so we fetched and checked them by hand. They are included in the figures using the same logic and show the pattern sharply: all four block the training bot, all four leave Googlebot open, and yet they are not consistent with one another.

BrandGPTBotOAI-SearchBotPerplexityBotClaude-SearchBotGoogle-ExtendedGooglebot
NOSblockedopenblockedopenblockedopen
NU.nlblockedblockedblockedblockedblockedopen
ADblockedblockedblockedblockedblockedopen
RTLblockedopenblockedopenblockedopen

NU.nl and AD, both owned by DPG Media, also block ChatGPT's search bot, while NOS and RTL leave it open and therefore remain citable. NU.nl and AD also use a bare Claude rule that, under prefix matching, catches the current Claude-SearchBot as well; NOS and RTL leave it open.

A closer look

Three more things that stand out

The fight is over AI training, not the search index

Blocking Google-Extended does not take you out of AI Overviews. Those draw on Googlebot's search index, which is open almost everywhere. The fight is about training, while the access that matters for AI Overviews goes untouched.

Google-Extended
14.1%
Googlebot
0.7%

Outdated blocklists

The retired Anthropic user-agent names are blocked more often than the current Claude crawlers. A rule like that no longer does anything, while the current Claude crawlers stay open.

anthropic-ai
12.7%
Claude-Web
12%
Claude-SearchBot
8.5%

The new tools sit unused

The mechanisms built specifically for steering AI are barely used. In practice nearly everything runs on robots.txt, with all the bluntness that comes with it.

5.9%

have an llms.txt file

0.0%

use the nosnippet directive

For you

Should you block GPTBot or the AI search bots?

  1. 1

    Know the difference between training bots, search index bots and live-fetch bots. Only the last two decide whether you appear in AI answers.

  2. 2

    Check your robots.txt for OAI-SearchBot, PerplexityBot and Claude-SearchBot. If you block them, you cannot be cited in AI search. Make sure that is what you want.

  3. 3

    Know that blocking Google-Extended does not take you out of AI Overviews. The only robots.txt route out is blocking Googlebot, which costs you your organic traffic.

Run the free AI visibility check (in Dutch)Check your robots.txt for AI bots (in Dutch)

Methodology

Download the full research document

All figures, the per-domain data and the complete crawler classification are in the full research document (PDF), which is written in Dutch. The raw dataset is available on request to researchers and journalists.

Download the study (PDF, in Dutch)
Method and limitations

We examined the 500 most popular .nl domains according to the Tranco ranking, a research-grade list of the most visited domains. For each domain we fetched the public robots.txt and determined, per AI crawler, whether the site root is fully blocked (Disallow: /), with correct group selection, prefix matching of user agents, and the rule that an Allow wins when match lengths are equal. No AI model is involved; the measurement is deterministic and repeatable.

Of the 500 domains, 460 were reachable in the automated crawl and 40 were not. Of those 460, 422 had a readable robots.txt; the remaining 38 blocked access to that file with a 403 or 401 and were excluded rather than counted as non-blocking. Four major news brands (NOS, NU.nl, AD and RTL) were among the unreachable domains because of the network configuration of the measuring machine; their public robots.txt was fetched directly, scored with exactly the same logic and added, which brings the denominator to 426. The llms.txt and nosnippet figures come from the full measurement round in which llms.txt and the homepage were also fetched.

We report what a robots.txt declares and what that means according to the documentation of OpenAI, Google, Perplexity, Anthropic, Apple, and Meta. We do not infer intent, because the file cannot tell you why a site blocks a crawler. To assign rules to crawlers we use prefix matching, as Google does: a generic rule for Claude, for example, therefore also applies to the specific Claude crawlers such as Claude-SearchBot. A measurement also shows declared policy, not whether a bot complies with it, and it is a snapshot taken on a single reference date.

Author: Matt Timmermans, Timmermans Media. Measured on June 17, 2026.

Cite this study

Source: Timmermans Media, AI Crawler Study Netherlands, June 2026, timmermansmedia.com/research/ai-crawlers-robots-txt/, measured June 17, 2026.

This page is the English edition of a study first published in Dutch. Read the original Dutch version.

Glossary

Dutch terms on this page

The study used Dutch questions and Dutch websites. These are the Dutch names and words that come up.

AD
Dutch national daily newspaper (Algemeen Dagblad), owned by DPG Media.
BN DeStem
Dutch regional daily newspaper for western North Brabant.
BNR Nieuwsradio
Dutch national business and news radio station, comparable to Bloomberg Radio.
Brabants Dagblad
Dutch regional daily newspaper for the province of North Brabant.
Dagblad van het Noorden
Dutch regional daily newspaper for the provinces of Groningen and Drenthe.
De Correspondent
Dutch member-funded online platform for in-depth journalism.
De Gelderlander
Dutch regional daily newspaper for the province of Gelderland.
De Groene Amsterdammer
Dutch weekly magazine about politics, culture, and society.
De Limburger
Dutch regional daily newspaper for the province of Limburg.
De Telegraaf
Dutch national daily newspaper with the country's largest print circulation.
de Volkskrant
Dutch national daily newspaper with a progressive editorial line.
Delpher
Dutch Royal Library's digital archive of historical newspapers, books, and magazines.
DigiD
Dutch government's digital login that citizens use for public services online.
DPG Media
Belgian media company that owns AD, NU.nl, and many Dutch regional newspapers.
Gemeente Amsterdam
City government of Amsterdam; gemeente means “municipality.”
het Financieele Dagblad
Dutch financial daily newspaper, comparable to The Wall Street Journal.
Het Parool
Dutch daily newspaper based in Amsterdam.
Leeuwarder Courant
Dutch regional daily newspaper for the province of Friesland.
Naturalis
Dutch national natural history museum and research institute in Leiden.
NOS
Dutch public broadcaster's national news service, comparable to NPR or PBS.
NPO Radio 1
Dutch public radio station for news and current affairs.
NRC
Dutch national daily newspaper known for in-depth news and analysis.
NU.nl
Dutch online news site, one of the most visited in the country.
Omroep West
Dutch regional public broadcaster for the area around The Hague and Leiden.
Omrop Fryslân
Dutch regional public broadcaster for Friesland, partly in the Frisian language.
Overheid.nl
Dutch government portal for laws, regulations, and official publications.
RTL
Dutch commercial television broadcaster and news brand (RTL Nederland).
RTV Noord
Dutch regional public broadcaster for the province of Groningen.
Trouw
Dutch national daily newspaper with Protestant roots.
Zorgkaart Nederland
Dutch review site where patients rate doctors, hospitals, and other care providers.
Astronaut floating in space as a visual for the Timmermans Media call to action

Which crawlers does your site let in?

Get found in the AI era

We make sure you let the right crawlers in and get cited consistently in ChatGPT, Claude, Gemini and Google AI Overviews.

Book a free consultation