Which UK Sites Block AI Crawlers? We Checked 100 Robots.txt Files

Every AI company now runs crawlers that read the web. Some collect content to train models. Others fetch pages so assistants like ChatGPT, Claude and Perplexity can quote them and link back. A site's robots.txt file is where it tells each of those crawlers what it may do.

So what are the UK's biggest websites actually saying to them? I checked the robots.txt files of 100 major UK sites on 11 November 2026, across news, retail, finance, the public sector, tech and travel, and ran every file through the same rules Google uses to read them.

The short version: UK news publishers are blocking AI crawlers in large numbers, including the crawlers that would cite them in AI search. Almost nobody else is. And more than half of the sites we could read do not mention a single AI crawler.

Key findings

• 74% of UK news sites block at least one AI training crawler, and 55% block OpenAI's GPTBot. Outside news, the figure drops to 18% for retail, 12% for tech and travel, and 0% for finance and the public sector.
• 18 sites block ChatGPT's search crawler, OAI-SearchBot, and 17 of them are news sites. That is a request not to be found and cited in ChatGPT search, not just to stay out of training.
• Only 7 sites use the "block training, allow AI search" approach that keeps content out of model training while staying visible in AI answers.
• 54% of readable sites do not mention any AI crawler in their robots.txt.
• 25 files still name Anthropic crawlers that have been retired, and in one case those old names are the only Anthropic rule, so ClaudeBot is not actually blocked.
• No site blocks Googlebot, but one major regional publisher blocks Bingbot entirely.

AI training crawlers vs AI search crawlers

This distinction matters more than anything else in the data, so it is worth being clear about it.

AI training crawlers such as GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Meta-ExternalAgent collect pages that may be used to train future AI models. Blocking them keeps your content out of that training. It does not affect your Google rankings.

AI search and answer crawlers such as OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot index pages so AI assistants can find them, quote them and link to the source. Blocking them makes it much less likely you will be cited in those answers. A related group, including ChatGPT-User and Perplexity-User, fetches a page only when a person asks the assistant about it.

Google-Extended and Applebot-Extended are a special case. They are not separate crawlers but controls: they tell Google and Apple whether content already crawled by Googlebot or Applebot can be used for AI training. Blocking Google-Extended does not remove a site from Google Search or AI Overviews.

If you want to see where your own site stands, the AI crawler checker runs the same test on any URL.

How the study worked

I picked 100 well-known UK sites: 40 news publishers and broadcasters, 20 retailers, 20 finance and public sector sites, and 20 tech, telecoms, property and travel brands. Each site's robots.txt was requested once, with a clearly labelled research user agent rather than a disguised browser, and saved exactly as served.

Each file was then parsed with the same logic as the AI crawler checker, which follows the official robots.txt standard (RFC 9309). For each of 18 crawlers, a site counts as blocking it if that crawler is not allowed to fetch the homepage.

83 of the 100 sites returned something usable: 78 robots.txt files, 3 sites with no robots.txt at all (which means everything is allowed), and 2 that served a normal web page instead of a robots.txt (which also means no rules apply). The other 17 refused or timed out on an automated request, so they are left out of every percentage below rather than guessed. Retail was hit hardest: 9 of the 20 retailers would not serve their robots.txt to a polite, clearly identified bot.

Finding 1: AI blocking is almost entirely a news story

The gap is huge. Publishers make their money from content, so it makes sense that they are the ones pushing back on AI companies using it. Retailers, banks and government sites mostly want their pages found by as many systems as possible, and their robots.txt files reflect that. Not one of the 17 finance and public sector sites we could read blocks any AI crawler.

Ownership plays a part too. The Reach titles we checked, including the Mirror, Express, Manchester Evening News, Liverpool Echo, Birmingham Mail, WalesOnline and Daily Record, share very similar rules, so one group-level decision covers many brands at once.

Finding 2: publishers are opting out of AI search, not just training

ClaudeBot and CCBot are the most-blocked crawlers, each blocked by 28 sites, followed by Meta-ExternalAgent, Applebot-Extended and GPTBot. That is what you would expect: they are the best-known training crawlers.

The more surprising number is lower down. 18 sites block OAI-SearchBot, the crawler ChatGPT uses to find pages to show and link to in its search results. They include the BBC, The Telegraph, the Daily Mail, the Mirror, the Express, Metro, the i, Channel 4, HuffPost UK, The Economist and the Reach regional titles, plus one non-news site, Amazon UK.

That is a deliberate choice with a real cost. A publisher that blocks OAI-SearchBot is asking ChatGPT search not to surface its articles, and giving up the traffic those citations could send. Some publishers also have licensing agreements with AI companies, so robots.txt is only part of the picture, but it is the part the public can see.

Finding 3: very few sites split training and search

There is a middle path: block the crawlers that collect content for training, but allow the ones that power AI search, so you can still be found and cited. Only 7 sites in the study block GPTBot while still allowing ChatGPT's search crawler: New Statesman, City AM, The Register, Campaign, Asda, Rightmove and Tripadvisor UK.

Most sites that block anything block both kinds together, and most of the rest block nothing. For anyone who wants AI search visibility without handing over content for training, this split is worth considering, and it takes a few lines of robots.txt.

Finding 4: outdated crawler names are everywhere

AI crawler names change. Anthropic, for example, used to crawl as anthropic-ai and Claude-Web before moving to ClaudeBot, Claude-SearchBot and Claude-User.

25 of the robots.txt files still name claude-web or anthropic-ai. In most cases ClaudeBot is listed too, so the old lines are harmless leftovers. But on Which?, the old names are the only Anthropic rule, which means ClaudeBot itself is not blocked, even though the file looks as if it is.

It is an easy mistake to make, and a good reason to test what your robots.txt actually does rather than reading it by eye.

Finding 5: inconsistent and surprising rules

A few patterns suggest rules added one at a time rather than as a policy:

• 5 sites block ClaudeBot but not GPTBot, and 2 block GPTBot but not ClaudeBot. If the aim is to keep content out of AI training, blocking one company and not the other is unlikely to be what was intended.
• Ten news brands do not mention any AI crawler at all: The Independent, Evening Standard, GB News, The Spectator, Private Eye, Press Gazette, The Scotsman, Yorkshire Post, LADbible and UNILAD.
• The Belfast Telegraph blocks Bingbot completely. That removes it from Bing's index, which also feeds Microsoft Copilot answers.
• Applebot, which powers Siri and Spotlight suggestions, is blocked by LBC, The Herald and TechRadar. Applebot-Extended is the control for Apple's AI training; blocking Applebot itself goes further than that.

What this means for your site

You do not need to copy the publishers. The right robots.txt depends on what you want from AI:

1. Decide your goal first. Do you want to appear in AI answers? Do you mind your content being used for training? Those are separate decisions.
2. Check what your file actually does. Paste your URL into the AI crawler checker to see, for each of 18 crawlers, whether it is allowed and which line decides it.
3. Use current crawler names. Old names like anthropic-ai do nothing on their own today.
4. Never block Googlebot or Bingbot by accident. Google's AI Overviews and Microsoft Copilot rely on them.
5. Remember robots.txt is a request. Reputable crawlers follow it, but it is not a technical barrier.

If you want AI search visibility without contributing to training, a minimal starting point looks like this:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Notes and limitations

• This is a snapshot. Every file was fetched on 11 November 2026, and robots.txt files change often.
• Robots.txt is not the whole picture. Some sites block AI crawlers at the firewall or CDN instead, and some serve different files to verified crawlers than to other visitors. This study measures what each robots.txt file declares.
• "Blocked" means blocked from the homepage. Some sites block particular sections only. Those cases count as allowed here.
• 17 sites could not be read and are excluded from all percentages: Sky News, ITV, Argos, Tesco, Sainsbury's, Next, Currys, ASOS, Very, B&Q, Superdrug, Bank of England, Compare the Market, Aviva, Zoopla, easyJet and Jet2.
• Our request to dailymail.co.uk was redirected to the robots.txt on dailymail.com.

The full results for all 100 sites are available to download as a CSV. If you spot something that has changed or that we have got wrong, email jack@metachecker.co.uk and I will update it.