sector,site,domain,result,checked_on,GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,Meta-ExternalAgent,CCBot,Bytespider,Amazonbot,OAI-SearchBot,ChatGPT-User,Claude-SearchBot,Claude-User,PerplexityBot,Perplexity-User,DuckAssistBot,Googlebot,Bingbot,Applebot
News & publishing,BBC,bbc.co.uk,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,blocked,blocked,blocked,allowed,allowed,allowed
News & publishing,The Guardian,theguardian.com,robots.txt read,2026-11-11,allowed,blocked,allowed,blocked,blocked,blocked,blocked,blocked,allowed,allowed,blocked,blocked,blocked,allowed,blocked,allowed,allowed,allowed
News & publishing,The Times,thetimes.com,robots.txt read,2026-11-11,partial,blocked,blocked,blocked,blocked,blocked,blocked,blocked,partial,partial,blocked,blocked,blocked,blocked,blocked,partial,partial,partial
News & publishing,The Telegraph,telegraph.co.uk,robots.txt read,2026-11-11,blocked,blocked,partial,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed
News & publishing,Financial Times,ft.com,robots.txt read,2026-11-11,allowed,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed,allowed,allowed,blocked,blocked,allowed,allowed,allowed,allowed
News & publishing,The Independent,independent.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,Daily Mail,dailymail.co.uk,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed
News & publishing,The Sun,thesun.co.uk,robots.txt read,2026-11-11,partial,blocked,blocked,blocked,blocked,blocked,blocked,blocked,partial,partial,blocked,blocked,blocked,blocked,blocked,partial,partial,partial
News & publishing,Daily Mirror,mirror.co.uk,robots.txt read,2026-11-11,blocked,blocked,allowed,blocked,blocked,blocked,allowed,allowed,blocked,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed
News & publishing,Daily Express,express.co.uk,robots.txt read,2026-11-11,blocked,blocked,allowed,blocked,allowed,blocked,allowed,allowed,blocked,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed
News & publishing,Evening Standard,standard.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,Metro,metro.co.uk,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed
News & publishing,The i Paper,inews.co.uk,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed
News & publishing,Sky News,news.sky.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
News & publishing,ITV,itv.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
News & publishing,Channel 4,channel4.com,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,blocked,blocked,allowed,allowed,blocked,allowed,blocked,allowed,allowed,allowed
News & publishing,GB News,gbnews.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,LBC,lbc.co.uk,robots.txt read,2026-11-11,partial,blocked,blocked,blocked,allowed,allowed,blocked,allowed,allowed,partial,allowed,allowed,blocked,allowed,allowed,allowed,allowed,blocked
News & publishing,HuffPost UK,huffingtonpost.co.uk,robots.txt read,2026-11-11,blocked,blocked,allowed,allowed,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed
News & publishing,The Economist,economist.com,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,allowed,blocked,blocked,allowed,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed,allowed
News & publishing,New Statesman,newstatesman.com,robots.txt read,2026-11-11,blocked,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,The Spectator,spectator.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,City AM,cityam.com,robots.txt read,2026-11-11,blocked,blocked,partial,blocked,blocked,blocked,partial,blocked,partial,partial,partial,partial,partial,blocked,blocked,partial,partial,partial
News & publishing,Private Eye,private-eye.co.uk,no robots.txt (everything allowed),2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,Press Gazette,pressgazette.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,Manchester Evening News,manchestereveningnews.co.uk,robots.txt read,2026-11-11,blocked,blocked,allowed,blocked,blocked,blocked,allowed,allowed,blocked,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed
News & publishing,Liverpool Echo,liverpoolecho.co.uk,robots.txt read,2026-11-11,blocked,blocked,allowed,blocked,blocked,blocked,allowed,allowed,blocked,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed
News & publishing,Birmingham Mail,birminghammail.co.uk,robots.txt read,2026-11-11,blocked,blocked,allowed,blocked,blocked,blocked,allowed,allowed,blocked,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed
News & publishing,WalesOnline,walesonline.co.uk,robots.txt read,2026-11-11,blocked,blocked,allowed,blocked,blocked,blocked,allowed,allowed,blocked,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed
News & publishing,The Scotsman,scotsman.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,The Herald,heraldscotland.com,robots.txt read,2026-11-11,blocked,blocked,allowed,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,blocked,allowed,allowed,blocked
News & publishing,Daily Record,dailyrecord.co.uk,robots.txt read,2026-11-11,blocked,blocked,allowed,blocked,blocked,blocked,allowed,allowed,blocked,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed
News & publishing,Belfast Telegraph,belfasttelegraph.co.uk,robots.txt read,2026-11-11,blocked,blocked,blocked,allowed,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,blocked,allowed
News & publishing,Yorkshire Post,yorkshirepost.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
News & publishing,LADbible,ladbible.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,partial,allowed,allowed
News & publishing,UNILAD,unilad.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,partial,allowed,allowed
News & publishing,The Register,theregister.com,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed,allowed,allowed,blocked,blocked,allowed,allowed,allowed
News & publishing,TechRadar,techradar.com,robots.txt read,2026-11-11,allowed,allowed,allowed,blocked,blocked,allowed,blocked,blocked,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,blocked
News & publishing,Which?,which.co.uk,robots.txt read,2026-11-11,partial,partial,partial,allowed,allowed,allowed,blocked,partial,allowed,partial,partial,partial,partial,partial,allowed,allowed,allowed,allowed
News & publishing,Campaign,campaignlive.co.uk,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,Amazon UK,amazon.co.uk,robots.txt read,2026-11-11,blocked,blocked,blocked,allowed,blocked,blocked,blocked,allowed,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed
Retail & e-commerce,Argos,argos.co.uk,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,Tesco,tesco.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,Sainsbury's,sainsburys.co.uk,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,Asda,asda.com,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,John Lewis,johnlewis.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,Marks & Spencer,marksandspencer.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,Next,next.co.uk,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,Boots,boots.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,Currys,currys.co.uk,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,AO,ao.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,ASOS,asos.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,boohoo,boohoo.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,Very,very.co.uk,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,Screwfix,screwfix.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,Wickes,wickes.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,B&Q,diy.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,Ocado,ocado.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Retail & e-commerce,Superdrug,superdrug.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Retail & e-commerce,JD Sports,jdsports.co.uk,no robots.txt (everything allowed),2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,GOV.UK,gov.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,partial,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,NHS,nhs.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,partial,allowed,allowed
Finance & public sector,UK Parliament,parliament.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Office for National Statistics,ons.gov.uk,no robots.txt (everything allowed),2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Met Office,metoffice.gov.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,The National Archives,nationalarchives.gov.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Police.uk,police.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Bank of England,bankofengland.co.uk,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Finance & public sector,Financial Conduct Authority,fca.org.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Barclays,barclays.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,HSBC UK,hsbc.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Lloyds Bank,lloydsbank.com,"served a web page, no rules",2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,NatWest,natwest.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Nationwide,nationwide.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Monzo,monzo.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,MoneySavingExpert,moneysavingexpert.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Compare the Market,comparethemarket.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Finance & public sector,MoneySuperMarket,moneysupermarket.com,robots.txt read,2026-11-11,partial,partial,allowed,allowed,allowed,partial,allowed,partial,partial,partial,partial,partial,partial,partial,allowed,allowed,allowed,allowed
Finance & public sector,Confused.com,confused.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Finance & public sector,Aviva,aviva.co.uk,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Tech & travel,BT,bt.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Sky,sky.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Virgin Media,virginmedia.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Vodafone UK,vodafone.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,partial,allowed
Tech & travel,O2,o2.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,EE,ee.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Rightmove,rightmove.co.uk,robots.txt read,2026-11-11,blocked,allowed,allowed,allowed,allowed,blocked,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Zoopla,zoopla.co.uk,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Tech & travel,Auto Trader,autotrader.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,partial,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Gumtree,gumtree.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Trainline,thetrainline.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,National Rail,nationalrail.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,British Airways,britishairways.com,"served a web page, no rules",2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,easyJet,easyjet.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Tech & travel,Jet2,jet2.com,could not be read,2026-11-11,,,,,,,,,,,,,,,,,,
Tech & travel,Tripadvisor UK,tripadvisor.co.uk,robots.txt read,2026-11-11,blocked,blocked,blocked,blocked,blocked,blocked,blocked,blocked,partial,partial,allowed,allowed,partial,allowed,allowed,allowed,partial,partial
Tech & travel,Just Eat,just-eat.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Deliveroo,deliveroo.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Ticketmaster UK,ticketmaster.co.uk,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed
Tech & travel,Channel 5,channel5.com,robots.txt read,2026-11-11,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed,allowed