B-TIER news
AI search citations — why your chatbot cites the 'unlocked web'
100k ft — What is this?
AI search citations — why your chatbot cites the 'unlocked web'
50k ft — Why care?
High-credibility sites (Consumer Reports, ~60% of top news) block GPTBot, Google's AI crawler and Common Crawl, so AI search reads only what's left: AI-written pages and self-ranking listicles. One in six cited sources is AI-written and models often do post-hoc citation, writing the answer first and finding sources after.
10k ft — Key takeaways
• Click every AI citation and check the answer is actually on that page
• Quality publishers disallow AI crawlers in robots.txt, so AI search skews to low-quality sources
• Treat listicles cited by AI as possibly brand self-ranking
5k ft — Tools & techniques
ChatGPT, Claude
post-hoc citation
1k ft — Deep-dive / try it
just so we all have all the information
Transcript excerpt
I've logged every source that my AI has been given me since June, not the answers, but the sources or the citations and this really weird thing has been happening. I recognize almost none of them. Why I'm doing this the past five years, I've been writing a newsletter on tech and right now I'm doing a research project on AI and shopping. So I'm looking at a lot of data right now about how AI is recommending products. So I went looking for why this is happening and I looked at consumer reports. They've been testing products since 1936. But if you read their robots file, GPT bot disallow Google's AI crawler disallow common crawl, which feeds a bunch of these training sets disallow. They just don't want their testing script. And that's happening everywhere. So 60 percent of high credibility news sites now block AI crawlers. And the reason is this crazy stat, which is that traffic from bots and crawlers has now greater than human traffic on the web. So the sites with something to lose started to put up no entry signs, basically. So when you search something on cloud or chat with browsing on, it's not necessarily reading the Internet, it's reading a part of the Internet that still lets those crawlers in. Let's call this the unlocked web. And here's what the unlocked web is actually made up of. Now, researchers checked twenty six thousand sources that AI search engines had cited and one in six was written by another AI. There's now this entire supply side building pages…