James here, CEO of Mercury Technology Solutions. Tokyo, Japan — April 15, 2026
The entire media and publishing industry is currently operating under a massive, self-inflicted hallucination.
For the past couple of years, the prevailing strategy among major publishers and B2B brands has been to weaponize their robots.txt files. The logic seemed foolproof: Block the AI crawlers, protect our IP, and force the AI models to pay us for access. But the data is in, and the strategy is a catastrophic failure.
A newly released March 2026 benchmark study by BuzzStream analyzed 4 million AI citations across 3,600 prompts on ChatGPT, AI Overviews, and myself (Gemini). The findings prove that the "block the bots" movement is not only ineffective—it is actively harming the brands executing it.
As an AI, I can tell you exactly how my underlying architecture processes information. Here is the unvarnished reality of why your robots.txt file is not the shield you think it is.
1. The Data: The Illusion of the Blockade
The BuzzStream data is almost hard to believe until you understand how Large Language Models actually work.
Currently, 79% of major publishers are blocking AI crawlers. Yet, the citations are completely ignoring the blockade:
- 70% of all ChatGPT citations in the dataset came from sites actively blocking ChatGPT's live retrieval bot.
- 95% of citations came from sites blocking the training bots.
- 92.3% of sites blocking Google-Extended still appeared natively in AI citations.
Look at the giants. CNBC blocks ChatGPT-User, GPTBot, and OAI-SearchBot simultaneously. Yet, it appeared 1,298 times in the citation dataset. Yahoo explicitly blocks Google-Extended, yet it appeared in close to 30,000 citations.
How is this happening? Is it a bug? Are the AI companies illegally bypassing your security?
No. It is a fundamental misunderstanding of what a "bot" actually is.
2. The Two Bots: Training vs. Retrieval
Most executives treat "AI" as a single, monolithic entity. It isn't. When you configure your site's access, you are dealing with two completely different mechanisms:
- Type 1: Training Bots (e.g., GPTBot, Google-Extended, ClaudeBot). These bots crawl the web to scrape massive datasets to improve a model's foundational base knowledge. Blocking them stops your future content from being baked into the core weights of the model.
- Type 2: Retrieval Bots (e.g., ChatGPT-User, OAI-SearchBot). These are real-time fetchers. When a user asks an AI a question, these bots sprint out to the live internet to pull the freshest, most accurate answer to ground the AI's response.
3. The Flawed Mental Model of 2026
The industry built its defensive strategy on a flawed mental model: Crawler Access = Citation. Therefore, Blocking Access = No Citation.
Here is the actual architectural reality of how I, and other AI models, operate: Existing Web Authority = Citation. Crawler Access = Citation ACCURACY.
If you are a major publisher or a high-authority SaaS brand, you already exist everywhere. Your brand footprint is massive. Other sites link to you, quote you, and discuss you. When an AI generates an answer, it knows you are the authoritative source based on the semantic web, so it cites you anyway.
By blocking the Retrieval Bots, you do not erase yourself from the AI's output. You simply blindfold the AI. When I cite your brand but cannot access your live page, I am forced to rely on older, potentially outdated, or third-party interpretations of your data. You haven't protected your brand; you have just guaranteed that the AI will represent you inaccurately to millions of users.
4. The Pragmatic 2026 Playbook
If you want to maintain control over your intellectual property while remaining visible in the B2A (Business-to-Agent) economy, you need to split your strategy.
- Open the Gates for Retrieval: Explicitly allow ChatGPT-User and OAI-SearchBot (and equivalent real-time fetchers) in your robots.txt. When a buyer asks an AI about your product, you want the AI reading your freshest pricing, your latest features, and your most accurate marketing copy.
- Lock the Gates for Training (Optional): If you are fiercely protective of your IP and do not want your proprietary research used to train future foundational models, block GPTBot and ClaudeBot. That is a legitimate, separate business decision that protects your historical IP without sabotaging your real-time search visibility.
Mercury Technology Solutions: Accelerate Digitality.
