Why would an AI search provider have a long-term incentive to trust information that a brand deliberately keeps separate from its public-facing website?
This is a question I’ve been coming back to as more conversations about optimizing for AI search seem to involve creating content specifically for agents, rather than people.
The logic is understandable: AI systems need information, sometimes in formats or structures that aren’t particularly useful to a human visitor. If we can make that information easier for agents to consume, surely that’s a good thing?
Yes, but there is a significant difference between making information more accessible to AI and creating an alternative version of your website, or even your brand, that exists primarily to influence what AI systems say about you.
We’ve spent decades dealing with versions of this problem in SEO, and I’m not convinced we’re taking those lessons seriously enough.
How Can SEOs See This Coming?
Anyone who has worked in SEO for long enough will have seen search engines evolve in response to the means used to manipulate them.
Early SEO was full of practices designed to influence search engines rather than help users: hidden text, keyword stuffing, doorway pages, cloaking, and eventually increasingly sophisticated content and link manipulation.
Search engines didn’t develop rules against these practices because they disliked SEOs (mostly). They did so because allowing websites to falsify relevance and authority undermined the quality of their results.
Google’s history gives us a fairly clear picture of this evolution:
- 2011: Panda targeted low-quality content.
- 2012: Penguin tackled link spam.
- 2022: Google’s Helpful Content system sought to better reward content created to help people, rather than primarily attract search traffic.
- 2024: Google’s updated spam policies made the position on scaled content abuse clearer, irrespective of whether the content was created by AI or humans.
This is a very condensed version of things, and the reality was far more complicated, but the pattern is important.
As search engines became more sophisticated, so did the methods used to influence (spam) them. Search engines responded by investing more effort into deciding what information was trusted and rewarded.
Google still explicitly identifies cloaking as spam. I am not saying that cloaking’s definition is as simple (or east to detect) as that, but the parallels are too close to ignore.
So What Are We Building For AI?
We’re now seeing a collection of approaches designed to make information available to AI systems and agents.
The llms.txt proposal is one example, providing a structured way of directing language models towards useful website content. There are also agent-specific markdown pages, dedicated information endpoints, structured feeds and increasingly sophisticated methods of supplying information directly to agents.
Cloudflare’s Markdown for Agents is a good example of serving an alternative version of existing content. The information originates from the same page, but is presented in a format that’s potentially easier for agents to process.
That’s fundamentally different from creating new information that exists only for AI consumption.
Making the same information accessible in different formats isn’t the problem.
Making different information available to different audiences is where things become considerably more complicated.
I can understand why brands might be attracted to this as getting content onto a website has never been particularly easy. Content ownership, CMS limitations, design requirements, development resources, governance, approvals and the competing priorities of different teams all play a part.
If you can bypass all of that by publishing a text file, or hosting a collection of AI-specific content somewhere else, you have an appealing shortcut.
The Problem Of Trust And The Risk/Reward Here
There’s another important difference between traditional search and AI search: Google isn’t the only significant actor here. As I’ve explored previously, digging into ChatGPT’s search sources, there isn’t a single, consistent retrieval process underpinning every AI-generated answer as well.
Different systems, search providers, tools, and retrieval pipelines can all play a role in what information is discovered and ultimately used.
Some of these processes inherit established “search” infrastructure, including the systems those that evaluate the relevance and reliability of information. But beyond Google or Bing, other sources behave differently.
That doesn’t mean ChatGPT or other AI systems automatically trust everything they’re given. But there is a considerable difference between having mechanisms to identify reliable sources and having the experience that traditional search engines have dealing with people attempting to manipulate those mechanisms.
We also need to separate the ability to retrieve information from the decision to trust it. When I looked at how curious agents actually are, I saw how they use the least effort to retrieve, summarize, and move on. Just because the right answer was available doesn’t mean it would always be found and used.
If an agent encounters information created specifically to influence its answer:
- How should it establish whether that information is accurate, authoritative, or consistent with what the brand tells its actual customers?
- What happens when the machine-readable version says one thing, but the public website says something else?
- Who decides which is more trustworthy?
- And, perhaps most importantly, why should AI providers continue to accept information that is deliberately separated from what users can independently verify?
I don’t think we have answers to these questions yet.
Haven’t We Been Here Before?
I don’t think it is fair or truth to claim that every AI-specific content format is a form of cloaking, or that making information available to agents is manipulative. Structured data, XML sitemaps, APIs, and product feeds have been part of the web for years. There are entirely legitimate reasons to give machines information in different formats – in fact, I’d insist you do it!
But there’s a difference between helping machines understand your business and creating content just to influence AI output. Especially when that content isn’t represented anywhere people can readily access.
The incentive to manipulate is obvious, and whilst AI search isn’t traditional search, the incentives of AI search providers are opposed to this. They need to return useful, accurate answers and maintain user trust. Why would they want to encourage an ecosystem of increasingly fragmented, difficult-to-verify sources that exist primarily to influence their answers?
Perhaps these systems will become better at validating information against other sources. Maybe agent-specific information will be treated with greater scrutiny, or its influence limited without independent corroboration.
We don’t know exactly how this will evolve, but assuming that today’s emerging practices will remain acceptable, effective, or even useful in the long term seems worryingly optimistic or naive.
Getting The Information Customers Need Onto A Website Has Always Presented Challenges
We’re “solving” an old problem with a new shortcut, and if it is too good to be true, it may well be!
For years, content strategies have been constrained by what businesses want to publish and what their internal processes allow them to maintain. Often this creates a gap between the information customers want and the information brands make available.
Generative AI has made this more apparent and has been a significant multiplier here! People can now ask detailed, specific questions about products, services, properties, experiences, and practically anything else, expecting useful answers without having to navigate a website to piece the information together themselves. That creates a much greater demand for clear, accessible, and specific information.
If that information isn’t available, AI systems may have to look elsewhere or simply fail to answer the question correctly. So we do need to reconsider the information we’re publishing, how it’s structured and how different systems access it.
But why should the solution be to create another content estate? One that requires separate maintenance, governance, potentially another tech stack, and additional processes to ensure it remains consistent with the information customers can actually see?
Most struggle to keep one version of their website accurate and useful… I’m not sure creating a second, less visible one is the obvious solution.
Useful For AI, Useful For People
I don’t think the answer is to reject agent-specific formats or insist that AI systems should consume websites in exactly the same way as people. That would ignore the genuine differences in how these systems operate, and the opportunities they create.
Equally, assuming the solution is simply to create content specifically for AI is avoiding a problem we’ve been trying to solve for years.
The goal should be to make genuinely useful information available to both, from a consistent, authoritative source that can be maintained, validated, and, where appropriate, independently verified.
Information that’s useful only to AI creates a governance problem and raises questions of trust.
Information that’s useful only to people increasingly creates a visibility problem.
There is a sensible middle ground here, and it doesn’t require us to invent another version of the web. I’ve written previously about why Google’s reluctance to endorse an entirely separate set of AI optimization practices makes sense. There’s little incentive for search providers to legitimize another industry of techniques they will then need to police.
The more we encourage brands to create information specifically for agents, detached from the public-facing web, the more we risk recreating problems that search engines have spent decades trying to solve.
I’m not convinced that building a shadow web for AI is progress. It may just be another shortcut we’ll spend the next decade trying to undo.
More Resources:
- Google’s Mueller Says llms.txt Can’t Help LLMs Differentiate Sites
- A Little Clarity On SEO, GEO, And AEO
- Google Answers Question About SEO For AI Agents
This post was originally published on Chris Green SEO.
Featured Image: DC Studio/Shutterstock
