Robby Stein, Vice President of Product for Google Search, recently sat down for an interview where he answered questions about how Google’s AI Mode handles quality, how Google evaluates helpfulness, and how it leverages its experience with search to identify which content is helpful, including metrics like clicks. He also outlined five quality SEO-related factors used for AI Mode.
How Google Controls Hallucinations
Stein answered a question about hallucinations, where an AI lies in its answers. He said that the quality systems within AI Mode are based on everything Google has learned about quality from 25 years of experience with classic search. The systems that determine what links to show and whether content is good are encoded within the model and are based on Google’s experience with classic search.
The interviewer asked:
“These models are non-deterministic and they hallucinate occasionally… how do you protect against that? How do you make sure the core experience of searching on Google remains consistent and high quality?”
Robby Stein answered:
“Yeah, I mean, the good news is this is not new. While AI and generative AI in this way is frontier, thinking about quality systems for information is something that’s been happening for 20, 25 years.
And so all of these AI systems are built on top of those. There’s an incredibly rigorous approach to understanding, for a given question, is this good information? Are these the right links? Are these the right things that a user would value?
What’s all the signals and information that are available to know what the best things are to show someone. That’s all encoded in the model and how the model’s reasoning and using Google search as a tool to find you information.
So it’s building on that history. It’s not starting from scratch because it’s able to say, oh, okay, Robbie wants to go on this trip and is looking up cool restaurants in some neighborhood.
What are the things that people who are doing that have been relying on on Google for all these years? We kind of know what those resources are we can show you right there. And so I think that helps a lot.
And then obviously the models, now that you release the constraint on layout, obviously the models over time have also become just better at instruction following as well. And so you can actually just define, hey, here are my primitives, here are my design guidelines. Don’t do this, do this.
And of course it makes mistakes at times, but I think just the quality of the model has gotten so strong that those are much less likely to happen now.”
Stein’s explanation makes clear that AI Mode is encoded with everything learned from Google’s classic search systems rather than a rebuild from scratch or a break from them. The risk of hallucinations is managed by grounding AI answers in the same relevance, trust, and usefulness signals that have underpin classic search for decades. Those signals continue to determine which sources are considered reliable and which information users have historically found valuable. Accuracy in AI search follows from that continuity, with model reasoning guided by longstanding search quality signals rather than operating independently of them.
How Google Evaluates Helpfulness In AI Mode
The next question is about the quality signals that Google uses within AI Mode. Robby Stein’s answer explains that the way AI Mode determines quality is very much the same as with classic search.
The interviewer asked:
“And Robbie, as search is evolving, as the results are changing and really, again, becoming dynamic, what signals are you looking at to know that the user is not only getting what they want, but that is the best experience possible for their search?”
Stein answered:
“Yeah, there’s a whole battery of things. I mean, we look at, like we really study helpfulness and if people find information helpful.
And you do that through evaluating the content kind of offline with real people. You do that online by looking at the actual responses themselves.
And are people giving us thumbs up and thumbs downs?
Are they appreciating the information that’s coming?
And then you kind of like, you know, are they using it more? Are they coming back? Are they voting with their feet because it’s valuable to you.
And so I think you kind of triangulate, any one of those things can lead you astray.
There’s lots of ways that, interestingly, in many products, if the product’s not working, you may also cause you to use it more.
In search, it’s an interesting thing.
We have a very specific metric that manages people trying to use it again and again for the same thing.
We know that’s a bad thing because it means that they can’t find it.
You got to be really careful.
I think that’s how we’re building on what we’ve learned in search, that we really feel good that the things that we’re shipping are being found useful by people.”
Stein’s answer shows that AI Mode evaluates success using the same core signals used for search quality, even as the interface becomes more dynamic. Usefulness is not inferred from a single engagement signal but from a combination of human evaluation, explicit feedback, and behavioral patterns over time.
Importantly, Stein notes that just because people use it a lot, presumably in a single session, that the increased usage alone is not treated as success, since repeated attempts to answer the same query indicate failure rather than satisfaction. The takeaway is that AI Mode’s success is judged by whether users are satisfied, and that it uses quality signals designed to detect friction and confusion as much as positive engagement. This carries over continuity from classic search rather than redefining what usefulness means.
Five Quality Signals For AI Search
Lastly, Stein answers a question about the ranking of AI generated content and if SEO best practices still help for ranking in AI. Stein’s answer includes five factors that are used for determining if a website meets their quality and helpfulness standards.
Stein answered:
“The core mechanic is the model takes your question and reasons about it, tries to understand what you’re trying to get out of this.
It then generates a fan-out of potentially dozens of queries that are being Googled under the hood. That’s approximating what information people have found helpful for those questions.
There’s a very strong association to the quality work we’ve done over 25 years.
Is this piece of content about this topic?
Has someone found it helpful for the given question?
That allows us to surface a broader diversity of content than traditional Search, because it’s doing research for you under the hood.
The short of it is the same things apply.
- Is your content directly answering the user’s question?
- Is it high quality?
- Does it load quickly?
- Is it original?
- Does it cite sources?
If people click on it, value it, and come back to it, that content will rank for a given question and it will rank in the AI world as well.”
Watch the interview starting about the one hour and twenty three minute mark:
