Google CEO Sundar Pichai said Google has spent 25 years learning how to measure user satisfaction in Search, including engagement, sessions, return behavior, and bounce-backs, saying that those metrics help Google improve AI Search. When asked about the concept of Google Search traffic dwindling to zero, the Google Zero scenario, Pichai insisted that the reality is quite different.
The interview gives the impression that Pichai downplays Google Zero concerns in a way that may frustrate publishers and SEOs who are worried about declining search referrals.
Google Tracks User Satisfaction Metrics In AI
One of the interesting takeaways from this interview is that Google’s CEO Pichai confirmed that Google is using user satisfaction metrics to gauge how well AI search is performing.
His confirmation was in a part of the interview where The Verge’s Nilay Patel challenged Sundar Pichai about an AI Overview result for “best Chromebook” in which AI Overviews gave one recommendation, Reddit surfaced another, and The New York Times offered a completely different recommendation further down the page. Patel used the example to ask Pichai for his opinion of the user experience in AI Mode and AI Search.
Patel asked:
“And I’m just curious how you think that experience for consumers is today in AI mode and where you think it should go?”
Pichai explained that AI Mode organizes information and provides context, including links to source content. But he also explained that the AI answers also provide an opinion, which is what Patel was referring to.
Regarding search quality, he said that Google has over 25 years experience tracking user experience and correlating that to improving search (the product). Interestingly, he said it’s not a short-term improvement, it’s a long term improvement.
Pichai explained:
“I think one of the great things we find with search is it’s easy to measure user satisfaction. For 25 years, we have learned to measure user happiness, user satisfaction in a correlated way with improving the quality of the product, not for short-term, that’s why we do these long-term studies.
And if you get it wrong, or get any experience wrong, it shows in the metrics and we course correct.
And we pride on the ability to track this over longer term, be it engagement, sessions, returning to a topic, the number of bounce backs they do.
So very, very sophisticated way of looking at it.
I think in some areas like that, I think the experience will continue to evolve.”
SEOs tend to think of Google’s engagement metrics like clicks as something that directly affects websites. But many research papers published by Google show that click data is noisy and you can’t really use it to promote or demote individual sites with just a handful of clicks. In fact, the United States DOJ antitrust exhibits confirms that click data only becomes meaningful at scale. What that means is that patterns begin to show in the billions of searches and clicks.
AI Search Is Evolving Too Fast?
Patel asked Pichai if the experience in the search example was good. Pichai acknowledged that the AI Overviews example was probably more opinionated than it should have been and that there is room for improvement.
He explained that AI search was a “fast evolving space” and because of that he would expect these kinds of less than ideal search results to happen.
Pichai continued:
“I think it’s probably more opinionated than it should be for the particular query you showed me.
That’s how that was my reaction as a user, right?
So I think that’s a scope for improvement is how I would say in a fast evolving space.
And, you know, but I would expect that to happen in the product, right?
Like my intuition there is, oh, that’s way more opinionated.”
That’s an interesting answer because it sounds like he’s saying that Google’s user satisfaction and engagement metrics can’t keep up with the fast pace of changes due to how fast the technology is evolving.
Personalization Can Alter AI Search Results
Pichai then suggested that one of the reasons for the poor AI search result was that it may have been personalized to Nilay Patel and that the personalization may have skewed the search result.
This is really interesting because it points to how sensitive AI search can be, where complex or repeated queries can result in hyper-personalized answers, but in edge cases. That means that SEOs and business owners can’t 100% rely on keywords recommending their website every single time for a particular query. There may be edge cases where Google recommends other products or websites.
How often would these edge cases show? Pichai continued his answer which described a personalization scenario that changes the query for some, but not all.
He continued:
“There is some chance that’s personalized to you may be testing it in a way that you’re uniquely personalizing.
You are the reason that query might not be exactly representative, though I think because I know how you review all these things, like, you know, so there is some chance you’re in the 0.0001%.”
That answer, in my opinion, describes two things.
1. It seems to imply that for many queries the AI search may tend to show a representative group of answers to questions, similar to the way regular organic search would present a reliable group of sites.
2. His answer also indicates that AI search can be sensitive to users and that their queries can influence the answer the AI shows.
For example, it’s somewhat well known that leading questions can influence the answers that an LLM provides.
A leading question can be something like:
“Explain why SEO results in Human Slop content.”
And ChatGPT may reply with:
“The core problem is that SEO often turns writing into a response to visible signals instead of a response to real reader need. Writers look at ranking pages, extract the repeated patterns, add the same subtopics, answer the same People Also Ask questions, use the same headings, and produce something that technically covers the topic without adding judgment, experience, or a point of view.
That creates content that is “human” in the sense that a person assembled it, but “slop” in the sense that it is derivative, padded, and optimized around what search appears to reward.”
Circling back to Sundar Pichai’s response, his answer indicates that users can change the answers that an AI search provides and that can result in less reliable traffic from search.
Traditional Search quality systems were built around links, rankings, satisfaction, return behavior, and other user behavior signals. AI Search adds a layer on top of that traditional search, where the traditional search underlies the answers surfaced by AI. But that experience is also more complex than a ranked list of results.
It may also make the success or failure of that AI search experience harder for Google to judge through the same signals that worked for earlier versions of Search.
Personalization Can Create Search Outliers
Personalization might create a gap between aggregate measurement and individual experience. Google’s metrics may show how Search performs across large groups of users, but infinite personalization can create results where the metrics of the masses do not explain the experience of the individual.
Google may still be able to measure satisfaction at scale. But what about businesses and SEOs? Google shows data from AI Search but it’s blended in with the traditional classic search data. That makes attribution more difficult.
Search Metrics May Not Capture Public Distrust Of AI
Patel then moved from the Search result itself to a broader question about AI and public distrust.
He noted that Google may be measuring user satisfaction and seeing large usage numbers, but he noted that public sentiment around AI is often negative. He cited young people disliking AI, former Google CEO Eric Schmidt being booed at a college graduation speech, and public opposition to data center construction.
Nilay Patel asked:
“Well, this is kind of why I’m asking about infinitely personalizable results, right?
And I’m also asking if the experience is good… There’s the stuff you can measure about user satisfaction. And then there’s how the public feels about AI.
And I think there’s a pretty yawning gap in, “hey, there’s these user numbers going up and we’re close to a billion users” and the free products people are experiencing, how good they might be.
And then just the polling data, young people dislike AI… You can go ask them and they will tell you in measurable ways they dislike it.
Eric Schmidt, the former CEO of Google, booed at a college graduation speech he was giving.
Seven in 10 Americans opposed data center construction.
There’s some gap between the product experiences people are having and how they feel about the technology.
Do you think you can close that gap?
Do you think these products are good enough?”
That’s a really good question because it challenges Google’s own user satisfaction metrics, that maybe Google doesn’t really understand how users feel about AI search results. How can Google say that people love it when young people are on record as not liking AI and most people don’t want more AI data centers.
Search satisfaction metrics can measure whether users continue interacting with Search. They may not indicate whether users are actually happy with AI search.
Pichai did not dismiss that concern in his answer:
“I think it is a very profound topic, and I feel like you’re linking the two things.
I think people, rightfully so, AI is the most profound technology humanity is going to deal with. It’s happening at a very fast pace.
I don’t think humans are evolved for processing this much change, and the rate of change, particularly over the last few years, you know, incredibly high.
And people rightfully so, particularly with all what they’re hearing, I think people are trying to understand the future and in the personal context of their lives, including what it means at an economic level and so on, right?
And so to me, it really makes sense why there is anxiety around this technology. And I think we should be very attuned to that.”
Does AI Have A Marketing Problem?
Patel asked Pichai directly whether AI has a marketing problem. Pichai insisted it does not.
“No, I don’t think so. That’s the point I’m making. I’m in fact arguing against it. I think it makes sense to me why people would feel concerns about it, feels natural to me.
People are standing and telling about how AI could make a lot of jobs go away, right? And like, why wouldn’t you feel a sense of anxiety about it, right?
I think those are deeper issues, which we have to tackle as a society.
…All I’m pointing out is it’s a multi-layered problem.”
When it comes to publishers, businesses, and SEOs, Google may have more than a PR problem, there is a lot of anxiety about search traffic declines.
Google Responds To Google Zero Concerns
Patel then moved from user concerns to publisher concerns.
He raised the idea of Google Zero, the scenario in which Google referrals trend down toward zero. Patel cited Conde Nast CEO Roger Lynch, who said publishers should plan as if search traffic is zero.
“Every year, search traffic was down more than we had forecast,” Patel quoted Lynch as saying. “So last year, I told our teams, assume there is no search. You have to have your businesses planned as if search is zero.”
Patel asked Pichai how he would respond to one of the world’s major publishers saying it can no longer depend on Google Search traffic.
Pichai’s answer focused on saying that Google is sending traffic to a wider range of traffic, like forums.
He responded:
“Look, first of all, the information ecosystem is so much broader beyond Google. By far, we see it in the data, you see it everywhere.
So any publisher over the last 10 years, I would look at Verge and I would say where you were when you first took over, how much it’s evolved since then, the types of content you make, where all you put that content out, how all you users are coming to you. I think it’s exceptionally dynamic.
And so it makes sense to me every publisher is adapting to this new world. We are, you know, adapting to the evolving world, how users are consuming technology, we had to do when the world shifted from web to mobile.
We are shifting it from a world of mobile to people having ongoing conversations, chatting with these products, talking to them, consuming it in voice and many different form factors.
People are expressing preferences for various types of content, they’re looking for user-generated content, they’re looking for podcasts, they’re looking for that.
Through it all, we are very committed to both meeting user expectations and also getting them to connecting them to what’s out on the web.
And just even in the last year, even since we’ve launched these features, we’ve gone back, we’ve added more links. Another area where behavior is changing.”
Nilay Patel pushed back on Pichai, asking if a big brand like Conde Nast should assume their search traffic will be zero.
Google’s CEO insisted that the content pie is growing and that Google is sending traffic to that larger Internet that is represented by a wide range of content that goes beyond traditional text content.
He answered:
“I always view people understand their businesses better. I’m not in a position to tell, you know, such iconic publisher what they should think about their business or plan.
If they are building content, which is high quality and people like it, I expect us to reflect that in our products.
That much I can commit to them, right?
I think more than any other company, through this evolution, we are working very hard to make sure people can get connected and we are planning to do it in search, in Gemini, and that’s still underpins a lot of what we do.
But there is evolution. As the technology improves, low quality clicks get filtered out. That’s a natural evolution we see.
We see it in our metrics. Bounce clicks are going down. And so those are all dynamics. And people are going to a wider array of information.”
Takeaway
Pichai’s comments show how Google is thinking about AI Search is disconnected from how “iconic brands” like Conde Nass, smaller site owners, and SEOs are thinking about search traffic. Google sees it as an evolution but everyone else on the other side of that search box only sees crisis.
He does have a point that there are more kinds of sites that Google is sending referrals to, but that comes too late for forums, many of which withered after Google stopped sending traffic to them beginning around 2013. Thousands of forums across many topics have disappeared.
The other factor, which Pichai somewhat acknowledged, is that Google’s user satisfaction metrics may be showing how users are engaging with AI search but they may not be showing how people really feel about AI search.
From an SEO viewpoint, it’s interesting that Google is still using those traditional metrics but the interview also raised the issue that those metrics may not be enough to calibrate a rapidly changing technology like AI, which can have a negative impact in the quality of results.
Watch the interview here, beginning at around the twenty minute mark:
