The First-Ever UX Study Of Google’s AI Overviews: The Data We’ve All Been Waiting For

One thing I need you to understand about the groundbreaking data I’m about to show you is that no one has ever done this kind of analysis before.

Ever.

To our knowledge, no other independent usability study has explored a major web platform at this scale.

AI changes everything, and search is at the forefront.

Together with Eric van Buskirk and his team, I conducted a behavioral study that provides us with unique and mission-critical insights into how people use Google, especially AI Overviews (AIOs).

This data allows us all to better understand how people actually use the new feature and, therefore, better optimize for this new world of search.

We captured screen recordings + think-aloud sessions on 70 people (≈ 400 AIO encounters) to see what really happens when Google shows an AIO.

We tracked their scrolls, hovers, dwells, comments, and even their emotions!

The effort to gather and evaluate this data was high. It required:

• A solid five-figure USD investment.
• A team of six people.
• Combing through 13,500 words of annotations.
• Sifting through 29 hours of recordings.
• So many hours we lost count.

I want to call out that the study was directed by Eric Van Buskirk.

We designed the questions, focus points, and the method together, but Eric hired collaborators, ran the study, and delivered the results. Once the study was finished, we interpreted the data together.

Here’s a three-minute video summary of the results:

 

Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!

Executive Summary

Our usability study puts hard numbers behind what many SEO pros have sensed anecdotally:

• Traffic drain is real and measurable. Desktop outbound click-through rate (CTR) can fall by two-thirds the moment an AIO appears; mobile fares better, but still loses almost half its clicks.
• Attention stays up-screen. Seven in 10 searchers never read past the first third of an AIO; trust and visibility are won – or lost – inside a few lines.
• Demographics define user behavior. Younger mobile users embrace AI answers and social proof; older searchers still dig for blue links and authority sites. Query intents with high-risk outcomes (like Your Money or Your Life searches) also cause users to dig more into search for validation.
• The decision filter has changed. Brand/authority is now the first gate, search intent relevance the second; snippet wording only matters once trust is secured.
• Residual clicks follow community proof and video. Reddit threads, YouTube demos, and forum posts soak up roughly a third of the traffic that AIO leaves behind.

Together, these findings show that visibility, not raw referral traffic, is becoming the main currency of organic search.

Key Takeaways

Before you dig into the overall findings, here are the high notes:

1. AIOs kill clicks, especially on desktop: External click rates drop when an AIO block appears.
2. Most users skim only the top third of the panel: Citations or mentions for your brand must surface early to be seen. Median scroll = 30% of panel height; only a minority of users scroll past 75%.
3. Trust is earned through depth: Scroll-depth and stated trust move together (ρ = 0.38). Clear sources high up accelerate both trust and scroll-stop rate.
4. Age and device shape engagement: 25 to 34-year-olds on mobile are the power users: They pick AIO as the final answer in 1 of 2 queries.
5. Community and video matter post-AIO: When users do leave the SERP, many outbound clicks go to Reddit, YouTube, or forum posts – social proof seals decisions.

Methodology Summary

You’ll find a detailed methodology at the end of the article (and a methodology deep dive from Eric here), but here’s a short summary of how the data was collected:

We asked 70 U.S. searchers (42 on mobile, 27 on desktop) to complete eight real-world Google queries – six that trigger an AIO and two that do not – while UXtweak recorded their screens, scrolls, clicks, and think-aloud commentary.

Over 525 task videos (≈ 400 AIO encounters) were frame-by-frame coded by three analysts who logged scroll depth, dwell time, internal vs. external clicks, trust statements, and emotional reactions for every SERP element that held attention for at least five seconds.

The resulting 408 annotated results provide the quantitative spine – and the qualitative color – behind the findings you’re about to read.

We asked participants to complete these eight tasks:

1. Using Google Search, find a tax accountant in your area by searching as you typically would.
2. What are the best months to buy a new car?
3. Find a portable charger for phones under $15. Search as you typically would.
4. Find out how to transfer from PayPal to a bank.
5. Search Google for “promo code for a car rental.”
6. Search for two or three reasons why artificial sweeteners might cause health problems.
7. Search Google for “sell gift cards for an instant payment,” and imagine you have to choose one of the services mentioned.
8. Search Google for “how to waterproof fabric boots at home.”

1. How Do Users Actually Read AIO Content?

Several analyses have examined the impact of AIOs on click-through rates and organic traffic.

But no one has yet looked into how users actually engage with AIOs – until now.

In our analysis, we captured how far down the AIO users scroll, when they click the “show more” button, and where they dwell on the page.

Key Stats:

An overwhelming 88% of users clicked “show more” to expand truncated AIOs.

We measured how far down* the participants scrolled who looked at the AIO result for at least five seconds:

• Average scroll depth: 75%.
• Median scroll depth: 30%.

*0% = they never scrolled inside the box; 100% = they reached the very bottom at least once.

A few outliers skew the average.

The median is much more telling: Most users stop reading AIOs after the top third.

In total, 86% of participants “skimmed quickly,” meaning they didn’t take much time to read everything in the AIO but scavenged for key insights.

Dwell times averaged between 30-45 seconds, indicating meaningful user engagement rather than superficial interactions.

Eric, director of the study, found that 40% of questions end with statements like “I usually don’t go past this” or “AIO answers all my questions.”

The remainder, almost a third of the sessions, show people scanning AIOs, then choosing a brand site, video, Reddit thread, or .gov/.edu result instead. (“I like AIO, but I still prefer Reddit,” was a sentiment we heard.)

But who scrolls further down the AIO?

• Young people: Ages 25-34 years.
• Mobile users: An average of 54% mobile users vs. 29% desktop users keep scrolling the AIO.
• Searchers with an intent that reflects high stakes: Think tasks that involve financial or medical queries. Low-stakes searches, like coupon codes, are the opposite. Here’s a look at the average scroll depth across intents:
  • Health YMYL – 52%.
  • DIY or how-to – 54%.
  • Financial YMYL – 46%.
  • Decision timing (“best month to buy…”) – 41%.
  • Promo code queries – 34%.

When we asked participants about how much they trust AI-generated summaries, we got an average of 3.4 – quite high!

Why It Matters:

Similar to classic search results, aim to be cited as high up the AIO as possible to be the most visible.

When optimizing, we also need to consider the stakes of each individual, singular search query and what it might take for a person to verify a claim or find a trustworthy solution, whether the search is in YMYL topic or a general, traditionally low-risk topic.

This is more practical than the YMYL framework we’ve been using for a long time. The more a user has to lose when making the wrong decision, the more likely they are to engage deeply with AIOs.

Ultimately, our study shows that users engage more with an AIO out of skepticism. The higher the stakes are for a decision, the more they question the AIO. And the more they work to validate the AIO with sources outside of it.

Insight:

Users treat an AIO as a fact sheet: quick scan, expand if needed, minimal internal navigation.

You can see this in the difference between average and median scroll depth. Only a few users scrolled down to 75% of the AIO.

Users who end the task saying they trust the AIO are the same ones who have scrolled far enough to read citations or expanded paragraphs. Authoritative sources showing up high in the AIO accelerate trust.

Practical Takeaways:

1. Most people will never reach the bottom of the AIO, so valuable mentions and citations are only up high, similar to how classic search results work.
2. Similar to optimizing for Featured Snippets, when targeting AIOs, keep answers in content blocks concise, to the point, and simple.
3. Invest in your positioning, messaging, and becoming an authoritative source in your area of expertise. That way, users recognize your brand in the SERPs – and ideally before they search.

I’m dropping more insights and guidance on how to apply these learnings for paid subscribers later this week. Make sure you don’t miss it. Upgrade here.

2. What’s The Click-Through Behavior Like When AIOs Are Present?

AIOs give users answers before they click on web results.

Therefore, the logical question is how much less traffic can websites expect when AI Overviews show up?

Key Stats:

Everyone obviously wants to know how AIOs impact click-through rates. But clicks are just a proxy for completed user journeys.

While these are usually hard to track, we were able to figure out exactly when participants completed their journeys based on their commentary and screen tracking.

The remaining ~80% of queries were answered using:

• Organic and sponsored results.
• Community forums.
• Videos.
• Map packs.
• Other prominent SERP Features, like People Also Ask.

This observation actually fits Google’s narrative of AIOs being a “jumping-off point,” but I want to be clear that AIOs also kill a lot of clicks in the process, and our tasks require a higher level of skepticism than many highly searched queries.¹

Of course, there’s a difference between commercial and informational queries that we must keep in mind.

Notably, 4 out of 5 users progressed past the AIO, so ranking in the first organic or paid slots remains critical for monetizable queries.

Most answers (81 % on desktop and 78 % on mobile) for transactional and/or commercial queries came from other non-AIO SERP elements, such as:

• Organic links.
• Discussions and forums.
• Featured snippets.
• Promo‑code aggregators.
• Sponsored results.

But for AIO actions that took place in the Overview itself, here’s what we found:

• On mobile, 19% of participants clicked a citation-related element within the AIO panel, such as a link icon or hyperlinked text (excluding “show more” clicks).
• On desktop, users clicked internally within an AIO just 7.4% of the time.

Overall, our main AIO blocks contained few hyper-texted links, and on desktop, these links were nearly absent. The primary click out of the main panels was the (somewhat confusing) link icon.

Why It Matters:

The data we gathered from this part of the study confirms a few things:

• Don’t expect too much traffic, even when you’re cited high up the AIO. Traffic loss is inevitable and probably impossible to compensate for. (But there is hope.)
• Revenue models tied to sessions are suffering and will suffer more. (For example: Sites that rely on ads and affiliate models.)
• Marketing dashboards that track only visits are under-valuing visibility wins or hiding looming losses.
• The SERP battleground is shifting from rank to AIO presence. Budgets and optimization practices have to follow.

Insight:

The data shows that users treat AIO primarily as a read-only summary.

Users read, decide, and stay put.

Outbound traffic is the exception, not the rule. When AIOs are absent, outbound click rates rise to an average of 28% on desktop and 38% on mobile.

Notice how simple questions don’t require click-throughs in the following clip from the study:

And yet, there are cases in which organic results convince users to be better than AIOs:

Practical Takeaways:

Optimize content for AIO citations, but don’t measure referral traffic (ie, clicks) for success.

Instead, measure visibility by monitoring the following:

• Impressions (easy but fuzzy).
• Citation rank (how high up the AIO you’re cited).
• Share of Voice (how often you’re cited, how high, and how you show up in the organic results).

You also need to immediately communicate to leaders, stakeholders, partners, and clients that organic traffic is already or about to drop significantly.

For those who are subscribed to the paid version of The Growth Memo, we have a prepped slide deck to help you communicate these changes to your stakeholders coming out this week.

3. How Do People React To SERPs Emotionally?

Emotions drive decisions more than rationale.

As I wrote in Messy Middle:

“We’re more emotional animals and make more decisions from our gut than we like to admit.”

Besides engagement, we also wanted to know how users feel about the results they’re seeing.

Why? Because emotions have an impact on our decisions, from clicks to purchases.

Key Stats:

Why It Matters:

Emotion is tied to risk. Searchers are internally asking What’s at stake? When making a decision to trust a result

And as a result, high-stakes niches – or even expensive products – receive more skepticism and scrutiny from users.

This skepticism plays out in the form of clicks – a.k.a. your opportunities to convince people that you’re trustworthy.

The good news: Users don’t rely on AIOs only for YMYL queries; they also validate and verify with classic results.

In low-risk niches where the threat of picking a wrong answer is low – like coupon codes or certain informational queries – brands can focus on page speed and price.

Insight

Overall, here’s what stuck out the most from this segment of the study:

• Hesitation or confusion spikes on medical or money queries when AIOs cite unknown brands.
• Reassurance-seeking (opening a second organic link “to be sure”) appears in 38% of sessions where an AIO is present.
• No-reaction silent scans dominate product or local-intent tasks.

For high-stakes queries, users care about authoritative sources, as you can see in this clip from the study:

But organic results can still win if they signal better relevance.

Practical Takeaways:

1. Sites in the health and finance spaces have a higher chance of seeing lower traffic losses from AIOs.
2. Aim to get mentioned or linked from highly authoritative sources, like .gov sites.
3. Prioritize trust-building in your on-page experience to catch those double-check clicks. You can do this with visible editorial guidelines, expert authors + reviewers, and high-effort content production (original graphics, etc.).

4. What Influences The Type Of Result A User Chooses As Their Final Answer?

Up until now, my mental model of search – and I would argue the industry’s as well – was that users pick results by relevance: “Does it answer my question?”

But that has changed, and I think AI is a big reason.

Key Stats:

We grouped over 550 think-aloud comments into four recurring themes to explain the new user behavior in the search results:

Source Trustworthiness (= Primary Click Motivator)

Whenever a recognised brand, authority site, .gov or .edu appeared, it was chosen first in 58% of the cases where such a link was present.

Comparison/Validation (= Secondary Driver)

After reading an AIO or Blue Link, 18% of users still opened a Reddit thread, YouTube video, or second organic result “just to double-check.”

Snippet/Preview Relevance (= Speeds Decision)

After clearing the trust gate, users scanned the two-line snippet, bolded query terms, or AIO phrasing. When the snippet looked off-topic, users skipped even trusted domains.

Top-Of-Page Visibility (= Skews The Decision)

Limited viewport and thumb ergonomics make “position-0” features (AIO, featured snippet) and rank-1 organic vastly more influential on phones.

First-screen links were chosen 71% of the time. Users only scroll when the topic feels risky.

Why It Matters:

It’s not just about matching the intent of the query anymore. The old notion of “search intent relevance only” is outdated.

Brand authority and trustworthiness compound: Once you’re trusted, you likely outrank unknown rivals – even without richer snippets.

Of course, placement matters, and SERP real estate above the fold is scarce … and skews user decisions.

Trust is the core ingredient when it comes to anything AI. Search is no exception.

Insight:

Users apply a rapid two-step filter that looks like this:

“Do I trust this result?” → “Does this result answer my question?”

Look at these two clips from the study and notice how the participant selects results he explicitly trusts:

You’ll hear the participant state:

• “Yelp is a good resource that I use a lot, so I’d probably click Yelp.”
• “I’ll try to find one that has decent reviews and that’s nearby.”
• “I trust Yelp.”

You’ll also hear this in Clip No. 2 above:

• “US News and World Reports is trustworthy. Edmonds is trustworthy.”
• “I picked this ’cause US News and World Reports is a trusted source, and they have a clear answer right here in the key takeaways.” (Note: They make information easy to find.)

Of course, there is nuance to this two-step filter.

The director of this study, Eric, adds the following observations:

1. How-to and evergreen intents (like waterproofing boots, selling gift cards, or coupon hunting) are easiest to satisfy for AIOs. Users feel the AI is “tried and tested” and “super helpful” for these stable facts.
2. Location-sensitive or personal-risk queries trigger more skepticism. One study participant shared aloud that, “It only says New York … that doesn’t help me,” and another shared, “I’d go straight to PayPal for accuracy.”
3. Medical-risk examples show a mixed approach: Some users praise the concise summary, others insist on cross-checking with authoritative sources like the Mayo Clinic or the NHS.

Ultimately, we’ve noticed that the more time users spend reading the AIO, the higher their chance of trusting the answer and being influenced by it.

This is a priming effect: Once a brand or concept appears in the AIO, it remains top-of-mind.

Practical Takeaways:

1. Trust is the gatekeeper. One of the biggest drivers of SEO success is how well you’ve earned “share of mind” before someone even sees your brand in an AIO or search result. Maybe that was always true, but AIOs make it non-negotiable now.
2. Being present in the AIO (ideally high up) is valuable because it leaves an impression on users. That’s where impressions as a metric become more valuable.

5. How Do Demographics Influence Search Behavior And Interactions With AIOs?

We often talk about user intent in SEO, but completely ignore demographics.

History has shown that technical jumps, like what we’re living through with AI right now, have a bigger impact on younger demographics.

The same is true of Search.

Key Stats:

Our research found stark differences in how people of different ages engage with the search results.

Below, you’ll see the percentage of time when an organic result was chosen, whether there was an AIO present for the user’s review or not.

Why It Matters:

One-size-fits-all SEO practices don’t work anymore.

Just like for other social or content platforms, segmentation by demographic becomes as critical as segmentation by keyword intent.

Insight:

AIO adoption is generational.

Older audiences are still relying heavily on classic organic results. Younger demographics are more likely to focus on the AIO and validate with Reddit.

Practical Takeaways:

Prioritize content and SERP Feature bets by age segment.

For brands that target an older audience, double down on classic organic search. Don’t over-index on AIOs. The exception here would be queries with a local intent or online shopping searches.

In these cases, user intent overpowers age preferences.

Quick reminder here: Premium subscribers get expanded info later this week. Upgrade to paid.

6. How Do Devices Impact User Behavior?

Devices reflect search context.

Mobile devices are used more often on the go, which is why mobile searches are more likely to have local intent and SERP Features like local packs.

Mobile users are also more restrained in their behavior due to smaller screen real estate. These factors are also reflected in how users engage with AIOs.

Key Stats:

Why It Matters:

You need to ensure mobile snippets and structured data are flawless; they get more scrutiny.

Mobile is now the primary remaining source of incremental Google traffic. Optimize for it first.

Insight:

Vertical scrolling and thumb ergonomics make mobile users dig deeper and click out more.

And when an AIO is missing, users revert to classic “blue link” behavior, especially on mobile, where more than one-third of searches produced a click to a non-Google site.

Practical Takeaways:

You need to track and compare mobile and desktop SERPs.
We missed this in classic SEO, and now it’s so much more important.

To prioritize which format you optimize for, you must validate that you get more mobile users to your site first (use Google Search Console).

If mobile is important to your target audience, regularly run separate mobile rank and snippet audits. And optimize the above-the-fold experience, in addition to:

• Making the page skimmable.
• Shortening time-to-value on the page (essentially, the time it takes to resolve the query or reach an insight from your site).
• Simplifying navigation on the page and site.

7. How And When Do Users Engage With Community-Based, Video-Based, And Shopping Carousel Content?

The controversial rise of Reddit often leaves us wondering why Google gives community content so much prominence across all topics and verticals.

Our study explains what users really do.

Key Stats:

We looked at where clicks go when users leave Google or want to validate answers:

Keep in mind that SERP Features and corresponding user behavior vary by question or task performed.

In this study, only one task surfaced video results: “how to waterproof fabric boots at home.”

And here’s how users in this study responded to video results:

• Users watched the preview frames, hovered for autoplay, then clicked through to YouTube in 5 of the 7 cases.
• Although videos made up less than 2% of all logged elements, their 37-second dwell time exceeds AIO dwell time itself (31 seconds).
• Users linger to watch autoplay previews or scroll thumbnails before deciding to click through.

For shopping-related tasks, we noticed the following:

• 30% of clicks go to local packs.
• 26.4% of clicks go to shopping modules (product grids).
• 13.2% of clicks go to text ads.
• 40% of clicks went to paid-organic results (text + PLAs).
• 7 out of 10 clicks bypassed classic organic links in favour of Google‑curated verticals or ads.

By the way, Amazon was a huge competitor to the shopping carousel.

Many people said, “I would just go to Amazon” (see clip below):

This study participant states: “Typically, I go to Amazon … scroll past the sponsored results and look for something with a lot of reviews.”

Why It Matters:

Social proof platforms (Reddit, YouTube) absorb the demand that AIOs can’t satisfy. Be present there.

Insight:

Community proof-points matter. When users leave the SERP after looking at an AIO, community links receive a lot of those clicks (18% when AIOs are not present).

People – especially the younger cohort that trusts AIO the most – use forums to get a (validating) voice from another human. Users in their 20s to 30s clicked Reddit or YouTube far more than older cohorts.

For some queries, like how-tos, users skip the AIO intentionally because they expect richer media, like videos.

Practical Takeaways:

1. Invest in Organic Reddit (or the most relevant forum in your industry) when and if it appears for your most relevant queries. Seek both citations and social proof, as they reinforce each other.
2. Optimize video thumbnails and the first 15 seconds. Users decide whether or not to click from the autoplay preview; if the opening doesn’t show the task in action, they skip.

Conclusion: Welcome To The New World Of Search

You made it to the end! Congratulations to you and your attention span (or did you just scroll here 🤔?).

To summarize everything you just (hopefully) read: If your brand isn’t surfaced in the first third of an AIO, it’s effectively invisible.

Search has flipped from a click economy to a visibility economy.

And within that economy, the new currency is authority, which now outranks search intent relevance.

Users ask, “Do I trust this brand?” before they even consider the answer.

If I had to boil the findings down to one sentence, it would be this: Users treat an AIO as a fact sheet: They quickly scan, expand if needed, and use minimal internal navigation.

Top Takeaways For Operators:

1. Shift KPIs from clicks to presence. Track how often, how high, and for which queries your brand appears in AIO.
2. Lead with authority. Invest in expert endorsements, .gov/.edu links, and PR that earns immediate trust.
3. Package answers for skimmers. Key-fact boxes, bullets, and schema matter more than ever.
4. Own the validation click. Seed Reddit threads, video demos, and comparison guides – users still seek a second opinion.
5. Segregate desktop and mobile strategy. Treat desktop as a branding surface; fight for mobile if you need traffic.

Top Takeaways For Decision Makers:

1. Expect – and budget for – a structural drop in organic sessions. AIOs cut outbound clicks roughly in half on desktop and by a third on mobile; revenue models tied to sessions (ads, affiliate) need hedging strategies.
2. Shift KPIs and tooling from “rank” to “share of voice in AIO.” Track how often, how high, and for which queries your brand appears in the panel; classic position-tracking alone masks looming losses. Keep in mind we’re still refining the new metrics model.
3. Invest in authority signals that secure trust instantly. Recognition by .gov, .edu, expert reviewers, or high-profile PR sways 58% of users to choose a cited source first. Brand trust precedes relevance in the new decision filter.
4. Allocate resources to validation channels – Reddit, YouTube, forums – where many residual clicks go after an AIO. Owning the follow-up click preserves influence even when Google keeps the first.

Open Questions That Still Matter

• Citation mechanics. How does Google choose which sources surface in the collapsed AIO, and in what order?
• Attribution leakage. Will Search Console or GA ever expose AIO-driven impressions so brands can value “on-SERP” exposure?
• Monetization models. If outbound traffic keeps shrinking, how will publishers, affiliates, and SaaS products replace lost session-based revenue?
• Personalization vs. authority. Will future AIOs weigh personal history over global trust signals – and can brands influence that balance?
• Regulatory impact. Could antitrust or copyright actions force Google to show more outbound links  – or fewer?
• Behavior over time. Do users acclimate to AIOs and eventually click less (or more) as trust grows?

Hint: Paid subscribers can get answers to these questions (and can send me any question that’s top of mind!) related to this study.

Additional Resources

Other primary research that puts the qualitative data into perspective:

• The Data Behind Google’s AI Overviews
• Testing Google’s Post-AIO Traffic Claims
• How is answer engine optimization different from SEO?
• The impact of AI Overviews on SEO: Analysis of 19 studies

Methodology

Study Design And Objective

We conducted a mixed-methods, within-subjects usability study to quantify how Google’s AI Overviews (AIO) change user behavior.

Each participant completed eight live Google searches: six queries that consistently triggered an AIO and two that did not. This arrangement lets us isolate the incremental effect of AIO while holding person-level variables constant.

Participants And Recruitment

Sixty-nine English-speaking U.S. adults were recruited on Prolific between 22 March and 8 April 2025.

Eligibility required a ≥ 95% Prolific approval rate, a Chromium-based browser (for the recording extension), and a functioning microphone.

Participants chose their own device; 42 used mobile (61%) and 27 used desktop (39%).

Age distribution was: 18-24 yrs 29%, 25-34 yrs 30%, 35-44 yrs 12%, 45-54 yrs 17%, 55-64 yrs 3%, 65+ yrs 3%.

A pilot with eight users refined instructions; 18 further sessions were excluded for technical failure and four for non-compliance. The final dataset contains 525 valid task videos.

Task Protocol

Each session ran in UXtweak’s Remote Moderated mode.

After reading a task prompt, the participant navigated to google.com, searched, and spoke thoughts aloud. They declared a final answer (“I’m selecting this because…”) before clicking “Done” in an overlay.

Task set:

1. Local service (“find a tax accountant near you”) – no AIO.
2. Decision timing (“best month to buy a car”) – AIO.
3. Low-cost product (“portable charger < $15”) – no AIO.
4. Transactional YMYL (“transfer PayPal to bank”) – AIO.
5. Coupon/deal (“car-rental promo code”) – AIO.
6. Health YMYL (“why artificial sweeteners might cause health problems”) – AIO.
7. Finance YMYL (“sell gift cards for instant payment”) – AIO.
8. DIY how-to (“how to waterproof fabric boots”) – AIO.

Capture Stack

UXtweak recorded full-screen video (1080p desktop or device resolution mobile), cursor paths, scroll events, and audio. Recordings averaged 25 min; incentives were $8 USD.

Annotation Procedure

Three trained coders reviewed every video in parallel and logged one row per SERP element that held attention ≈ for 5 seconds or longer. Twenty-three variables were captured, grouped as:

• Structural – participant-ID, task-ID, device, query.
• Feature – element type (AIO, organic link, map pack, sponsored, video pack, shopping carousel, forum, etc.).
• Engagement – scroll depth inside AIO (0/25/50/75/100%), number of scroll gestures, dwell-time (s), internal clicks, outbound clicks.
• Behavioral – spoken reaction (hesitation, confusion, reassurance-seeking, none), reading style (skim, re-read, etc.), AIO button used (show-more, citation click, carousel chip).
• Outcome – final answer, satisfaction flag, explicit trust flag.

The research director (Eric van Buskirk) spot-checked 10% of videos. Inter-coder agreement: dwell-time SD ± 3 s; Cohen’s κ on trust category = 0.79 (substantial).

Data Processing And Metrics

Annotations were exported to Python/pandas 2.2. Scroll values entered as whole numbers were normalised to fractions (e.g., 80 → 0.80).

The 99th percentile of dwell was Winsorised to dampen outliers. This produced 408 evaluated SERP elements and ≈ 350 valid AIO observations.

Statistical Analysis

Descriptives (means, medians, proportions) were stratified by device, age, and query intent.

Spearman rank correlations tested monotonic relationships among scroll %, dwell, trust, and query-refinement counts (power >.8 to detect ρ ≥ .25).

Welch t-tests compared mobile vs desktop means; McNemar χ² compared click-through incidence with vs without AIO.

Reliability And Power

With n ≈ 350 AIO rows, the standard error for a proportion of .50 is ≈ .05; correlations ≥ .30 are significant at α =.05. Cross-coder checks ensured temporal metrics and categorical judgements were consistent.

Limitations

Sample skews young (58% ≤ 34 yrs) and U.S.-based; think-aloud may lengthen dwell by ~5-10 s. Coder-judged trust/emotion involves subjectivity despite reliability checks.

Study window overlaps Google’s March 2025 core update; SERP UI was in flux. Findings generalise to Chromium browsers; Safari/Firefox users were not sampled.

Ethical Compliance

Participants gave informed consent; recordings stored encrypted; no personally identifying data retained. Study conforms to Prolific’s ethics policy and UXtweak TOS.

This narrative supplies sufficient procedural and statistical detail for replication or secondary analysis.

1 AI Overviews: About last week 

---

SEJ’s Content & SEO Strategist Shelley Walsh prerecorded an interview with Kevin before the launch to talk about his research. For more explanation about his findings, watch below.

More Resources:

• AI Overviews Data Shows Massive Changes In Search Results
• Data Shows Google AI Overviews Changing Faster Than Organic Search
• SEO In The Age Of AI

---

Featured Image: Paulo Bobita/Search Engine Journal