TL;DR
Does ChatGPT search the web for every answer? A MentionBird study found gpt-5-mini returned no visible web citations on 29.7% of API answers, compared with 0.6% for gemini-2.5-flash. But a missing citation is not proof that no retrieval occurred. Citation-only audits measure what appeared in the answer, not whether the model accessed the web. Separate brand mentions from page citations, record the model and product tested, and repeat queries enough times to understand variability before drawing conclusions about AI search behaviour.
Does ChatGPT search the web every time you ask it something? A recent study tried to answer that, and the results are more nuanced than the headline suggests.
MentionBird published a study with a headline that caught my attention: “Nearly 1 in 3 ChatGPT answers never touches the web.”
It’s a strong headline. There’s just one problem with how we might interpret it.
The study didn’t observe whether the model accessed the web. It measured whether the answers contained web citations. Those are two different things, and the distinction matters more than it might first appear.
Across a panel of commercial-intent questions, MentionBird found that gpt-5-mini returned no web citations in 29.7% of answers. For gemini-2.5-flash, the equivalent figure was just 0.6%.
That’s a substantial difference. But it doesn’t tell us that the first model skipped the web nearly a third of the time. It tells us that nearly a third of its answers displayed no web citations.
I’ve been looking at this distinction in my own GEO research because it’s easy to confuse what an AI system shows us with what happened behind the scenes. When a page isn’t cited, we don’t automatically know whether it was never retrieved, retrieved and ignored, or used without attribution.
If we’re measuring AI visibility, those aren’t interchangeable outcomes.
The study behind the headline
MentionBird, a commercial AI visibility platform, ran the same set of buying-related questions against two models daily between 27 April and 20 August 2026.
The questions focused on areas such as ecommerce, SaaS and agency services. Think of prompts asking for the best software for a particular job or recommendations for a specific type of service.
The researchers used the developer APIs for gpt-5-mini and gemini-2.5-flash, recording each answer and any web citations it returned.
That API detail is important. The study wasn’t testing ChatGPT.com or the Gemini consumer app. Those products can use different models, tools and retrieval behaviour. We shouldn’t take a result from one API configuration and present it as a measured rate for the consumer application.
Here’s what MentionBird reported.
| Measure | gpt-5-mini | gemini-2.5-flash |
|---|---|---|
| Answers with no web citations | 29.7% | 0.6% |
| Questions never receiving citations over four months | 8.4% | Not reported |
| Questions always receiving citations | 26.8% | Not reported |
The first figure comes from Edition 1, Table 1. The question-level figures come from Table 2.
There are two limitations worth keeping alongside those numbers. MentionBird hasn’t published the actual prompts, so an independent researcher cannot reproduce the exact panel. And the questions were selected around commercial buying intent, not to represent everything people ask AI systems.
The publisher also sells an AI visibility tracking product. That doesn’t invalidate the research, but it’s relevant context when evaluating the claims and how they’re presented.
A citation tells us what appeared, not what happened
This is the part I think deserves more attention.
Imagine asking an AI system to recommend a service. It retrieves several pages, reads them, produces a response and includes no links.
From the outside, that answer looks exactly like one generated without retrieval. Unless we can inspect the tool calls or supporting logs, we cannot distinguish the two.
The reverse is also possible. An answer can display a source without that page necessarily supporting the specific sentence next to the citation.
A citation is an observable output. Retrieval is an underlying process.
MentionBird used the presence of citations as its indicator of web search. The study acknowledges that a model could retrieve information without displaying citations, but its figures cannot tell us how frequently that happened.
That leaves several different explanations for the same missing citation. The model might not have searched. It might have searched but not selected your page. Or it might have encountered your content without crediting it in the answer.
A citation-only study cannot separate those cases.
For anyone auditing AI visibility, this matters. If your reporting system treats every missing citation as proof that the model didn’t search, you’re assigning a cause that the evidence doesn’t establish.
You might then spend time trying to fix retrieval when the real issue is source selection, attribution or something else entirely.
The same question didn’t always get the same treatment
The 29.7% headline attracted attention, but another result from the study interested me more.
Across four months, 8.4% of the qualifying questions never produced citations from gpt-5-mini. Another 26.8% always did.
Together, those groups account for 35.2% of the questions. The remaining questions moved between cited and uncited answers.
The question-level analysis covered 585 prompts with at least 20 runs, rather than the entire panel.
Why does that matter?
Because a single overall citation rate hides considerable variation between questions. Two prompts submitted to the same model can have very different citation histories. One may repeatedly produce answers with sources, while another may rarely show any.
If you’re tracking whether your brand appears in AI answers, knowing which type of question you’re measuring can be more useful than knowing the model’s average citation rate.
Consider a question that has never returned a visible citation across repeated tests. You can still measure whether your brand appears in the answer. But you can’t expect to measure linked citations in an answer that consistently displays none.
Now compare that with a question that regularly returns several sources, none of which are yours. That’s a different observed problem, and it deserves a different investigation.
Neither pattern tells us by itself why the model behaved that way. It does, however, give us a better starting point than treating every absent link as the same failure.
The unexpected result: answers without citations changed more
MentionBird’s second report examined how frequently recommended brands changed between consecutive answers.
One might expect answers without web citations to be more stable. If a model is relying on information already learned during training, perhaps it would keep recommending the same companies. Answers grounded in fresh web information might change more often.
The study found the opposite pattern.
For questions that never displayed citations, a brand’s presence in the top five recommendations changed on 43.7% of transitions. For questions that always displayed citations, the figure was 36.7%.
| Question group | Top-five brand flip rate |
|---|---|
| Never showed citations | 43.7% |
| Always showed citations | 36.7% |
That’s a seven-percentage-point difference.
It’s interesting because it challenges the assumption that uncited answers must be more consistent. But the evidence is narrower than a general claim that retrieval stabilises recommendations.
The never-cited group contained just 23 questions and 100 brand–question pairs, which MentionBird identified as its reporting minimum. Its observations were also taken a day apart, so the analysis couldn’t separate model sampling variability from changes in available web information.
We can say that the never-cited group showed a higher brand flip rate in this panel. We cannot say that displaying sources caused the other group to be more stable.
It’s a result I’d want to test again with a larger question set and better control over retrieval.
Evidence from the consumer apps tells a different story
There is another study worth placing alongside MentionBird’s work, although the figures should not be compared directly.
Allaham and Diakopoulos examined citation behaviour across the consumer interfaces of ChatGPT, Copilot, Gemini and Perplexity. Their study used 712 queries covering politics, health and the environment, with web search enabled for ChatGPT.
Across 2,848 responses, 252 contained no in-text citations. That’s 8.8%. Health-related queries had the highest uncited share, at roughly one in nine.
That isn’t evidence that the ChatGPT app has an 8.8% no-citation rate. The figure combines four engines and a very different set of questions.
Nor can we directly compare it with MentionBird’s 29.7%, which came from a specific API model answering commercial-intent prompts.
What the two studies have in common is a measurement boundary. Both record the citations displayed in answers. Neither provides the underlying retrieval traces needed to determine whether an uncited response involved web access.
Different products, models and prompts produced different observed citation rates. The underlying retrieval question remains open.
What I’d change in an AI visibility audit
The practical problem is that many AI visibility reports put too much weight on a single outcome: was the page cited or not?
That’s useful information, but it isn’t enough to diagnose what happened.
I’d separate the record into the following fields:
| Field | What I’d measure |
|---|---|
| Citations shown | Whether the answer displayed sources, and how many |
| Your citation | Whether your specific page appeared among those sources |
| Brand mention | Whether the answer named your brand, with or without a link |
| Tested surface | API, logged-out app or logged-in app, with the model recorded where available |
| Citation frequency | The share of repeated runs for that question that displayed citations |
This is my proposed reporting approach, not a method validated by MentionBird.
I’d also avoid assigning retrieval status unless there is direct evidence of retrieval. A citation tracker can tell you which links appeared. It cannot, by itself, reconstruct every page the system accessed or determine which source influenced the wording of an answer.
The distinction between mention and citation is equally important. A brand may appear in an answer without a linked source. That is observable visibility, but it isn’t evidence that the system retrieved the brand’s website during that particular call.
I’ve discussed this broader measurement problem in AI Citation Tracking: What Your Tool Actually Measures and API vs Browser AI Citation Audits.
Why repeated testing matters
There’s another reason not to base an audit on one answer.
MentionBird compared individual observations of a brand’s top-five presence with the rate measured over the four-month panel. A single check disagreed with that longer-run rate 30.7% of the time for gpt-5-mini and 33.0% for gemini-2.5-flash.
That doesn’t mean every AI visibility test is unreliable. It means one response is a poor substitute for a repeated measurement when the outcome varies between runs.
I’d start by grouping questions according to how often they show citations. A question that rarely displays sources belongs in a different reporting category from one that almost always does.
Then I’d examine brand mentions and citations within each group.
For questions that repeatedly return no citations, I’d investigate recognition and answer content without assuming retrieval didn’t occur. For questions that regularly cite sources but exclude my page, I’d investigate source selection and retrieval evidence where available.
Those are investigative priorities, not confirmed diagnoses.
The question set itself matters too. If your customers ask about local services, software comparisons and technical definitions, those different query types may behave differently. Combining them into one visibility score can conceal those differences.
That is why I’d record the query, surface, model and repeated outcomes rather than simply marking the brand visible or invisible.
What we still don’t know
After reading both studies, I see three questions that deserve further testing.
The first is how often a model retrieves content without displaying a citation. Neither study provides tool-call traces or server-side fetch records that could establish that rate. Verified server logs, aligned with individual test calls, could help identify some retrieval events preceding uncited responses. They still wouldn’t tell us whether the retrieved content influenced the answer.
The second is whether MentionBird’s question-level citation patterns hold in consumer products. The 8.4% never-cited and 26.8% always-cited figures came from an API panel using a specific model. The ChatGPT app may behave differently depending on its model, retrieval configuration and account state.
The third is how much these patterns depend on the query category. Commercial recommendations are only one part of AI search. Legal, health, news and technical questions may produce different citation behaviour.
These results are also tied to a specific period, 27 April to 20 August 2026, and to the models tested then. They provide a dated baseline, not a permanent citation rate.
The conclusion I draw
The central lesson isn’t that ChatGPT searches less than Gemini, or that a third of its answers come from memory.
MentionBird didn’t directly measure either of those claims.
It measured visible citations and found substantial differences between two API models. It also found that citation behaviour varied by question and that recommendations changed more frequently in the small group of questions that never displayed sources.
Those findings are useful when we keep their measurement boundaries intact.
For me, the main point is simple: an absent citation is an observation, not an explanation.
Record what appeared in the answer. Record the model and product you tested. Repeat the questions enough times to understand how variable the outcome is.
If you want to know whether retrieval actually happened, you need evidence beyond the citation itself.
Otherwise, you’re measuring a blank space and giving it a cause you haven’t established.
The next time someone asks “does ChatGPT search the web?”, the honest answer is: we can see what it cited, but we cannot always see what it retrieved.
Frequently asked questions about ChatGPT web search
Does ChatGPT search the web for every answer?
The MentionBird study cannot answer that directly. It found that gpt-5-mini returned no visible web citations in 29.7% of answers in its API panel. It did not establish how many of those answers involved no web retrieval, and the figure does not describe the ChatGPT consumer app.
If an AI answer has no citations, does that mean it didn’t search?
No. It means the answer displayed no citations. The system may or may not have retrieved web content. Without tool traces or other retrieval evidence, we cannot distinguish those possibilities.
Does Gemini cite sources more often than ChatGPT?
In MentionBird’s specific API panel, gemini-2.5-flash returned answers without citations much less often than gpt-5-mini: 0.6% compared with 29.7%. That does not establish the same relationship between their consumer applications.
Will the same question always produce citations?
Not necessarily. In MentionBird’s four-month analysis of qualifying gpt-5-mini questions, 8.4% never displayed citations, 26.8% always did, and another 33.5% displayed them on 75% to 99% of runs. Citation behaviour varied across questions and repeated calls.
What should an AI visibility report record?
At minimum, separate brand mentions from page citations, identify the product and model being tested, and report outcomes across repeated runs. If retrieval wasn’t directly observed, label it unknown rather than treating a missing citation as evidence that no search occurred.
Sources
- MentionBird (2026). Nearly 1 in 3 ChatGPT Answers Never Touches the Web, Edition 1. Published 20 August 2026; updated 14 September 2026. Source for the model-level citation rates and question-level citation frequencies.
- MentionBird (2026). ChatGPT and Gemini Replace 64% of Their Top Five Recommended Brands From One Answer to the Next, Edition 2. Published 27 August 2026; updated 14 September 2026. Source for brand flip rates and single-observation variability.
- Allaham, M. and Diakopoulos, N. (2026). Synthetic Sources?: Auditing Generative Search Engine Citations for Evidence of AI-Generated Sources. arXiv:2605.23684; AIES 2026.
Version 1.0: Research review of API citation behaviour and the limits of using visible citations as a proxy for web retrieval.
Have questions about this topic? Contact The GEO Lab · Return to homepage

