Ten concrete things that shift between two citation rate measurements even when your content hasn’t — the sources of the noise floor every GEO experiment has to account for
TL;DR
Two AI citation measurements of the same URL on the same query can produce different results. The difference isn’t measurement error — it’s real variance in a stochastic system. Ten variables drive most of it: model version, retrieval index state, session context, query order, sampling temperature, time of day, geography, cache state, competing sources, and iteration count.
You can’t eliminate them. You can hold each one constant, randomise across it, or measure over enough iterations that it averages out. Designing a GEO experiment without accounting for these ten is how you get a result that doesn’t replicate.
The same query, run against the same page on two consecutive days, can produce different citation outcomes without any content change. Ten variables are responsible — and not controlling for them makes every GEO experiment result uninterpretable, regardless of how large the measured effect appears.
Why the Same Query Gives Different Answers
The first time I tried to measure whether a content change had improved citation rate, I ran the query once before and once after. The number went up. I concluded the intervention had worked. I was, charitably, an idiot.
The GEO Stack measurement layer — Layer 3 — exists precisely because citation rate cannot be read reliably from a single run: the variables documented here are the reason a controlled experimental design is required.
Running the same query a second time before the intervention would have shown me the number was different the second time too. Running it five more times would have shown me a range that entirely contained the post-intervention result. The change I thought I’d measured wasn’t from the content — it was from the simple fact that AI citation systems don’t return the same answer twice.
This is the thing most GEO “case studies” in 2026 get wrong. They measure once before, once after, and declare a result. Without a variance estimate, that kind of measurement is unfalsifiable. The number changed. Why? Could be the content. Could be any of the ten things below. There’s no way to tell from two data points.
The noise floor — the variance you see when nothing changes — is the precondition for every other claim. Before you can say an intervention worked, you need to know how much the metric moves when nothing has been done to it. The variables below are where that movement comes from.
Where the Variance Comes From
The ten variables split into three layers, corresponding to where in the AI pipeline the variance is introduced.
Designing a probe set that controls for these variables requires understanding the ten probe query types — different query types introduce different amounts of variance at each pipeline layer.
Layer 1
Platform state
Model, index, cache
Layer 2
Session context
Order, history, geo
Layer 3
Generation
Sampling, competition
The layer matters because it determines how you control for the variance. Platform-layer variables can’t be held constant — they belong to the AI operator, not you. The best you can do is measure across them (multiple days, multiple model versions) and report a range rather than a point. Session and generation variables can be held constant through experimental discipline: same session setup, same sampling parameters, randomised query order.
The Ten AI Citation Variables
Ordered by the magnitude of variance each variable produces in the Lab’s observations. The first four are the ones that will ruin your experiment if you don’t control for them. The last six are smaller but still real.
The most controlled experiment the Lab has run — E016, the noise floor measurement — established the baseline variance range. That baseline and its protocol are documented in the Month 0 baseline post.
01
Model version
High
What changes
The AI operator silently updates the model behind the API — new weights, new retrieval heuristics, new citation policy.
Observed effect
Citation rate can shift by ten percentage points or more overnight with no correlation to any content change. The version string in the API response may or may not indicate the update.
How to control
Pin to a dated model version where the API allows it. Log the version string on every request. Treat any measurement across a version change as a separate experiment, not a continuous one.
02
Retrieval index state
High
What changes
The AI system’s retrieval index is updated as new content is added and old content is re-crawled. The set of candidate sources for any given query shifts over time.
Observed effect
A URL that was cited heavily last week may be displaced by a newer, more authoritative source this week. Your content hasn’t changed; the competitive set has.
How to control
Record the cited sources on every measurement, not just whether the target URL was cited. A drop in citation rate accompanied by a new domain appearing in the cited set points at index change, not content decay.
03
Session context
High
What changes
Queries within the same session accumulate context. Earlier questions prime the model toward specific entities, topics, or sources for later ones.
Observed effect
The same query issued as the first question in a fresh session and the tenth question in an ongoing one can produce different citations. Session warmth biases retrieval and generation.
How to control
Start a fresh session for every iteration. If batch-running queries, randomise order across iterations to distribute session-warmth effects across the sample.
04
Sampling temperature
High
What changes
Most AI systems use non-zero temperature at generation time, which introduces randomness in token selection. Even with identical inputs, the output varies.
Observed effect
The model may cite the same source with different anchor text, cite a different source entirely, or cite no source at all — across runs of a query where every other variable is held constant.
How to control
Set temperature to 0 where the API allows it. For APIs that don’t expose the parameter, accept temperature variance as a floor and use higher iteration counts to average it out.
05
Query order within a batch
Medium
What changes
Within a single session, the order in which queries are issued changes how the model attends to each one. Query three benefits (or suffers) from the context set by queries one and two.
Observed effect
Smaller than full session-context variance, but measurable. A fixed-order batch systematically over- or under-counts citation rate for queries later in the batch.
How to control
Randomise order across iterations, not just within one. Each query should appear in every position across the iteration set.
06
Time of day
Medium
What changes
AI operators run different model weights, rate-limit policies, and cache states at different times. Peak-hour requests may hit a different system configuration than off-peak ones.
Observed effect
Measurable differences in citation rate between, for example, 3am and 3pm on the same day. Usually single-digit percentage points, but consistent enough to confound a poorly scheduled experiment.
How to control
Run measurements at the same time of day across experimental conditions. For longitudinal tracking, schedule a fixed measurement window.
07
Geographic origin
Medium
What changes
Requests from different geographic regions may be routed to different data centres, hit different cache states, and receive region-adjusted retrieval priors.
Observed effect
A query that cites your URL consistently from an EU IP may cite it less often from a US IP, even for content with no regional specificity.
How to control
Pin the request origin. Run all measurements from the same IP range. Document the origin in the experiment metadata.
08
Cache state
Medium
What changes
AI systems cache intermediate results — retrieval results, embeddings, partial responses. A query hitting a warm cache returns differently than one hitting a cold cache.
Observed effect
The first run of a novel query may show different behaviour from the fifth run of the same query minutes later. Cache warmth often stabilises citation rate within a session but introduces between-session variance.
How to control
Hard to control without operator access. The practical workaround is to report the first-iteration behaviour separately from subsequent iterations, or to discard the first N iterations as warmup.
09
Competing sources in the retrieval set
Low
What changes
Even within a single query session, the retrieval step may pull different candidate sources each time. Your URL may be in the candidate set on iteration three and missing from it on iteration four.
Observed effect
Citation rate reflects both whether your URL is in the candidate pool and whether it’s selected from the pool. Two sources of variance combine.
How to control
Where the API exposes the retrieval set, log it. This separates “not retrieved” failures from “retrieved but not cited” failures — two different problems with different solutions.
10
Iteration count
Low
What changes
The number of times the query is run. Not a source of variance itself — a multiplier on whether the variance averages out.
Observed effect
One iteration gives you a point estimate with uncertainty equal to the full noise floor. Five iterations reduce uncertainty substantially. Ten iterations are usually enough to detect effect sizes below five percentage points.
How to control
Use at least five iterations per query as a baseline. Use ten when the intervention being tested is expected to produce a small effect. More iterations cost more but buy you the ability to detect subtler changes.
The Ten at a Glance
One reference table. Use this when designing an experiment — every row needs a decision about how you’ll handle that variable.
| # | Variable | Layer | Magnitude | Control method |
|---|---|---|---|---|
| 01 | Model version | Platform | High | Pin dated version; log string per request |
| 02 | Retrieval index state | Platform | High | Record cited sources; track competitive set |
| 03 | Session context | Session | High | Fresh session per iteration; randomise order |
| 04 | Sampling temperature | Generation | High | Set to 0 where possible; increase iterations otherwise |
| 05 | Query order in batch | Session | Medium | Randomise position across iterations |
| 06 | Time of day | Platform | Medium | Fixed measurement window |
| 07 | Geographic origin | Platform | Medium | Pin request origin; document IP range |
| 08 | Cache state | Platform | Medium | Discard warmup iterations; separate cold vs warm |
| 09 | Competing sources | Generation | Low | Log retrieval set where exposed |
| 10 | Iteration count | Experimental | Low | Minimum 5; use 10 for small expected effects |
Designing Around the Ten
Three design principles follow directly from the variables above.
In practice: the Tier 1 vs Tier 2 citation gap post shows how applying these design principles changes the interpretation of a 60-percentage-point citation rate difference between query tiers.
First: measure baseline variance before attributing any change. A pre-intervention noise floor is the unit against which post-intervention changes are interpreted. Without it, you have a point estimate and no idea what it means. Five days of measurement with no content change is the Lab’s working minimum; ten is better.
Second: control the cheap variables; report the expensive ones. Session context, query order, temperature, and geographic origin can all be held constant with basic experimental discipline. Model version and cache state often can’t — so log them, report them, and treat measurements across a model change as separate experiments rather than continuous ones.
Third: iteration count is the adjustable knob. When you can’t hold a variable constant, you can usually average it out by running more iterations. Five iterations per query is the baseline. Ten is the standard for detecting small effects. Twenty is warranted when model-version drift is suspected during the experiment window.
A practical mapping of when each variable matters across the experiment lifecycle:
| Stage | Variables to decide | What the decision determines |
|---|---|---|
| Pre-experiment setup | 01 Model version, 04 Temperature, 06 Time of day, 07 Geography | Whether the experiment has any chance of being replicable at all |
| Query batch design | 03 Session, 05 Order, 10 Iteration count | How much within-batch variance will average out versus confound |
| During measurement | 02 Index state, 08 Cache, 09 Competing sources | Whether observed changes are attributable to the content or the platform |
| Reporting | 01, 02 logged; 06, 07 documented | Whether another researcher can reproduce the result or dismiss it |
The single most common failure mode. Running a one-shot measurement before and after an intervention and declaring victory when the number goes up. The number would have gone up, or down, without the intervention. You cannot tell the difference from two data points. The variables above are why.
Frequently Asked Questions
Why do AI citation rates change between measurements?
AI citation systems are stochastic at multiple layers: the retrieval step samples from an index that changes as content is added and removed, the generation step introduces variance through temperature and sampling parameters, and the caching and personalisation layers shift results between sessions. Even when the input query and target content are identical, the output is not deterministic. The observed variance across repeated measurements is called the noise floor, and establishing it is a prerequisite for attributing any citation rate change to a content intervention.
How many of the same query should I run to get a reliable citation rate?
At least five iterations per query is the working minimum in the GEO Lab’s experiments, and ten is better where time and API cost allow. One iteration gives you a single sample from a stochastic distribution, which is indistinguishable from noise. Five gives you a point estimate with usable uncertainty. Ten lets you detect effect sizes below the single-digit percentage range.
Does the order of queries in a batch affect the results?
In some AI systems, yes. Session-level context accumulates across queries in the same session — earlier queries can prime the model’s attention toward specific entities, topics, or sources for later queries in the batch. The fix is to randomise query order across iterations, or to issue each query in a fresh session with no prior context.
What is the biggest source of variance in AI citation measurement?
In the GEO Lab’s experiments, model version changes produce the largest single source of variance. A silent model update on the provider’s side can shift citation rate by 10 percentage points or more overnight, with no correlation to any content change. Time of day, query order, and session state produce smaller but still measurable variance. Establishing the noise floor across multiple days, not a single day, is the only way to separate model-level drift from content-level intervention.