AI citation tracking scores need a measurement boundary. A high score on the questions the tool can answer does not make it capable of answering the rest.
TL;DR
AI citation tracking scores need a measurement boundary. In The GEO Lab’s E108 static benchmark, the citation-check instrument agreed with the USO Gold answer key on 78 of 79 comparable mention and citation determinations, or 98.7%. Across four states outside the checker’s scope, the key supplied 115 definite determinations, including 42 observed signals. The checker left all of those unresolved. The result supports reporting agreement alongside coverage; it does not establish live-search accuracy. As of October 2026, no AI citation tracking tool I have evaluated publishes both scores side by side.
The result I could have put in the headline was 98.7%.
That was the agreement score when I tested my AI citation tracking instrument against a fixed benchmark. It is encouraging, but it leaves out a large part of what I learned.
The checker was designed to assess mentions and citations. The benchmark asked about other states too, including retrieval and whether information was included in context. My checker could not assess those. A high score on the questions it could answer did not make it capable of answering the rest.
That is the part I would want to see before relying on an AI visibility report: what does the score cover, and what falls outside it?
AI citation tracking benchmark: what E108 tested
AI citation tracking measurement requires a known baseline to test against. I ran E108 on 20 August 2026, using citation-check E101-v-next, the version dated 6 August 2026. The full methodology, case-level results, and frozen artifacts are documented in the E108 instrument benchmark report.
The input was the USO Gold Test Set v0.1, pinned to the USO Evidence Protocol v0.9.5 release. Each case supplied response text and a visible-citations array. There were no live queries to ChatGPT, Google or another search product. All 50 cases returned a result, with zero processing failures.
| Part of the test | Recorded condition |
|---|---|
| Instrument | citation-check E101-v-next |
| Benchmark | USO Gold Test Set v0.1 |
| Run date | 20 August 2026 |
| Input | Supplied response text and visible-citations arrays |
| Cases returned | 50 of 50 |
| Live search queries | None |
I registered the experiment post-hoc on 24 August 2026. There was no deposited preregistration before execution. The controls had been declared in correspondence before the run: classify without consulting the answer key, freeze the classifications before opening it, and avoid per-case tuning. Those details belong beside the result. A reader should be able to assess how the test was run before deciding how much weight to give it.
Going deeper? The GEO Experiments — Design, Measure, Learn ebook covers experiment design, measurement protocols, and how to interpret results like the ones in this post.
What the AI citation tracking agreement score covers
AI citation tracking agreement in E108 used 79 comparable determinations: 40 for mentions and 39 for citations. A determination is one judgement about one state. It is not a separate search query or a separate benchmark case.
| State | Comparable determinations | Agreements |
|---|---|---|
| MENTIONED | 40 | 39 |
| CITED | 39 | 39 |
| Total | 79 | 78 |
The combined result was 78/79, or 98.7%. That is agreement with one answer key on the comparable part of one static benchmark. Calling it “98.7% accurate at tracking AI visibility” would extend the claim beyond the test.
Only three citation-positive cases
The citation result needs another detail: only three comparable cases were citation-positive in the key. The other 36 were citation-negative. The checker agreed on all 39. That is a result worth retaining, but three positive cases give limited evidence about how AI citation tracking handles a wider range of citations. I would want a larger set of positive cases before making a broad detection-performance claim.
The one disagreement was about entity identity
AI citation tracking can fail at entity identity, and the E108 disagreement demonstrated this. The disagreement came from case G012. The checker detected the string “Atlas” and returned OBSERVED for the mention state. The key returned NOT_ESTABLISHED because entity identity had not been established.
The name was present. The unresolved question was whether it referred to the intended entity. For brand tracking, that is a practical distinction. A matching name can belong to another company, product or person. Counting the text alone can make an unrelated appearance look like brand visibility.
I would want an audit to preserve the evidence used to identify the entity. If the surrounding answer does not resolve the identity, the report should retain that uncertainty. This is the same kind of ambiguity that the 30-check citation protocol encounters when brand names overlap with common words.
What AI citation tracking leaves out
AI citation tracking instruments have scope boundaries. The checker’s declared scope covered MENTIONED and CITED. It could not determine four additional benchmark states: RETRIEVED, INCLUDED, RECOMMENDED, and ADVERSELY_TREATED.
Across 40 scorable cases, those four states produced 160 out-of-scope slots. The checker returned NOT_ESTABLISHED for all of them. The benchmark key had more information:
| Key’s determination for out-of-scope slots | Slots |
|---|---|
| NOT_OBSERVED | 73 |
| OBSERVED | 42 |
| NOT_ESTABLISHED | 45 |
| Total | 160 |
The 73 negative and 42 positive determinations make up the 115 definite judgements outside the checker’s coverage. Someone asking “was my brand visibly cited?” may find a mention-and-citation checker suitable. Someone asking “was my page retrieved but left uncited?” needs evidence the checker did not assess. The retrieval-to-citation pipeline explains why those questions need different observations at each stage.
A refusal can be appropriate and still leave a gap
The checker stayed within its declared scope. It did not invent retrieval or recommendation findings from evidence it was not designed to assess. That was appropriate behaviour. It also meant that 115 definite key determinations were unavailable from the checker’s output. I would not add those 115 to the one comparable-set disagreement and call them classification errors. I would report them as a coverage boundary.
Keep uncertain results out of the zero column
AI citation tracking reports become misleading if every unresolved state is stored as zero. Zero can imply that something was checked and found absent. An unresolved result says the available evidence or the instrument did not establish it. The distinction matters for the AI visibility denominator — treating NOT_ESTABLISHED as zero inflates the count a rate divides by.
| Result label | What the report should preserve |
|---|---|
| OBSERVED | The evidence supports presence under the coding rules |
| NOT_OBSERVED | The evidence supports absence under the coding rules |
| NOT_ESTABLISHED | The state remains unresolved; record why |
| NOT_APPLICABLE | The assessment does not apply; keep it separate |
In E108, case G040 contained a truncated response and was ruled NOT_ESTABLISHED. The ten reporting-integrity cases (G041–G050) fell outside the checker’s scope and were treated as NOT_APPLICABLE. All 50 cases remained part of the returned run. The agreement denominator was still 79 comparable state determinations.
What I would ask for in an AI citation tracking report
AI citation tracking reports should pair the agreement score with coverage information. As of October 2026, I have not seen a commercial AI visibility tool that publishes both side by side. Here is the reporting checklist I use when reading a citation score:
| Report field | What I would want to see |
|---|---|
| Outcome definition | What counts as a mention or citation |
| Measurement scope | Which states the tool can assess |
| Unit of analysis | Whether the count refers to cases, queries, responses or state determinations |
| Denominator | Which observations entered each calculation |
| Positive-case count | How much positive evidence sits behind the detection claim |
| Unresolved results | Counts and reasons, retained separately from absence |
| Validation setting | Static supplied cases, API responses or consumer-interface captures |
| Version | The instrument and benchmark versions used |
For a practical GEO audit, I use this checklist to check whether the report can answer the question the client is asking. If the question concerns citations, show the visible attribution. If it concerns retrieval without citation, explain what evidence establishes retrieval. A field labelled “visibility” needs a definition before its percentage can be interpreted.
What AI citation tracking still needs to test
AI citation tracking validation is not complete after one static benchmark. E108 tested one instrument version against one static benchmark. It did not establish performance on current consumer search interfaces, and it was not an independent replication of the checker by another team.
The next evaluation I would want would include more citation-positive cases and additional entity-identity cases. For live use, it would also need captures from the intended platform, collected under recorded conditions. Changes to the checker could alter both its agreement and its coverage. A version that assesses more states would need validation for those states, rather than inheriting the old score.
The frozen E108 result remains evidence about that run. A claim about the checker’s present performance needs evidence from the version being used now. The ten measurement variables that change between runs are a useful starting checklist for what to hold constant.
Frequently asked questions
Does 98.7% agreement mean the AI citation tracking checker is 98.7% accurate?
AI citation tracking agreement and accuracy are different claims. The 98.7% score means the checker agreed with the USO Gold key on 78 of 79 comparable mention and citation determinations. It does not establish overall AI visibility accuracy or live-search performance.
Can an AI citation tracking tool establish that a page was retrieved?
E108’s checker could not. Retrieval was outside its declared scope. A tool making a retrieval claim needs evidence and validation for that stage. The distinction between retrieval and citation maps to different pipeline stages with different failure modes.
Why does the positive-case count matter in AI citation tracking?
A combined score can contain many negative cases. E108’s comparable citation set had three positives and 36 negatives, so the positive examples covered a limited range. A high score on a negative-heavy set proves the tool identifies absence correctly. It says less about how it handles a diverse set of positive detections.
Is NOT_ESTABLISHED the same as no citation?
No. NOT_ESTABLISHED records an unresolved state. The reason might be insufficient evidence or a limitation in what the instrument can assess. Scoring it as zero corrupts the denominator and hides the measurement gap.
Were the 115 out-of-scope determinations errors?
They were coverage gaps against definite key judgements, not disagreements within the checker’s comparable mention-and-citation set. The checker had declined those states because retrieval, inclusion, recommendation, and adverse treatment fell outside its scope.
Was E108 preregistered?
No. The run took place on 20 August 2026 and was registered post-hoc on 24 August. Controls were declared beforehand in correspondence, but no preregistration was deposited before execution.
Key GEO Lab Takeaway
- AI citation tracking agreement scores apply to the tested task. E108’s 98.7% covered 79 mention and citation determinations on a static benchmark — not live-search accuracy, not retrieval, not recommendation.
- The G012 identity disagreement showed that matching a name is not enough. Entity identity requires evidence that the name refers to the intended entity, not a homonym.
- The 115 out-of-scope definite determinations are a coverage boundary, not classification errors. An AI citation tracking tool that reports only what it can assess is behaving correctly — but the coverage gap must be visible in the report.
- Before acting on an AI citation tracking score, check whether the outcome you care about was actually measured. If the question concerns retrieval without citation, a mention-and-citation checker cannot answer it.
Ready to apply this? Run the 10 GEO audit checks on your pages, and use the reporting checklist above to evaluate any AI citation tracking score before making decisions on it.
Questions? Contact The GEO Lab.