If you run any form of AI-visibility tracking — a SaaS tool, a spreadsheet of ChatGPT answers, a monthly “citation score” — a preprint published on October 11, 2026 deserves your attention. Researchers from the National University of Singapore and Xi’an Jiaotong University tested whether the two cheap stand-ins for ChatGPT — Google’s organic results and OpenAI’s own Responses API — reproduce the pages ChatGPT actually cites. Combined, they matched about 26.2% of the pages ChatGPT cited in their GPT-5.5 Instant tests. The other 74% of citations were invisible to both proxies. That finding does not indict any specific tracking tool — the paper tested none — but it does indict a category of measurement: anything that infers ChatGPT citations from an API call or a Google SERP instead of observing the actual answers.

Before the numbers, the labels this piece is not allowed to skip: this is a preprint, not a peer-reviewed paper (arXiv does not peer-review), built on 80 questions run against four ChatGPT settings, with most data collected between March 29 and July 29, 2026 — a window that predates the current model lineup. It is the best public evidence we have on this specific question, and it is still preliminary. The details below come from the paper itself (https://arxiv.org/abs/2610.15382, Yunxuan Jiang, Shiqi Zhang, Anqi Xing, Xiangyu Chang, Xiaokui Xiao, submitted October 11, 2026) and from Search Engine Journal’s report on it (Matt G. Southern, October 18, 2026: https://www.searchenginejournal.com/openais-api-only-partly-matched-chatgpts-citations-paper-finds/593304/) — the SEJ article carries the granular statistics we cite here.

What the study did

The researchers recorded the full data flow of ChatGPT’s web application during real, signed-in sessions — the searches ChatGPT performed, the candidate pages returned, and the pages it finally selected as citations. They asked 80 questions (40 English, 40 Chinese, drawn from Search Arena, a public database of real user queries, spanning recommendations, fact lookups, synthesis and analysis), and ran each question 20 times on GPT-5.3 Instant and GPT-5.5 Instant (two older settings got fewer runs because they were decommissioned mid-collection).

For the stand-ins: up to five pages of Google’s organic results per question via SerpAPI (about 43 results on average), and 20 queries each to the OpenAI Responses API (gpt-5.3-chat-latest and gpt-5.5) with web search enabled — all collected within three days of the ChatGPT runs for the same query. The comparison was done at three levels of strictness: exact URL, hostname (site), and entity.

What it found

Every figure below is from the paper as reported by SEJ:

  • Google + API together matched 26.2% of the pages GPT-5.5 Instant cited, and 28.3% for GPT-5.3 Instant.
  • At the site level the match was better: 48.7% of cited sites for GPT-5.5, 50.3% for GPT-5.3. Sites match far more often than exact pages — a proxy that “finds the same website” is not the same as “finds the same citation.”
  • Google alone matched only 6.4–9.4% of cited pages (and 22.8–26.1% of cited sites). The API did better on its own — its full candidate list accounted for 22.5–22.9% of cited pages.
  • Run-to-run variation is large: 44.5% of the distinct pages ChatGPT returned for a given question appeared in only one of the 20 GPT-5.5 Instant runs (40.8% for GPT-5.3). A single-run measurement is, on this evidence, close to noise.
  • ChatGPT never searched using the exact query wording. In both GPT-5.3 Instant and GPT-5.5 Instant runs with visible searches, ChatGPT transformed the question into its own search terms in every case. Only the older GPT-5.2 Instant used the exact wording, in 5.5% of runs.
  • Language drift is a real factor: 69.6% of runs on Chinese questions in GPT-5.5 Instant searched in English (67.0% in GPT-5.3). If your site has no English presence, a Chinese-language buyer question may never touch it.
  • Model identity matters mid-study. Two collection sessions switched to a different model after 14 and 16 runs, and the authors therefore recommend checking which model generated each response — a reminder that “ChatGPT” is not a stable measurement target.

Two honesty notes. First, the paper’s own scope: it is an observational study of four settings on 80 questions — directionally strong, not a census. Second, its figures describe the models current in March–July 2026; per SEJ, every tested setting predates the newer ChatGPT lineup, so treat the exact percentages as a snapshot, not a constant.

Why this breaks a whole class of AI-visibility numbers

Most AI-visibility tooling — and most internal “let’s just ask the API” setups — works by inference rather than observation, because observing the real product at scale is slower and harder to automate. The study quantifies how much the inference costs:

  1. Page-level mismatch. A Google-rank-based or API-based tracker will find the sites behind ChatGPT citations roughly half the time, but the exact pages only about a quarter of the time. For a B2B SaaS, the page is the unit that matters: being cited for your pricing page versus your marketing homepage is a different business fact, and the proxy cannot tell them apart.
  2. Query rewriting breaks keyword mirroring. ChatGPT never searches your keyword. It rewrites. A tracker that measures “does page X appear in the top results for [exact keyword], via the API” is measuring a query ChatGPT never ran.
  3. Single-run scores are not scores. If 44.5% of distinct pages appear in only one of 20 runs, a tool that asks each query once and reports a stable “citation score” is reporting the idiosyncrasies of one run. The paper’s authors explicitly recommend re-running prompts and treating run-to-run variation as a first-class metric.
  4. Model labels are part of the data. Because sessions in the study switched models mid-collection, and because the industry’s own monitoring data shows citation volatility spiking around model changes (SEJ cites Sistrix finding day-to-day variation in German-language ChatGPT citations jump to 47% around the GPT-5.5 change, against a normal 1–2%), any number that does not record which model and setting produced each answer is un-reproducible by construction.

None of this is an accusation at any vendor. The paper did not test any commercial tracking tool, and a well-built tracker will know its own proxy’s error bars. The practical consequence is that two trackers reporting different “ChatGPT citation” numbers for the same brand are not necessarily wrong — they may be measuring two different stand-ins, at two different strictness levels, with different run counts. That is the question the study hands you.

What “good methodology” looks like now

This is the checklist the study’s findings imply, and the one we hold any measurement — including our own — to:

  • Observe the surface, don’t just proxy it. Where the goal is “what does a buyer see,” the ground truth is the answer in the product. API and Google results are fine as coverage estimates and regression checks — label them as such, and never publish them as the ChatGPT citation itself.
  • Multi-run, always. Multiple runs per query, with the run-to-run overlap reported alongside the headline. A number without a variance is a guess.
  • Model-labeled. Every recorded answer carries the model and setting that produced it. “GPT-5.5 Instant, run 3 of 20, [date]” — or the number is unusable for trend analysis.
  • Two strictness levels. Measure at page level and site level, and say which one the headline uses. Site-level numbers flatter; page-level numbers sting; both are true.
  • Re-test after every model change. The study’s window is March–July 2026 and the model lineup has already moved past it. Methodology should include a documented re-baseline when the underlying model changes.

These are not exotic requirements. They are what separates a measurement from a screenshot. We hold our own runs to the same bar — query sets, surfaces, run counts, strictness levels, what the monthly number claims and does not — and the general vocabulary for reading any AI-visibility number sits in AI visibility metrics, the basics. If you want the retrieval mechanics that explain why the proxies diverge from the product in the first place, how AI answers are generated is the companion piece.

If your current dashboard’s numbers were produced by a single-run, proxy-based, model-unnamed process, the study’s polite version of “your numbers may be wrong” is: your numbers are a specific, under-documented estimate of a moving target. The fix is not a better tool by itself — it is a documented method, run on your own money queries, that says what it did. That is what the methodology page is for, and it is also the first thing to ask any vendor, including us, before you trust a score.