Why the ChatGPT App and AI Visibility Tools Disagree — Same Questions, App vs API (2026)

Key takeaway: We entered the same four Korean questions into the ChatGPT app and the OpenAI API and compared the brands named in the answers. App–API brand-list overlap was 0.18–0.30 per question, lower than the overlap between repeated runs of the same path (API 0.39–0.53, app 0.46 on average). Changing the API side across six conditions kept overlap with the app at 0.17–0.26. The app screen and API answers are two different observations of the same service, so read an AI visibility tool's results as a trend across repeated runs under the same conditions, not against a single app check.

---

Three-line summary

• The app and the API return different brand lists for the same question. The gap was larger than run-to-run variation, and it went both ways — neither side was simply stricter.
• Changing API settings did not bring results closer to the app. All six conditions with a model swap or setting changes stayed between 0.17 and 0.26.
• The gap depends on the category. In categories with established leading brands, app and API lists were nearly the same; in categories with many new vendors, they differed widely.

For the same questions, app–API brand-list overlap was 0.18–0.30

When the same Korean questions were entered twice in the ChatGPT app and three times in the API, the brand lists overlapped far less between the two paths than within each path.

On October 25, 2026, RanketAI entered four questions — representative, recommendation, comparison and alternative — about the "AI search visibility diagnostics" category into both paths. The app was used signed out with a new chat per question, twice per question (8 answers). The API used web search with the search location set to Korea, three times per question (the default condition plus two request-setting variants, 12 answers). Brand lists were compared with overlap (the Jaccard index): the share of brands that appear in both answers out of all brands in either answer. 1 means identical lists; 0 means no brand in common.

Question	App ↔ API	API vs API	App vs app
Representative brands	0.24	0.42	0.38
Recommendation	0.18	0.39	0.44
Comparison	0.30	0.41	0.27
Alternatives	0.25	0.53	0.75
Average	0.24	0.44	0.46

Even repeated runs of the same path overlap only about half the time. That AI answers change from run to run is already well documented.

"ChatGPT and Google Search AI Overviews each returned the same brand list less than 1% of the time" — Search Engine Journal, report on the SparkToro and Gumshoe study (2026-01-30)

But the overlap between app and API (0.24 on average) was about half the overlap between repeated runs of one path (0.44–0.46). The difference between the two paths is larger than run-to-run noise — it comes from the paths themselves. Repeating a question many times makes each path's result more precise, but it does not close the gap between paths.

Six API conditions kept overlap with the app at 0.17–0.26

We tried to move the API closer to the app by changing the model and request settings, but no condition approached the within-path level (around 0.4).

On the same day, with the same questions, we repeated the API side under different conditions: one model swap and four request-setting changes. The specific models and setting values are part of our measurement configuration and are not disclosed.

API condition	Overlap with the app	Brands per answer
Default	0.26	8.0
Setting A	0.25	7.3
Setting B	0.21	6.5
Setting C	0.17	6.3
Setting D	0.25	6.2
Model B	0.24	8.6
(Reference) App	0.46 (app vs app)	9.4

Search behavior differs even within the API, by path and model. OpenAI's documentation describes API web search as a tool the model uses when it decides to, and search models as always searching before they answer.

"Chat Completions search models always search before responding; Responses search is a tool" — OpenAI API documentation, Web search

The documentation does not say that API search works the same way as search in the consumer ChatGPT app. In our measurement, no model or setting change moved the API closer to the app. Adjusting API conditions alone does not reproduce what the app shows.

The gap went both ways — the API is not simply stricter

Some brands appeared mostly in the app, and others appeared mostly in the API.

Brand (anonymized)	Out of 8 app answers	Out of 12 API answers
Brand A (new)	8	0
Brand B	6	2
Brand C	3	12
Brand D	1	8
Brand E	0	6
Brand F (shared)	8	11

Brand A appeared in every app answer and in no API answer. Brand C was the reverse: every API answer, but only 3 of 8 app answers. It is more accurate to see the app and the API as observing the same service from different angles. Looking at one side alone makes some brands look less visible than they are on screen, and others more visible.

74% of API answer links pointed to company homepages

API answers tended to build their lists from vendors whose homepages appeared in search results, while app answers often relied on comparison and round-up articles.

Of 103 links in 12 API answers — the same questions repeated three times under the default condition on the same day — 76 (74%) pointed to company homepages, and 12 to articles or comparison posts. App answers to the same questions often cited round-up articles covering several vendors and listed the vendors those articles described. A new brand whose homepage rarely shows up in search results but that is often covered in comparison articles, like Brand A above, appears often in the app and rarely in the API. Brand C, whose homepage does show up, is the opposite.

How the app searches and refines answers internally is not described in public documentation. The explanation above comes from comparing answers and their sources, and the app's ability to use personal memory and conversation context may also contribute to the gap.

In categories with established leaders, the app and the API nearly matched

The app and API lists differed widely in a category with many new vendors, but were nearly the same in categories with established leading brands.

In the same period, we entered two questions each from the customer support software and collaboration tool categories into the app again and compared them with three API runs from the October 2026 benchmark.

Category	App (1 run)	API (3 runs)
Customer support — representative brands	Leading brand listed first in Korea	First brand in all 3 runs
Customer support — recommendation	Leading brand listed first	First brand in all 3 runs
Collaboration — representative and recommendation (one Korean brand)	Not mentioned in either run	Not mentioned in any of 6 runs

The rest of the brand lists were also nearly the same across the two paths. These categories have well-known leaders, and their homepages were used as sources by both paths. Even with the same measurement method, how far results differ from the app depends on how mature the category is. The more vendors there are and the less settled the leaders, the more you need to allow for differences between the app and tool results.

Collecting app screens automatically has limits too

If you want results that match the app, you could observe the app screen — but that approach is also affected by sign-in state, location and personalization, and it is constrained by the terms of service.

OpenAI's Terms of Use list automated extraction among the things users may not do.

"Automatically or programmatically extract data or Output" — OpenAI Terms of Use, effective December 11, 2024, "What you cannot do"

So AI visibility tools either receive answers through the API or observe the app screen in other ways. Either way, the result is "the result under these conditions" and does not represent every user's screen. In our measurement, entering the same question twice in the same app produced lists that overlapped only 0.46 on average. When choosing a tool, it is more useful to check whether its collection method and conditions are disclosed than to rely on accuracy claims.

How to read AI visibility tool results

A tool's result is not something to compare against a single app check; it is a baseline for judging change across repeated runs under the same conditions.

1. Do not compare a score one to one with a single app result. App answers overlap only about half the time with each other, so one different result is not evidence. To check the app, follow How to Check If ChatGPT Mentions Your Brand: repeat the same question at least three times and record "how many out of 3."
2. Judge by the trend under the same conditions. Change can only be judged by comparing results with the same questions, AI services and measurement method. How much results move on re-measurement is covered in Monthly Change Was Smaller Than Same-Day Measurement Noise.
3. Assume competitor lists can differ by path. The competitors a tool shows are the brands that appeared alongside yours on that measurement path. In a new category, check the app with the same questions as a complement.
4. Consider category maturity. In categories with established leaders, tool results and the app are likely to be similar; in categories with many new vendors, the gap can be large.
5. Check five things when choosing a tool: the collection path (API or screen observation), how many times each question is repeated, the measurement date, whether question wording is disclosed, and whether full answer text is provided.

RanketAI's AI brand visibility analysis measures answers received by sending the same questions to each AI service's API, and shows the question wording and the full answer text on the results screen. Measurement criteria and limits are published in our methodology. When comparing diagnosis, analysis and measurement results with what you see in the app, we recommend checking the five points above in that order.

Limits of this measurement

This is one category, one point in time and one app environment, so the size of the figures cannot be generalized.

• The sample is small. Four questions, 8 app answers and 12 API answers (4–12 per condition). The control categories used two questions each and one app run.
• The app is one environment. Questions were entered signed out on one PC. App results can differ by sign-in state, location and device.
• Brand judgment can contain rule errors. Brands were judged with a fixed name dictionary, excluding source chips and link cards next to answers. Large differences such as Brand A's 8 vs 0 and Brand C's 3 vs 12 cannot be explained by judgment errors.
• This records the services as of October 2026. The ChatGPT app and API keep changing, so the same results may not hold.

FAQ

Related reading

• How to Check If ChatGPT Mentions Your Brand: 5 Steps — how to check the app yourself and record results
• What Does a GEO Score Measure? — how prompts, models and judges change the number
• Don't Measure AI Visibility Once — statistical criteria for repeated measurement
• Monthly Change Was Smaller Than Same-Day Measurement Noise — separating re-measurement variation from real change
• How to Vet a GEO Agency — seven questions that check measurement methods