Measuring AI citations means running a fixed set of questions, several times, against a model with web search, and counting how many answers mention your brand in the text and list your domain among the cited sources. The answer changes from one run to the next, so a question asked once tells you nothing. The same question asked ten times, every month, becomes a series you can track.
Why this matters and what makes an AI recommend a company is covered in the GEO pillar post and in How to appear in ChatGPT, which suggests the manual test: ask. This post automates that test.
Know what the script measures, and what it does not
The script uses the web search tool in OpenAI's Responses API. It does not query the ChatGPT app. The app has memory, personalization and whatever model is live that day. The API answers with the model and options you choose. This post uses the API as an indicator: if it neither finds nor cites you with search on, the app is unlikely to do better. The reverse is not guaranteed.
For each answer, the script records:
mentioned_brand: the name (or an alias) appears in the text, ignoring accents and case.cited_domain: some link cited in the text (aurl_citationannotation) is from your domain.domain_in_sources: your domain is in the full list of sources consulted, even if it was not cited.competitors: which competitors from your list were mentioned.searches: how many searches the answer made, which is what you pay for.
The difference between domain_in_sources and cited_domain is the most useful one. Being in the sources and not being cited means the model read your page and preferred another one.
Write the questions in a YAML file
brand: Capiva
aliases: ["Capiva", "capiva.tech"]
domain: capiva.tech
competitors: ["Competitor A", "Competitor B"] # example: replace with yours
country: BR # two-letter ISO code of the market you want to measure
repetitions: 3
questions:
- Which AI consultancies in Brazil help companies put agents into production?
- Who does AI visibility (GEO) audits for websites in Brazil?
- What is an AI harness and who writes about it?Write questions a customer would ask, without your company's name. "What does Capiva do?" only measures whether the model finds your site, not whether it recommends you. The competitor list shows who appears in your place; the names above are placeholders. If you do not yet know who your competitors are in the answers, run it once with an empty list and read the URLs in the cited_urls column.
Understand the API response
With search on, the Responses API output has one web_search_call item per search action and one message item with the text. The citations are in the text annotations:
{
"output": [
{
"type": "web_search_call",
"status": "completed",
"action": {"type": "search", "query": "...", "sources": [{"type": "url", "url": "https://..."}]}
},
{
"type": "message",
"content": [{
"type": "output_text",
"text": "...",
"annotations": [
{"type": "url_citation", "url": "https://...", "title": "...", "start_index": 0, "end_index": 10}
]
}]
}
]
}This format comes from the docs; it is not a captured response. The docs show query in the singular, but also say the search action usually carries queries, a list, and the SDK (openai 3.29.0) accepts both fields. Reasoning models can also return open_page and find_in_page actions, which have a url and no sources. The script reads none of these fields, so it works with any variant.
Three details from the docs that the script uses:
sourcesonly comes back if you ask for it, withinclude=["web_search_call.action.sources"]. It is the full list of URLs consulted, which is usually longer than the list of citations.user_locationwithcountry(a two-letter ISO code, such asBR) brings the search results closer to what someone asking from that country sees.- With
tool_choiceset toauto, the model decides whether to search. The script usesautoby default, because that is the normal behavior, and has a--force-searchoption that sendsrequired.
Write the script
#!/usr/bin/env python3
"""Measure whether the OpenAI model, with web search, mentions and cites your company.
Usage:
python3 measure_citations.py monitor.yaml --estimate # only shows how many calls it would make
python3 measure_citations.py monitor.yaml # runs and writes results.csv
"""
import argparse
import csv
import os
import re
import sys
import unicodedata
from datetime import datetime, timezone
from urllib.parse import urlparse
import yaml
from openai import OpenAI
SEARCH_COST_USD = 0.01 # US$ 10 per 1,000 web search calls, not counting tokens
def normalize(text: str) -> str:
no_accents = unicodedata.normalize("NFKD", text).encode("ascii", "ignore").decode()
return no_accents.lower()
def mentions(text: str, names: list[str]) -> bool:
target = normalize(text)
return any(re.search(rf"\b{re.escape(normalize(n))}\b", target) for n in names)
def in_domain(url: str, domain: str) -> bool:
host = (urlparse(url).hostname or "").removeprefix("www.")
return host == domain or host.endswith("." + domain)
def analyze(response, cfg: dict) -> dict:
citations, sources, searches = [], [], 0
for item in response.output:
if item.type == "web_search_call":
searches += item.action.type == "search" # only "search" is billed
sources += [s.url for s in (getattr(item.action, "sources", None) or [])]
elif item.type == "message":
for part in item.content:
for a in getattr(part, "annotations", None) or []:
if a.type == "url_citation":
citations.append(a.url)
text = response.output_text
return {
"searches": searches,
"mentioned_brand": mentions(text, cfg["aliases"]),
"cited_domain": any(in_domain(u, cfg["domain"]) for u in citations),
"domain_in_sources": any(in_domain(u, cfg["domain"]) for u in sources),
"competitors": ";".join(
c for c in cfg.get("competitors", []) if mentions(text, [c])
),
"n_citations": len(citations),
"cited_urls": " ".join(citations),
}
def ask(client: OpenAI, model: str, question: str, country: str, force: bool):
return client.responses.create(
model=model,
tool_choice="required" if force else "auto",
tools=[
{
"type": "web_search",
"user_location": {"type": "approximate", "country": country},
}
],
include=["web_search_call.action.sources"],
input=question,
)
def main(argv=None, client: OpenAI | None = None) -> int:
ap = argparse.ArgumentParser()
ap.add_argument("config")
ap.add_argument("--model", default=os.environ.get("MODEL", "gpt-5.5"))
ap.add_argument("--output", default="results.csv")
ap.add_argument("--estimate", action="store_true")
ap.add_argument("--force-search", action="store_true", help="tool_choice=required")
args = ap.parse_args(argv)
with open(args.config, encoding="utf-8") as f:
cfg = yaml.safe_load(f)
total = len(cfg["questions"]) * cfg.get("repetitions", 3)
print(
f"{total} responses. With one search each: US$ {total * SEARCH_COST_USD:.2f} + tokens "
"(the model may search more than once, or not at all)"
)
if args.estimate:
return 0
client = client or OpenAI() # reads OPENAI_API_KEY from the environment
new = not os.path.exists(args.output)
with open(args.output, "a", newline="", encoding="utf-8") as f:
w, total_searches = None, 0
for question in cfg["questions"]:
hits = 0
for run in range(1, cfg.get("repetitions", 3) + 1):
r = ask(
client,
args.model,
question,
cfg.get("country", "BR"),
args.force_search,
)
row = {
"when": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"model": args.model,
"question": question,
"run": run,
**analyze(r, cfg),
}
hits += row["mentioned_brand"]
total_searches += row["searches"]
if w is None:
w = csv.DictWriter(f, fieldnames=list(row))
if new:
w.writeheader()
w.writerow(row)
print(f"{hits}/{cfg.get('repetitions', 3)} mentions | {question}")
print(
f"{total_searches} billed searches: US$ {total_searches * SEARCH_COST_USD:.2f} + tokens"
)
return 0
if __name__ == "__main__":
sys.exit(main())The default model is gpt-5.5, which the docs recommend for new search integrations. Change it with --model or the MODEL variable. The CSV opens in append mode, so each run adds rows with the date and the model, and the history stays in a single file.
Name matching uses a word boundary after stripping accents. "Capivá" counts as Capiva; "capivara" does not. If your company's name is a common word, add more specific aliases, like the domain.
Test it without spending anything
The OpenAI client accepts an httpx http_client, and httpx has a fake transport, MockTransport. With it, the test goes through the real SDK, with the real parsing, with no network and no key:
"""Run the script end to end against a fake API (no network, no cost)."""
import csv
import json
import httpx
from openai import OpenAI
import measure_citations as m
REQUESTS = []
def fake_response(text, citations, sources):
return {
"id": "resp_test",
"object": "response",
"created_at": 0,
"model": "gpt-5.5",
"status": "completed",
"parallel_tool_calls": True,
"tool_choice": "auto",
"tools": [],
"output": [
{
"type": "web_search_call",
"id": "ws_1",
"status": "completed",
"action": {
"type": "search",
"query": "x",
"sources": [{"type": "url", "url": u} for u in sources],
},
},
{
"type": "message",
"id": "msg_1",
"status": "completed",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": text,
"annotations": [
{
"type": "url_citation",
"start_index": 0,
"end_index": 1,
"url": u,
"title": "t",
}
for u in citations
],
}
],
},
],
}
RESPONSES = iter(
[
fake_response(
"Capivá and Competitor A do that.",
["https://www.capiva.tech/en/diagnostic"],
[],
),
fake_response(
"I recommend Competitor B.",
["https://example.com"],
["https://capiva.tech/en/blog/x"],
),
fake_response("A capybara (capivara) is a rodent.", [], []), # must not count as a mention
]
)
def fake_api(request: httpx.Request) -> httpx.Response:
REQUESTS.append(json.loads(request.content))
return httpx.Response(200, json=next(RESPONSES))
def test_end_to_end(tmp_path):
cfg = tmp_path / "m.yaml"
cfg.write_text(
"aliases: [Capiva, capiva.tech]\ndomain: capiva.tech\n"
"competitors: [Competitor A, Competitor B]\ncountry: BR\nrepetitions: 3\n"
'questions: ["Who does GEO in Brazil?"]\n'
)
client = OpenAI(
api_key="test",
http_client=httpx.Client(transport=httpx.MockTransport(fake_api)),
)
output = tmp_path / "r.csv"
assert m.main([str(cfg), "--output", str(output)], client=client) == 0
request = REQUESTS[0]
assert request["tools"][0]["type"] == "web_search"
assert request["tools"][0]["user_location"]["country"] == "BR"
assert request["include"] == ["web_search_call.action.sources"]
assert request["tool_choice"] == "auto"
rows = list(csv.DictReader(output.open()))
assert [r["mentioned_brand"] for r in rows] == ["True", "False", "False"]
assert [r["cited_domain"] for r in rows] == ["True", "False", "False"]
assert [r["domain_in_sources"] for r in rows] == ["False", "True", "False"]
assert rows[0]["competitors"] == "Competitor A"
assert [r["searches"] for r in rows] == ["1", "1", "1"]$ pip install openai httpx pyyaml pytest
$ pytest -q
.
1 passed in 0.58sThe test checks what goes into the request (the web_search tool, the country, include, tool_choice) and what comes out in the CSV, including the "capivara" case, which must not count as a mention.
A second test file covers the variations the docs describe and the errors you will hit on the first run: open_page and find_in_page actions (they are not billed, so they do not count as searches), queries in the plural, an answer with no search, a feed source without a url, a wrong key (401), exhausted credit (429, code credit_balance_exhausted in the error list) and no network:
"""Response formats and errors the OpenAI docs describe (no network, no cost)."""
import httpx
import pytest
from openai import APIConnectionError, AuthenticationError, OpenAI, RateLimitError
import measure_citations as m
CFG = {"aliases": ["Capiva", "capiva.tech"], "domain": "capiva.tech", "competitors": []}
def client(handler):
return OpenAI(
api_key="test",
max_retries=0,
http_client=httpx.Client(transport=httpx.MockTransport(handler)),
)
def response(output):
return {
"id": "resp_1", "object": "response", "created_at": 0, "model": "gpt-5.5",
"status": "completed", "parallel_tool_calls": True, "tool_choice": "auto",
"tools": [], "output": output,
}
MSG = {
"type": "message", "id": "msg_1", "status": "completed", "role": "assistant",
"content": [{
"type": "output_text", "text": "See capiva.tech.",
"annotations": [{"type": "url_citation", "start_index": 0, "end_index": 5,
"url": "https://capiva.tech/en/blog/x", "title": "t"}],
}],
}
def test_reasoning_model_actions():
# search with "queries" (plural), open_page and find_in_page, as in the docs
out = [
{"type": "web_search_call", "id": "ws_1", "status": "completed",
"action": {"type": "search", "queries": ["geo brazil", "geo consultancy"],
"sources": [{"type": "url", "url": "https://capiva.tech/en"}]}},
{"type": "web_search_call", "id": "ws_2", "status": "completed",
"action": {"type": "open_page", "url": "https://capiva.tech/en"}},
{"type": "web_search_call", "id": "ws_3", "status": "completed",
"action": {"type": "find_in_page", "url": "https://capiva.tech/en", "pattern": "GEO"}},
MSG,
]
c = client(lambda r: httpx.Response(200, json=response(out)))
row = m.analyze(m.ask(c, "gpt-5.5", "q", "BR", False), CFG)
assert row["searches"] == 1 # open_page and find_in_page are not billed
assert row["domain_in_sources"] and row["cited_domain"] and row["mentioned_brand"]
def test_response_without_search():
c = client(lambda r: httpx.Response(200, json=response([MSG])))
assert m.analyze(m.ask(c, "gpt-5.5", "q", "BR", False), CFG)["searches"] == 0
def test_feed_source_without_url():
# the docs say third-party feeds (oai-sports etc.) show up in sources, but do not
# show the object; this format without "url" is an assumption, to make sure the script does not break
out = [{"type": "web_search_call", "id": "ws_1", "status": "completed",
"action": {"type": "search", "query": "x",
"sources": [{"type": "api", "name": "oai-sports"},
{"type": "url", "url": "https://capiva.tech"}]}}, MSG]
c = client(lambda r: httpx.Response(200, json=response(out)))
assert m.analyze(m.ask(c, "gpt-5.5", "q", "BR", False), CFG)["domain_in_sources"]
def error(status, code):
return lambda r: httpx.Response(status, json={"error": {"message": code, "type": code, "code": code, "param": None}})
def test_invalid_key():
with pytest.raises(AuthenticationError):
m.ask(client(error(401, "invalid_api_key")), "gpt-5.5", "q", "BR", False)
def test_no_credit():
with pytest.raises(RateLimitError):
m.ask(client(error(429, "credit_balance_exhausted")), "gpt-5.5", "q", "BR", False)
def test_no_network():
def fail(r):
raise httpx.ConnectError("no network")
with pytest.raises(APIConnectionError):
m.ask(client(fail), "gpt-5.5", "q", "BR", False)$ pytest -q test_documented_formats.py
......
6 passed in 0.64sThe max_retries=0 in the test client is there because the SDK retries 429 and 5xx responses twice on its own before raising the error. Outside the test, a 429 for lack of credit takes about 1.5 seconds to show up, and it looks like this:
openai.RateLimitError: Error code: 429 - {'error': {'message': ..., 'code': 'credit_balance_exhausted', ...}}Both tests run the script against responses built from the docs. I did not run it against the real API, so the outputs in this post are examples in the documented format, not measurements.
Estimate the cost and run it
$ python3 measure_citations.py monitor.yaml --estimate
9 responses. With one search each: US$ 0.09 + tokens (the model may search more than once, or not at all)According to the pricing table, web search costs US$ 10.00 per thousand calls, plus the tokens of the search content, billed at the model's price. The docs say every search action is billed, so an answer that searches three times costs three searches. That is why the script counts the searches and adds them up at the end. The tokens depend on the model and the length of the answers, and they are not in the estimate.
Once the estimate looks acceptable:
export OPENAI_API_KEY=...
python3 measure_citations.py monitor.yamlRead the results
#!/usr/bin/env python3
"""Summarize results.csv: mention and citation rate per question and per month."""
import csv
import sys
from collections import defaultdict
groups = defaultdict(lambda: [0, 0, 0]) # runs, mentions, domain citations
with open(
sys.argv[1] if len(sys.argv) > 1 else "results.csv", encoding="utf-8"
) as f:
for row in csv.DictReader(f):
g = groups[(row["when"][:7], row["question"])]
g[0] += 1
g[1] += row["mentioned_brand"] == "True"
g[2] += row["cited_domain"] == "True"
print("month mention citation n question")
for (month, question), (n, m, c) in sorted(groups.items()):
print(f"{month} {m / n:7.0%} {c / n:8.0%} {n:>2} {question[:60]}")With the CSV the test produces (three runs of one question, one with a mention and a citation):
month mention citation n question
2026-10 33% 33% 3 Who does GEO in Brazil?Three runs are too few. With ten per question, a 30% rate starts to say something. Compare months, not single runs, and note when the model changed. The model column exists for that.
To run it every Monday at 9am, one crontab line is enough:
0 9 * * 1 cd /path/to/monitor && .venv/bin/python measure_citations.py monitor.yaml >> monitor.log 2>&1How we use it at Capiva
Before measuring citations, we check the technical prerequisite. The site's robots.txt allows every bot, including OAI-SearchBot. That is the bot OpenAI uses to show sites in ChatGPT search answers. According to the crawler docs, sites that block OAI-SearchBot do not appear in those answers. We have not yet run the script against the API with a key, so we have no measured rate for Capiva to show.
To read your result, look at the three columns together. An illustrative example, not a measurement: if the question about GEO in Brazil gives 30% mentions, 10% citations and domain_in_sources true in half the runs, the model often finds your site but cites another page instead. The work is then on the page (a direct answer, your own data, a visible date), not on indexing. If domain_in_sources is always false, the problem comes earlier: the bot cannot find or cannot read the site.
Troubleshooting checklist
- [ ] No searches (
searches= 0). Withtool_choiceset toauto, the model can answer without searching. Use--force-searchor rephrase the question to ask for current information. - [ ] Your site never shows up in the sources. Check that robots.txt does not block
OAI-SearchBot. Changes to robots.txt take about 24 hours to apply, according to OpenAI. - [ ] Mention without citation. The model knows the brand but cites another source. Look at which URLs show up in
cited_urls. - [ ] False positive mentions. The name is a common word. Replace it with more specific aliases.
- [ ]
Missing credentials.OPENAI_API_KEYis missing from the environment.--estimateworks without a key. - [ ]
AuthenticationError: Error code: 401. The key is wrong or was revoked. Check the variable, not the script. - [ ]
RateLimitError: Error code: 429. Withcredit_balance_exhausted, the prepaid credit ran out. With a request limit, wait and run it again. The rows written before the error stay in the CSV. - [ ] The result changes on every run. That is expected. Raise
repetitionsand compare rates, not answers. - [ ] Cost higher than the estimate. Answers with several searches, plus the tokens of the search content. Look at the search total at the end of the run.
Frequently asked questions
Does this measure what ChatGPT app users see?▼
Not exactly. It measures what the API model answers with search on. It is a useful approximation, not a replica of the app.
How many repetitions per question?▼
Start with 3 to test the flow and go up to 10 when you measure for real. The cost grows linearly.
Can I restrict the search to certain sites?▼
Yes, with filters.allowed_domains or blocked_domains (up to 100 domains). To measure visibility, do not filter. You want the search as it is.
Can I use it with other models?▼
The script is specific to OpenAI's Responses API. The logic (fixed questions, repetition, CSV, rates) applies to any API that returns citations.
Read next
- What is GEO / SEO for AI (the pillar)
- How to appear in ChatGPT
- How to create an llms.txt
- How to build an MCP server in Python and configure it in Claude Code, if you want to expose this measurement to an agent
Want the measurement done and interpreted for your site? Take the free GEO Diagnostic.