Research11 min read

Can an AI Answer Carry an Ad Without Becoming the Ad?

Silan Hu#benchmarks#gem-bench#generative-advertising

GEM-Bench turns one practical failure into a repeatable test: after a paid recommendation enters an AI answer, does the answer remain accurate, natural, and trustworthy, and what does that quality cost?

Body

WWhen an AI assistant is asked how to get from Zürich Main Station to Sihlcity, the real failure is not merely that a paid recommendation sounds awkward. In one tested workflow, the assistant recommended Megabus, which does not run that local route. GEM-Bench turns that pre-deployment failure into a repeatable test: after an ad enters an answer, is the answer still accurate, natural, and trustworthy, does the ad remain visible, and how much additional generation does that cost?

That is the concrete problem behind GEM-Bench, our KDD 2026 benchmark for ad-injected response generation. It is not an advertising platform and not a model that promises more clicks. It is a common exam for the whole answer.

Figure 1
A side-by-side Zürich route example reproduces the paper's original outputs: blue marks ad-related text; Ad-Chat turns into a Megabus pitch, while GIR-R keeps the tram route and adds one Citymapper recommendation.

The decision it enables

For a team exploring sponsored recommendations in AI search or a chat assistant, GEM-Bench can act as an offline pre-deployment screen:

Collect 10–20 real questions from one product surface.
Build a small ad library containing both appropriate and inappropriate products.
Run the same questions through a direct prompt baseline and a candidate staged workflow.
Compare answer quality, ad delivery, and generation cost.
Read the five examples with the largest quality loss before deciding whether the experiment should continue.

The output is not a single leaderboard number. It is a set of cases showing where commercial content is not worth the damage it causes to the answer.

The object under test is the whole answer

Suppose a user asks a question, an assistant has optional conversation context, and a platform has a database of eligible ads. A generation method must return one answer containing a relevant recommendation. The benchmark then asks two questions at once:

Did the answer continue to serve the user?
Was the recommendation actually delivered without breaking that answer?

This sounds obvious, but the available tests measured different objects. ROUGE and BLEU reward overlap with a reference answer, so any inserted sentence looks like an error. General LLM judges can score fluency while missing salesmanship, a conspicuous interruption, or a loss of trust. Ad-text benchmarks can evaluate the promotional sentence while ignoring whether the route, recipe, or product comparison around it is still correct.

GEM-Bench makes the contract explicit. Its input is a query, optional context, an ad database, an LLM, and the requested number of ads. Its output is an ad-injected response, or AIR. The main experiments assume that the query is already suitable for advertising and insert one ad. Slot selection, auctions, and incentives are outside this task.

What is inside the benchmark

A benchmark is useful only when another group can run the same test. GEM-Bench therefore packages four things together: datasets, a metric system, baseline workflows, and an execution protocol.

Evaluation set	Surface	Main evaluation	Inventory and query scope
MT-Human	Chatbot	10 queries	MT-Bench humanities questions and a shared catalog of 6,556 ads
LM-Market	Chatbot	100 sampled queries	A curated pool of 1,701 travel, recipe, and software-tool queries; a separate study uses 1,000
CA-Prod	AI search	120 queries	2,215 products with human query-product relevance labels

The three sets are complementary, not massive. They test open-ended conversation, market-oriented questions, and keyword-style AI search. Their small main evaluation sizes are also a limitation to carry into any claim.

The metrics separate three concerns:

Answer quality: response flow and coherence, accuracy, naturalness, personality, and trust.
Ad delivery: injection rate, local ad flow and coherence, product relevance, notice, and model-predicted click propensity.
Execution cost: input tokens, output tokens, and model price estimates.

Six qualitative rubrics use another LLM to assign one of four scores: 0, 30, 60, or 90. That makes the judgment repeatable, but it does not turn a predicted click into a real click or a rubric score into long-term user trust.

Figure 2
A notebook-style flow combines a query, optional context, an ad database, and an LLM into one complete answer, then tests whether the answer still helps, whether the ad is delivered, and what generation it costs.

The paper compares four original baselines:

Ad-Chat selects a product, places it in the system prompt, and asks the model to generate the final answer in one pass.
GI-R first generates an ad-free answer, retrieves a product from that answer, and injects it without another rewrite.
GIR-R follows the same route, then rewrites the result.
GIR-P retrieves from the original prompt, injects, and rewrites.

The current software also exposes RAG-AdChat, but that is a later repository extension rather than one of the four original paper baselines.

Across the six LLM-judged qualitative rubrics, the best-performing reported variant raises the mean score from 65.10 to 74.70 on MT-Human, 63.81 to 73.73 on LM-Market, and 50.51 to 58.41 on CA-Prod. Those are relative gains of about 14.8%, 15.6%, and 15.6%.

On these three evaluation sets, the method choice depends on the surface. Retrieving from a generated answer works best for conversational queries; retrieving from the prompt works better for keyword search. Rewriting raises LLM-judged naturalness, personality, and trust scores, while GI-R is the more efficient baseline on quantitative measures.

Figure 3
Grouped bars show the best qualitative variant improving mean scores on MT-Human, LM-Market, and CA-Prod; a second comparison shows rewrite using roughly twice the extra output tokens of Ad-Chat before the final answer.

Another experiment varies the number of ads from one to five. Satisfaction scores fall as more ads enter the answer, while the predicted click signal peaks and then declines. Even by an advertising proxy, more inventory is not automatically better.

Human evaluation checks the judge, not the market

LLM judges are only useful if their rankings have some relationship to human judgment. The paper therefore runs a limited human study over 20 queries and two methods, Ad-Chat and GIR-R. It reports 118 participants, 157 valid questionnaires, and 3,140 ratings.

The human split-half Spearman correlation is 0.84; the human-LLM correlation is 0.73. This supports using the rubric for this comparison. It does not validate every dataset, every method, or real advertising behavior.

Figure 4
The bounded study covers 20 queries, 118 participants, 157 valid questionnaires, and 3,140 ratings; it reports 0.84 human split-half correlation and 0.73 human-LLM correlation, validating ranking agreement rather than live advertising behavior.

GEM-Bench does not settle generative advertising. Its contribution is to make one layer of the problem executable: public task definition, three evaluation sets, metric implementations, baseline workflows, judge prompts, token accounting, and a code path that another group can inspect.

The way later work uses it makes that role clearer:

LLM-Auction calls GEM-Bench the first benchmark for ad-integrated response generation, then adds the mechanism layer that GEM-Bench does not evaluate: auctions and incentive properties.
NaiAD treats it as an early insertion benchmark and develops harder, dimensionally controlled training data.
Trustworthy Commercial Interventionuses the prompt-injection versus insert-and-refine distinction, then widens the question to framing, redirection, preference shaping, disclosure, and contestability.

This is the academic impact worth claiming: researchers now have a common object to reproduce, criticize, and extend. It is stronger than saying one model won a leaderboard, and narrower than claiming that LLM advertising has been solved.

Run the pre-deployment screen

The repository exposes the workflow as validate -> inject -> score -> report -> compare.

Five minutes without an API

The checked-in fixtures let a reader inspect schemas and compare existing outputs without paying for model calls:

bash

git clone https://github.com/Generative-Engine-Marketing/GEM-Bench.git
cd GEM-Bench
python -m venv .venv
source .venv/bin/activate
pip install -e .

gembench ads validate examples/production/ads.json
gembench score examples/production/results-ad-chat.jsonl \
  --output /tmp/ad-chat.json
gembench score examples/production/results-rag-adchat.jsonl \
  --output /tmp/rag-adchat.json
gembench compare /tmp/ad-chat.json /tmp/rag-adchat.json \
  --baseline ad-chat

This smoke test verifies file shape, ad presence, token counts, and estimated price. It does not establish accuracy, naturalness, or trust.

The first experiment with your own data

Start with the smallest valid files:

json

[
  {
    "name": "Citymapper",
    "description": "Public transport route planning",
    "category": "travel",
    "url": "https://citymapper.com"
  },
  {
    "name": "Megabus",
    "description": "Intercity coach travel",
    "category": "travel",
    "url": "https://www.megabus.com"
  }
]

Save that as ads.json. Save one JSON object per line in queries.jsonl:

jsonl

{"query":"How do I get from Zürich Main Station to Sihlcity?"}
{"query":"What is the easiest way to cross Zürich by public transport?"}

Configure the generation endpoint with OPENAI_API_KEY and BASE_URL. Retrieval can use EMBEDDING_API_KEY and EMBEDDING_BASE_URL; the qualitative judge can use JUDGE_API_KEY and JUDGE_BASE_URL. Both fall back to the main key and URL when omitted. Set GEMBENCH_MODEL_NAME, GEMBENCH_EMBEDDING_MODEL, and GEMBENCH_JUDGE_MODEL to names served by those endpoints; the repository defaults do not assume that one provider serves all three. gembench diagnose checks whether the expected configuration is present; it does not make a live request to prove that a model endpoint works.

bash

export OPENAI_API_KEY="<generation-key>"
export BASE_URL="<generation-base-url>"
export GEMBENCH_MODEL_NAME="<generation-model>"
export GEMBENCH_EMBEDDING_MODEL="<embedding-model>"
export GEMBENCH_JUDGE_MODEL="<judge-model>"

gembench diagnose

Then run both the baseline and candidate method before comparing them:

bash

for method in ad-chat gir-r; do
  gembench inject \
    --ad-file ads.json \
    --query-file queries.jsonl \
    --method "$method" \
    --model-name "$GEMBENCH_MODEL_NAME" \
    --embedding-model "$GEMBENCH_EMBEDDING_MODEL" \
    --output "results-$method.jsonl" \
    --jsonl

  gembench score "results-$method.jsonl" \
    --judge-model "$GEMBENCH_JUDGE_MODEL" \
    --matrix has_ad \
    --matrix local_flow \
    --matrix global_coherence \
    --matrix accuracy_evaluation \
    --matrix naturalness_evaluation \
    --matrix trust_evaluation \
    --matrix out_token \
    --matrix price \
    --output "scores-$method.json"
done

gembench compare scores-ad-chat.json scores-gir-r.json \
  --baseline ad-chat
gembench report scores-gir-r.json

Generation, embeddings, and qualitative judges require the corresponding model credentials and incur model cost. The decision should not be “which method has the highest total score?” It should be “for which questions is the commercial insertion not worth the loss in answer quality and cost?”

The validity boundary

GEM-Bench is evidence about offline AIR generation. It does not currently measure:

live impressions, clicks, conversions, or revenue;
query eligibility, bidding, auction incentives, or advertiser utility;
how disclosure changes user understanding;
policy compliance, brand safety, or regulatory requirements;
implicit framing and preference shaping;
long-term user trust.

The quotation marks around the paper's CA-Prod “CTR” label are intentional: the reported value is query-product relevance accuracy, not an observed click-through rate. Separately, the Click rubric reports an LLM-predicted propensity. The human study is a limited consistency check, not an online field experiment. These are not footnotes to hide; they define when the benchmark can support a decision.

Use and cite GEM-Bench when the research question is how to generate and evaluate an answer containing an explicit ad. Do not use it as evidence that an ad system will increase real clicks, preserve long-term trust, or produce an incentive-compatible market.

The paper, open-source benchmark, and KDD 2026 record are public.