Methodology · Updated September 2026
Kettio ranks comparable ad creative for a specific buyer and conversion goal before media spend. Each ad gets a score, a written reason and a confidence signal. The ranking is directional: it tells you which ads deserve a live test first, and live media stays the final proof.
Definition
Synthetic audience testing shows ads to AI respondents modeled on a defined buyer instead of to a paid audience, then reads how they react. It answers the question a creative review meeting answers today, which drafts are worth spending on, without the room’s personal taste.
At Kettio it has a narrow job. It ranks a comparable batch of ads for one buyer and one goal. It does not simulate a market, predict revenue, or replace a live test. Its value is cutting ten drafts to the two or three that should get real budget.
The method
You describe the buyer the way you would brief a strategist: age range, what they are shopping for, how skeptical they are of ads, how urgent the need is. You also pick the response the ad has to earn. A scroll-stopping ad and a purchase-driving ad are scored against different questions.
A language model answers as that buyer, one ad at a time, in plain words. Ads are never shown side by side at this stage, so a strong ad cannot make a weak one look worse by contrast. Each ad gets several independent reads, not one.
This is the step called Synthetic Survey Rendering (SSR). Instead of asking the model for a number, which language models give inconsistently, we compare its written reaction to fixed reference answers on a five-point scale, from “no interest” to “need this now” for purchase intent. The closeness to each reference answer becomes a probability spread over the five points, and the spread gives the score.
When two ads land within a small margin of each other, a smaller panel compares just those two directly and votes. The votes decide the order, and the confidence signal tells you the call was close.
Each ad comes back with its score, the reaction behind it, and what the buyer noticed first. The rationale is the part you brief the next round from.
Validation
One benchmark checks the panel against what people say about ads. The other checks it against which ads people actually clicked. Neither is a purchase or CPA result, and they are never combined into one accuracy number.
ρ = 0.78
Spearman rank correlation between the synthetic panel’s scores and a University of Washington survey in which real people rated the same ads. 160 paired ads. A correlation of 1.0 would mean identical order.
Boundary: Measures what people say about ads in a survey. It is not a click, conversion or revenue result.
70.3%
On the public CreativeRanking dataset, for pairs of ads promoting the same product, how often the head-to-head panel picked the ad that actually earned the higher click-through rate. A coin flip scores 50%.
Boundary: Within-product pairs only, on click-through rate. It is not a purchase or CPA result, and it does not transfer to ads for unrelated products.
Two ways to rank
The synthetic buyer panel works with no history. Account calibration fits the model to an account’s own delivered ads, then grades itself only against labels that are reliable at the exposure those ads actually got.
Every plan. No history needed.
AI respondents modeled on the buyer and goal you choose react to each ad separately, so one ad can’t color the read on another. Each reaction is scored against fixed reference answers, and the scores become a ranking. When two ads land close together, a smaller panel compares them head to head.
ρ = 0.78 (0.775 unrounded) against the University of Washington ad-perception survey (n = 160 paired ads). That measures agreement with what people say about ads, not what they buy.
Agency pilots with delivery history.
A model fitted to the account’s own delivered ads: every image, its in-image text and copy, its impressions and its clicks. It learns what earns attention from this account’s buyers, then ranks new creative before it spends.
Numbers below. Accuracy is always reported next to the ceiling the account’s own data allows.
Account calibration
We pull every delivered ad with its image, in-image text, copy, impressions and clicks. The model learns which visual and copy traits earned attention for this account’s buyers, using only ads launched before the ones it is asked to rank.
An account with a few dozen ads can’t teach a model much on its own. The image model is first tuned on click data from larger accounts in the same category, then fitted to your account. Only the tuned model carries over; no account sees another’s creative or results.
These are two different questions: which concept (ad set) deserves budget this month, and which variant inside a concept wins. Copy and in-image text mostly decide the first; image details mostly decide the second. Every ranking says which question it answers.
The score orders creative by expected attention and on-site intent. It tells you what to put into live spend first, not what an ad will earn.
Labels and exposure
A model is only as good as the outcome it learns from. Most of our research went into finding which outcomes are reliable at the delivery real ads get.
At normal per-ad exposure, purchase counts are too small to separate ads: split one ad’s run into alternating days and its purchase ranking often disagrees with itself. Clicks hold up, so the model trains on clicks, and we check conversion against product-page views, the most reliable on-site signal we have found.
A click ranking is about 80% trustworthy after roughly 1,000 to 10,000 impressions per ad, depending on the account’s click rate. A purchase ranking needs 10,000 to 100,000 or more. Below that, an ad’s result is mostly noise.
Meta often gives one ad most of the spend on day one. We tested whether that pick converts better per impression; it doesn’t. A model that learns Meta’s delivery can look accurate while telling you nothing, so we grade per-impression rates and pull barely-delivered ads toward their batch average instead of calling them losers.
Before grading a model, we measure how often the label agrees with itself by splitting each ad’s run into alternating days. No predictor can beat that. We report accuracy next to its ceiling so you can see how much of the knowable signal the model captures.
Calibration results
Each number answers one question on one account, measured walk-forward: the model was trained only on ads launched before the month it ranked.
71.3%
How often the model ordered two concepts (ad sets) launched in the same month the same way their click rates did, on an account with about 1,000 delivered static ads. The click label’s own ceiling at this level is 81.1%, so this is 88% of what the data can show.
Boundary: Walk-forward on past launches across three launch months, ±3 points. Not yet graded on fresh launches.
71.4%
The same question on an account with 28 to 158 training ads per step, using the image model tuned on a larger account; it never saw this account’s labels during tuning. 71.4–72.0% on clicks and 72.7–73.2% on add-to-carts, checkouts and purchases across two training runs, against ceilings of 79.2% and 78.2%.
Boundary: Walk-forward on 31 past ad sets, ±4 points. Not yet graded on fresh launches.
Where it fits
Kettio is a pre-testing tool. It decides what goes into the paid test; the paid test decides what scales.
Limits
FAQ
An ad creative score estimates how strongly a specific buyer will respond to an ad before it runs. Kettio’s score is on a five-point scale for a chosen goal, such as purchase intent or scroll-stopping, and is meant for ordering comparable ads, not for comparing ads across unrelated products.
Synthetic audience testing means showing ads to AI respondents modeled on a defined buyer instead of to a live audience, then reading their reactions. At Kettio it produces a ranking of a comparable creative batch for one buyer and one conversion goal. It is a pre-spend filter; live media is still the final behavioral proof.
Partly. AI can order comparable ads by how a defined buyer is likely to respond, and that order agrees with real outcomes more often than chance: Kettio’s head-to-head panel matched real click-through winners in 70.3% of within-product pairs on the CreativeRanking dataset. AI cannot reliably forecast CPA, ROAS or purchase volume for a single ad.
Two separate benchmarks. The synthetic panel’s scores correlate with a University of Washington ad-perception survey at Spearman ρ = 0.78 (n = 160 paired ads). The head-to-head panel agrees with real click-through labels 70.3% of the time on within-product pairs in the CreativeRanking dataset. They measure different outcomes and should not be combined into one accuracy number.
Pre-testing evaluates ads before they run, to decide which drafts deserve budget. Post-testing measures ads after they run, using real clicks, purchases and cost. Pre-testing is faster and cheaper but directional; post-testing is the behavioral proof. Kettio is a pre-testing tool meant to shrink the number of ads that go into a paid test.
SSR is how Kettio turns a written reaction into a score. Rather than asking a language model to pick a number, it compares the model’s reaction to fixed reference answers on a five-point scale and converts that closeness into a probability spread over the five points. Numbers requested directly from language models are inconsistent; comparing written reactions is more stable.
No. The synthetic buyer panel works from the ads and the buyer description alone. Account calibration, offered to agencies with delivery history, uses the account’s past ads, impressions and clicks to fit the model to that account.