Visa's Agentic Commerce Benchmark, published on November 5, 2026, put 22 models from 12 labs through 15 shopping tasks and recorded a 42.8% failure rate; in roughly 54% of those failures the agent completed a purchase anyway. As of November 15, 2026, that benchmark remains the closest thing to a public scorecard for agentic commerce, and it landed in the same fortnight as the ChatGPT Wallet launch on November 13, which lets subscribers' agents pay with a tokenised card after a biometric check.
Two failure modes, one page problem
The first failure mode is a missing question. An agent that never asks about room size or budget will pick something plausible and wrong, and the consumer demand signal it was supposed to serve gets answered with the wrong item. The second is a missing attribute: the agent asks, the shopper answers, and the product page still does not state the number needed to decide. The benchmark's own split points at the first mode, since models that asked more questions scored far better. But the second mode is the one a merchant can fix this quarter.
That distinction matters because the two modes have different owners. Question behaviour belongs to the model provider and improves with each release. Attribute completeness belongs to the merchant and does not improve unless someone edits the page. Waiting for better models to solve a data problem is a expensive way to lose a year.
Why product data decides the outcome
Specs as answers
An agent can only weigh what the page states. If a listing says a dishwasher is quiet without giving a decibel figure, the agent cannot compare it against a competitor that publishes 44 dB. The practical result is that well-documented products win comparisons they would also win on merit, and poorly documented ones lose comparisons they might have won. Spec completeness is now a distribution variable, not a copywriting preference, and category growth in agent-mediated channels will follow whichever catalogues are easiest to read.
Variant collapse
Most catalogues are messier than they look. A single parent listing can hold dozens of variants with different capacities, colours and bundle contents, and the attributes are often buried in images or in a bullet list that mixes marketing claims with specifications. When an agent has to choose among variants and the differences are not machine-readable, it either picks the default or picks wrong. Both outcomes produce the same review: one star, wrong item.
The merchant split
Merchants are dividing into two camps, and the split is becoming a strategic choice rather than a technical one. Open ecosystems let outside agents read listings and complete purchases; closed ecosystems block them and keep the transaction inside their own assistant. Walmart and Shopify sit on the open side, with spending caps emerging as a compromise. Amazon, which blocked Meta's shopping agent from its store, sits on the closed side.
| Approach | Who is doing it | What it assumes about reliability |
|---|---|---|
| Open catalogue, capped agent spending | Walmart, Shopify merchants | Agents will improve; caps limit the damage meanwhile |
| Closed ecosystem, own assistant | Amazon and its shopping assistant | Control the agent and you control the error rate |
| Payment rails with approval step | Card networks, wallet providers | Human confirmation absorbs residual model error |
For a brand, the split has a practical consequence: the same product page is now read by several different agents with different tolerances. A page that satisfies one assistant's extraction rules may still fail another's. The work that travels across all of them is the boring part, consistent units, explicit compatibility, and a variant structure that does not require guessing.
What to fix first
Start with the attributes that decide a purchase and are currently written as adjectives. Capacity, dimensions, compatibility, what is in the box, and which variant differs from which. Then check consistency across the brand site, the marketplace listing and any retailer feed, because an agent that finds two different numbers will pick one and describe your product with it.
- Numbers over adjectivesReplace "large capacity" with the litre figure and the loaded dimensions.
- One number, everywhereBrand site, marketplace and retailer feeds should quote the same spec, not an older revision.
- Variants that explain themselvesEach variant needs its differentiator in text, not only in the image.
None of this guarantees a citation or a correct purchase. The benchmark shows that even strong models fail often enough that confirmation steps remain necessary. What it does change is the baseline: an agent working from complete, consistent attributes has fewer ways to be wrong, and a merchant with a clean catalogue spends less time explaining a one-star review that was never about the product.
FAQ
What does a 42.8% failure rate actually measure?
It measures multi-step shopping tasks, not page views. Each run required an agent to work out what the shopper needed, then either buy the right item or decline. A failure can be the wrong variant, a missed requirement, or a purchase that should not have happened.
Why do weaker agents still complete purchases?
Because most checkout flows were built for humans who can see what they are buying. An agent that misreads a spec still finds a buy button, and the error surfaces after payment as a return or a chargeback.
Which part of a product page matters most?
The attributes a shopper would ask about in a store: size, capacity, compatibility, what is in the box. Those are also the attributes an agent cannot infer. Marketing copy is useless to a model choosing between two variants.
Should brands optimise for agents or for shoppers?
The overlap is large. Clear specs, consistent variants and explicit compatibility help a human compare and help a model extract. The work that only helps an agent is small compared with the work that helps both.
