57.2% of shopping tasks passed; 42.8% did not, and more than half of the failures still ended in a payment. Visa's Agentic Commerce Benchmark, published on November 5 and read here as of November 17, 2026, ran 22 models from 12 labs across 15 shopping tasks with three attempts each. For instant retail operators the interesting part is not the pass rate but where the runs broke: stock visibility, delivery windows and store choice, the three fields a quick commerce order actually depends on.
Where the shopping agents actually broke
The benchmark scored a run as passed only when the agent finished the whole job: it found the item, picked a variant, set a delivery or pickup option and paid without a person taking over. Two thirds of the models never produced three clean runs in a row. The best model finished 82.2% of attempts and the weakest 28.9%, a spread wide enough that “our agent can shop” tells a category team almost nothing on its own.
Visa also cut the results by behaviour and by price. Models that asked more clarifying questions before acting passed 70.2% of runs, while the models that asked least passed 44.4%, which points at missing product attributes rather than missing reasoning. The five most expensive models passed 67.1% and the five cheapest 50.2%, so capability sets the ceiling and budget alone does not buy it.
Three fields decide whether an agent can check out
Stock has to be visible before the agent asks
An agent that cannot see per-store availability does not fail politely; it substitutes. In the runs summarised by the benchmark, agents that met an unavailable variant moved to a sibling SKU or another store instead of stopping, which is how a session ends in a purchase the operator never planned. Availability by store, not assortment width, is the field that keeps an order inside the trade area the network was built for.
Delivery windows and store choice break after stock
The second cluster of failures came at slot selection and store choice, where a page offers a radius the agent cannot interpret or a pickup window that expires mid-session. That is a data-contract problem rather than a marketing one: a timestamp in ISO format and a distance in kilometres are readable, “today before six” and “near you” are not. Operators who publish structured slots and explicit cut-off times keep more agent sessions inside their own on-demand delivery network, and they convert consumer demand for speed into orders they can route.
Both clusters appear in the same category on the same day, which is why a single fix rarely moves the number. A store that publishes stock but not slots fails one step later, and the failure is recorded against the merchant rather than against the data pipeline that produced it.
What the pass rates change in a category plan
| Benchmark metric | Result | What it means for an instant retail operation |
|---|---|---|
| Runs that completed a full task | 57.2% | Plan for retry; one clean run is not a repeatable flow |
| Runs that failed | 42.8% | Failures cluster at variant, slot and store selection |
| Failed runs that still ended in a purchase | 54% | The shopper takes over at payment; log the handoff |
| Models passing all three attempts | 34.2% | Consistency is the scarce trait, not peak ability |
| Best and weakest model | 82.2% / 28.9% | Model choice shifts the outcome more than prompt tuning |
Read as an industry trend signal, the spread matters more than the average. A category with tight variants, clear pack sizes and one naming convention will pass far more often than a category where the same product ships in four formats, and the benchmark's 15 tasks are broad enough to expose that gap. Agent traffic will therefore concentrate on the categories whose product data is already clean, at least until operators repair the rest.
There is a trust ceiling as well. Visa's Trust Index, cited alongside the benchmark, finds that 72% of consumers have used an AI assistant to shop while only 23% trust an AI to pay on their behalf. That gap explains the 54% number: a run that failed technically can still end in a completed purchase, because the shopper takes over at the payment step and finishes the order in the same session, often on a different site.
A weekly check a retail operations team can run
The check is small enough to repeat every week and specific enough to produce a number. Pick ten SKUs from the top of the category, run the same task through two agents, and record where each run stopped. Two consecutive weeks at the same failure point is a data defect rather than a model defect, and it belongs to the team that owns the product feed instead of the team that owns the campaign.
Keep the log in one place and score it the same way each week: the share of runs that reached checkout, the share that reached payment, and the share that needed a human. Those three numbers move before revenue does, and they are the only ones a category manager can act on without waiting for the next benchmark refresh.
FAQ
Do these failure rates mean agents are not ready to shop?
They mean single-run reliability is low, not that agents cannot buy. With a 57.2% pass rate and only 34.2% of models passing all three attempts, an operator should design for retry and handoff; the 54% of failed runs that still converted show where that recovery already happens.
Which data field removes the most failures?
Store-level availability with a timestamp. Agents that met an unavailable variant substituted rather than stopped, so a live availability field and an explicit cut-off time protect both the order and the trade area.
How should an instant retail team measure agent traffic?
Count sessions that reach checkout, sessions that reach payment and sessions that need a human, by category. Compare those three counts with your own app, because what looks like a model defect is often a stock visibility gap on the product page.
Sources
- T-3: review of Visa's AI shopping agent benchmark — Breakdown of the 22 models, pass rates and failure categories
- SuggestAPI: merchant guide to the Visa benchmark — Merchant-side reading of pass rates and task design
- DevPro Journal: AI shopping agents fail often — Coverage of the 54% recovery rate and platform implications











