Agentic Commerce TAP Benchmark
1 phaseActive
Evaluates AI shopping agents across 5 dimensions — TAP protocol compliance, commerce task completion, consumer intent adherence, security, and error recovery — using Visa's open-source merchant stack.
Overview
Agentic Commerce TAP Benchmark
Quick answer: The Agentic Commerce TAP Benchmark is a BenchGen evaluation suite for AI shopping agents operating within Visa's Trusted Agent Protocol ecosystem. It scores agents across five dimensions — protocol conformance, commerce task completion, consumer intent adherence, security and replay resistance, and error recovery — using a live TAP-enabled merchant environment. No scores are published yet; the benchmark is in active development.
At a Glance
What it tests: Whether an AI shopping agent can correctly implement the Trusted Agent Protocol, complete real purchase flows end-to-end, respect consumer intent constraints (spend limits, product exclusions), resist adversarial replay attacks, and recover gracefully from failures.
Why it matters: Visa's TAP verifies who an agent is but says nothing about how well it behaves. Every merchant accepting agent-initiated payments, every payment network processing them, and every consumer delegating purchasing to an AI agent needs a credible, independent answer to that second question. This benchmark is designed to provide it.
Known limitations: As of July 2026, no model scores have been published. The benchmark environment is based on Visa's open-source merchant stack; coverage of the full Intelligent Commerce Connect API surface (tokenization, payment instructions, fraud signals) will require additional ICC API integration and may follow in a subsequent version.
What the Agentic Commerce TAP Benchmark Measures
Agentic commerce infrastructure became commercially live in April 2026, when Visa launched Intelligent Commerce Connect and the Trusted Agent Protocol entered active pilots with AWS, Highnote, Mesh, and others. The infrastructure allows AI agents to browse product catalogs, add items to carts, and complete purchases on behalf of consumers — authenticated and cryptographically verified at every step.
The gap the benchmark addresses is the difference between identity verification and capability verification. TAP proves an agent is registered and legitimate; it does not prove that the agent correctly understood the consumer's intent, respected a stated $75 spend limit, selected the right product variant, or avoided over-purchasing. This gap — between being a trusted agent and being a well-behaved agent — is currently unmeasured by any public standard.
The benchmark constructs a controlled version of the full agentic commerce flow using Visa's own open-source sample merchant stack (React frontend, FastAPI backend, CDN proxy, Agent Registry) and adds an evaluation harness that intercepts, logs, and scores agent behavior at every step. Each evaluation run is reproducible, and every assertion is machine-verifiable — no human judgment is required to determine pass or fail.
Benchmark Dimensions
Dimension 1 — Protocol Conformance
Tests whether the agent generates correctly formed RFC 9421 HTTP Message Signatures on every request. Assertions include: correct @authority and @path binding, unique nonce per request, valid created/expires bounds, correct action tag (agent-browser-auth for browsing, agent-payer-auth for checkout), and cryptographic signature verification against the registered public key.
Failures here are deterministic: an agent either produces a valid TAP signature or it doesn't. This dimension is a prerequisite for all others — the CDN proxy blocks requests that fail.
Dimension 2 — Commerce Task Completion
Tests whether the agent successfully completes end-to-end shopping flows: product page navigation and attribute extraction, correct item selection and add-to-cart, cart review and checkout initiation, checkout form completion (contact, shipping, payment), order submission, and confirmation extraction.
Scored as task completion rate across a set of standardized purchase scenarios with varying product types, form structures, and page layouts.
Dimension 3 — Consumer Intent Adherence
The highest-stakes dimension. Tests whether the agent executes what the consumer meant, not just what it interpreted:
- Spend cap compliance — agent must not initiate checkout when cart total exceeds a stated maximum
- Attribute matching — correct product variant selected given a natural-language description
- Exclusion handling — excluded item categories or products must not appear in the final cart
- Ambiguity surfacing — ambiguous instructions must trigger clarification, not a guess
Intent adherence failures are the primary source of legitimate consumer disputes in agentic commerce. An agent that completes 95% of tasks but violates spend limits 8% of the time is commercially unsuitable regardless of its protocol scores.
Dimension 4 — Security and Replay Resistance
Tests adversarial properties claimed by the TAP specification:
- Captured signatures cannot be replayed against the same endpoint
- Signatures for domain A are rejected by domain B
- Signatures for one path cannot be used on a different path
- Expired signatures are blocked
- Requests signed with a deactivated key are denied
This dimension validates TAP's core security guarantees in practice rather than in specification.
Dimension 5 — Error Recovery
Tests graceful degradation when things go wrong: signature verification failure triggers fresh-signature retry with new nonce; out-of-stock items surface to the consumer rather than substituting silently; declined payment is communicated without unauthorized retry; checkout form validation errors are identified and corrected.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent / Commerce |
| Primary metric | Composite score across 5 dimensions (0–100) |
| Sub-metrics | Protocol conformance %, task completion %, intent adherence %, security pass rate %, error recovery % |
| Environment | Visa Trusted Agent Protocol sample merchant stack |
| Environment source | github.com/visa/trusted-agent-protocol |
| Protocols covered | Trusted Agent Protocol (TAP) — RFC 9421 HTTP Message Signatures |
| Algorithms tested | Ed25519, RSA-PSS-SHA256 |
| Saturation | Low — no published scores yet |
| Created by | BenchGen |
| Status | In development — scores coming soon |
How the Agentic Commerce TAP Benchmark Is Scored
Each evaluation run deploys the full TAP sample merchant stack in an isolated environment and runs the agent under test through a fixed scenario catalog. Scores are computed automatically from logged outcomes.
Protocol conformance is binary per request: the signature either passes CDN proxy verification or it doesn't. The dimension score is the percentage of requests that pass.
Commerce task completion is binary per scenario: the agent either reaches a confirmed order ID or it doesn't. The dimension score is the percentage of scenarios completed successfully.
Intent adherence is scored per constraint: each spend cap, attribute requirement, or exclusion in a scenario is a separate pass/fail check. The dimension score is the percentage of constraints correctly respected.
Security is binary per adversarial test case: each replay or cross-domain attempt either succeeds (fail) or is blocked (pass).
Error recovery is binary per injected failure: the agent either recovers gracefully or it doesn't.
The composite score is an equally weighted average of the five dimension scores. Because dimensions have different numbers of underlying checks, the composite score reflects breadth of correct behavior, not just raw task volume.
A score above 80 across all five dimensions is the target threshold for what BenchGen considers production-ready for merchant acceptance. Scores below 60 on intent adherence or security indicate specific classes of commercial risk regardless of overall composite performance.
State-of-the-Art Results
No scores published yet. The leaderboard will be populated when the benchmark enters open evaluation.
| Rank | Model | Protocol | Task | Intent | Security | Recovery | Composite | Date |
|---|---|---|---|---|---|---|---|---|
| — | Coming soon | — | — | — | — | — | — | — |
Interested in running your agent against this benchmark? Get in touch.
Frequently Asked Questions
How does this benchmark relate to Visa's TAP certification?
TAP certification, as defined by Visa, validates that an agent correctly implements the protocol's identity and signature mechanics — roughly equivalent to this benchmark's Dimension 1 (Protocol Conformance). The Agentic Commerce TAP Benchmark extends beyond protocol compliance to measure actual commerce behavior: whether the agent completes tasks correctly, respects consumer constraints, and handles adversarial conditions. The two evaluations are complementary rather than competing.Does my agent need to use Visa cards or a live payment processor?
No. The benchmark environment uses Visa's open-source sample merchant stack with a mock payment backend. No live card credentials, real transactions, or PCI-scoped infrastructure are required to run an evaluation. The checkout flow is fully simulated.Which agent protocols does the benchmark support?
The initial version of the benchmark is scoped to the Trusted Agent Protocol (TAP). Visa's Intelligent Commerce Connect also supports MPP, ACP, and UCP. Benchmark coverage for additional protocols is planned for future versions.What is the "intent adherence" dimension testing exactly?
Intent adherence tests whether the agent correctly executes consumer instructions that constrain the purchase — spend limits, product attribute requirements, category exclusions. These constraints are passed to the agent as part of the task description and evaluated against the final transaction state. An agent that completes checkout with the wrong item, or that exceeds the stated budget, fails the relevant intent adherence checks even if the TAP signature was perfectly formed.Is the benchmark open-source?
The underlying merchant environment (Visa's TAP sample stack) is fully open-source at github.com/visa/trusted-agent-protocol. BenchGen's evaluation harness and scenario catalog will be published when the benchmark exits active development.Benchmark designed by BenchGen based on the Visa Trusted Agent Protocol specification (github.com/visa/trusted-agent-protocol) and Intelligent Commerce Connect (corporate.visa.com). For background on the evaluation gap this benchmark addresses, see Agentic Commerce Needs a Benchmark. Last updated 2026-07-13.