Benchmark Results

Performance measurements for the a2a-protocol-sdk Rust implementation, generated by Criterion.rs statistical benchmarking.

Note: These numbers were collected on CI runners (ubuntu-latest). Absolute values will vary by hardware. Use these for relative comparisons between operations and for regression detection across releases — not as guarantees of production performance on your specific hardware.

To reproduce on your own machine: cargo bench -p a2a-benchmarks

Last updated: 2026-09-26 10:18 UTC
Rust version: rustc 1.98.1 (48a229cea 2026-09-01)
Platform: Linux-x86_64

Interactive dashboard: See the Benchmark Dashboard for charts, visual comparisons, and drill-down analysis of these results.

Transport Throughput

End-to-end HTTP round-trip latency through JSON-RPC and REST transports. All measurements use loopback (127.0.0.1) to isolate SDK overhead from network latency.

Protocol Overhead

Serialization and deserialization cost per A2A type. This is the baseline tax every message pays regardless of transport.

Includes protocol/payload_scaling benchmarks that measure pure serde cost from 64B to 1MB — the correct regression detection target for serialization changes. Also compares serde_json::to_vec vs SerBuffer (thread-local reuse) and from_slice vs from_str (borrowed deserialization) paths.

Task Lifecycle

TaskStore and EventQueue operations — the backbone of task management.

Concurrent Agents

Scaling behavior under parallel load — how latency changes as concurrency increases from 1 to 64 simultaneous operations.

Realistic Workloads

Production-like usage patterns: multi-turn conversations, mixed payloads, interceptor chains, and connection reuse vs per-request clients.

Error Paths

Cost of error handling — comparing happy path latency to error path latency. Production systems spend significant time on error paths; benchmarking only the happy path gives an incomplete picture.

Streaming & Backpressure

Stream throughput under varying event volumes and consumer speeds. Reveals buffering and flow-control overhead that synthetic single-event tests miss.

The default broadcast channel capacity is 256 events (raised from 64 in v0.5.0). Deployments with >256 events/task should use EventQueueManager::with_capacity() to set a higher value.

Data Volume Scaling

TaskStore performance at realistic data volumes (1K to 100K tasks). Shows how store operations scale as data accumulates over time.

Memory Overhead

Heap allocation counts and bytes per operation, measured via a counting allocator (#[global_allocator]). Values represent allocation counts or bytes — not time — encoded as nanoseconds for Criterion tracking.

Cross-Language Comparison

Rust-side criterion baselines for the workloads in benches/cross_language/. Every row below is this SDK only. The measured cross-SDK comparison — this SDK's server against the official Python SDK's server — is on Cross-Language Benchmark.

Enterprise Scenarios

Production-scale workloads modeling real deployments: multi-tenant isolation, push notification management, eviction under memory pressure, rate limiting, CORS handling, read/write mix ratios, and large conversation histories.

Production Scenarios

Full end-to-end workflows exercising the complete SDK pipeline in scenarios that real-world deployments encounter at scale: task reconnection, cold start latency, concurrent race conditions, multi-context orchestration, push config lifecycle, parallel agent bursts, and dispatch routing overhead isolation.

Advanced Scenarios

SDK capabilities exercising previously-unbenchmarked paths: tenant resolver overhead, agent card hot-reload and discovery, subscribe fan-out for reconnection bursts, streaming artifact accumulation cost, pagination full walk, and extended agent card round-trip.

Agent-Level Latency Under Fault

Not run by the benchmarks workflow; no results are published here. benchmarks.yml does not run this bench, so the two groups below have no tables. To measure it locally: cargo bench -p a2a-benchmarks --bench coordinator_chain_under_fault.

End-to-end latency through a 5-hop in-process coordinator chain as the links between hops are made progressively less reliable. Unlike every other benchmark on this page, this one does not measure SDK-layer overhead — it measures the characteristic an agent-harness reviewer actually wants: "what is the end-to-end latency of an agent chain when the network between agents is unreliable, and how well do per-hop retries absorb it?"

The topology is:

test client ─[link 0]─▶ coord 1 ─[link 1]─▶ coord 2 ─[link 2]─▶ coord 3 ─[link 3]─▶ coord 4 ─[link 4]─▶ leaf

Every coordinator forwards the message to the next hop via a pre-built A2aClient wrapped in a FaultInjectingTransport. Each link applies its own independent fault profile, so per-hop faults compound end-to-end the way they would in a real deployment. Coordinators 1–4 retry their downstream call up to 3 times on retryable errors; the bench harness additionally retries the top-level send_message up to 8 times so the published error rates have effectively-zero unrecoverable-failure probability.

Honest caveats — read these before interpreting the numbers:

• In-process, not network faults. The injected "error" is a synthetic ClientError::Timeout returned before the wrapped transport is called. This exercises the SDK's retry path faithfully, but does not exercise TCP congestion control, DNS resolution, or transport-level head-of-line blocking. Treat the numbers as "latency under SDK-level retransmission pressure," not "latency under real network loss."
• One topology. Sequential delegation is the simplest multi-agent shape. Critic loops, parallel fan-out with deadline propagation, and plan-and-execute with replanning would be more rubric-relevant — this benchmark does not claim to cover those.
• One benchmark does not retroactively make the other suites agent-level. It is deliberately additive: the first concrete data point in the "agent-level latency under fault" shape that the rest of the suite was missing entirely.

Group 1: per-hop latency injection (zero errors)

Varies per-link latency from 0 µs to 20 000 µs with zero synthetic errors, isolating the chain's latency-compounding factor from retry jitter. Five hops × per-hop latency gives the lower bound, plus the JSON-RPC loopback baseline (~2 ms for a five-hop chain with zero added latency).

Group 2: per-hop error injection (3 retries per hop + 8 outer retries)

Varies per-link synthetic-fault rate from 0% to 5% with zero added latency. Each coordinator retries its downstream call up to 3 times on retryable errors; the bench harness retries the top-level call up to 8 times. Records successful-path latency including retry cost, which is what "steady-state end-to-end latency under fault" means in practice.

Known Measurement Limitations

These notes help interpret benchmark results accurately and avoid misdiagnosing CI variance as real performance changes.

Streaming cross-thread scheduling

On N-core systems, `tokio::spawn` places the SSE builder task on a different worker thread with (N-1)/N probability, causing ~500µs cache-miss + work-stealing penalty. This was root-caused as the source of the ~24% bimodal distribution in all streaming benchmarks.

Mitigations (v1.0.0): The SSE builder uses `sleep` + reset (not `interval`) to eliminate timer wheel entries during active streaming. Transport streaming benchmarks use `worker_threads(1)` runtime to eliminate cross-thread variance entirely (24 high severe → 4 high mild outliers, 3× tighter confidence intervals).

Data volume get() at 100K tasks

data_volume_get/lookup reports 445 ns at 1K tasks, 447 ns at 10K and 214 ns at 100K. A 100K figure below the 1K/10K ones has previously been traced to a CPU cache warming artifact from the large populate_store() setup filling L1/L2 caches; a 4MB cache-busting step was added in v0.5.0 to flush caches between populate and measure. Where the 100K figure is still the lowest, that step has not removed the effect, and the 1K/10K numbers are the representative O(1) lookup baseline.

Stream volume per-event cost

The default broadcast channel capacity is 256 events (raised from 64 in v0.5.0). backpressure_stream_volume straddles it: 631.8 µs for 52 events, 1.80 ms for 252 and 3.22 ms for 502, so the per-event cost below and above capacity can be read directly from the table.

Production deployments expecting >256 events/task should increase EventQueueManager::with_capacity() to match their peak volume.

Transport payload insensitivity

Transport benchmarks (64B → 16KB) show a 31.5% latency increase for a 256× payload increase, because the 216.7 µs HTTP round-trip dominates. Serde regressions cannot be detected via transport benchmarks. Use the protocol/payload_scaling isolation benchmarks (64B → 1MB, pure serde) for serialization regression detection.

Connection reuse impact

Connection reuse saves 138.8 µs (39.1%) on loopback — 354.9 µs per request when the client is rebuilt each time, versus 216.1 µs when it is shared. On real networks with TLS the saving is larger still (TLS handshake dominates). Best practice: create one A2aClient at startup and share via Arc across request handlers.

Two consequences worth spelling out, because both have bitten this repo:

• A benchmark that builds a client inside its measured region is measuring client construction, not the thing it names. production_agent_burst does exactly this, and its per-agent cost tracks 354.9 µs — the rebuild-every-time number — rather than the shared-client one.
• Quoting this saving as a small percentage understates it by roughly 4×. It is a large fraction of a loopback request, not a rounding error.

What sharing a client is actually worth under concurrency

The bullet above says production_agent_burst tracks the rebuild-every-time number. That is true, and it invites a wrong inference: that sharing a client would move it to the reused figure. It does not, and production/agent_burst_client_sharing was added to measure the difference rather than reason about it. Both arms run the same server, the same three operations per agent, and the same burst sizes; only the client's provenance changes.

Sharing wins at every burst size, and the medians' 95% confidence intervals are disjoint in all three, so the direction is not noise. But the size of the win is about half what the single-request comparison above predicts: 40.0 µs (12.0%) per agent against the 138.8 µs (39.1%) that reused_client versus new_client_per_request would lead you to expect.

The reason is that the two arms differ in two coupled ways, not one. A shared client skips per-agent construction and per-agent connection setup, but it also puts every concurrent agent on one connection pool, and that contention gives part of the saving back. The sequential benchmark has no contention to pay, which is why its number is the optimistic bound rather than the forecast.

The practical reading: share the client — it is free to do and wins at every size measured — but size capacity from the burst figures, not from the single-request saving.

Deserialization allocation overhead

Deserialization allocates several times more than serialization (Task: 1045 vs 343 allocs). This is inherent to serde_json's parsing model: every field creates an intermediate String/Vec allocation during parsing. The serde_helpers::deser_from_str() helper enables serde_json's borrowed-data path for ~15-25% fewer allocations. The serde_helpers::SerBuffer provides thread-local buffer reuse for serialization, eliminating the 2.3× small-payload overhead.

History depth allocation scaling

History depth scales roughly linearly: memory_history_scaling records 1351 deserialization allocs at 1 turn and 23277 at 50 turns (526 and 12313 for serialization), so a 50-turn task costs 23277 deser allocs per store.get(). The serde_helpers module provides optimized paths; for maximum throughput on deep histories, consider storing pre-serialized bytes alongside parsed structs to avoid re-parsing on every read.

Artifact accumulation clone cost

The background event processor clones the full Task struct on each SSE event. Clone cost grows with artifact count: task_clone_at_depth is 129 ns with no artifacts and 67.9 µs with 500. For tasks with 500+ accumulated artifacts, consider batching event processing or using the planned copy-on-write artifact storage (tracked as a future optimization).

Slow consumer timer calibration

The backpressure/timer_calibration benchmarks measure actual tokio::time::sleep() durations on the CI runner. On shared runners, 1ms sleep ≈ 2.10 ms actual, 5ms sleep ≈ 6.12 ms actual. Slow consumer results should be interpreted against these calibrated durations, not the nominal sleep values.

Data volume save() wide confidence intervals

The data_volume/save/after_prefill/10000 benchmark reports wide confidence intervals ([1.4µs, 3.5µs], spanning a 2.5× range) and an 18% high severe outlier rate. This is caused by BTreeSet rebalancing spikes when the sorted index crosses internal node-split thresholds during insert. The median (~1.6µs) is representative; the wide CI reflects genuine variance from the B-tree data structure, not measurement noise. This is an acceptable tradeoff: the BTreeSet enables O(page_size) pagination queries vs O(n) full scans.

Dispatch routing: direct handler vs HTTP round-trip

production/dispatch_routing/direct_handler_invoke calls the request handler directly and reports 76.5 µs; full_http_roundtrip sends a message through a JSON-RPC server over a warm keep-alive connection and reports 214.4 µs. The difference is the dispatch and transport overhead the direct call bypasses.

Subscribe fan-out scaling

advanced/subscribe_fanout reports 658.6 µs with 1 subscriber, 838.5 µs with 5 and 1.16 ms with 10.

Agent burst scaling

production/agent_burst times the whole burst: 3.77 ms for 10 agents, 18.01 ms for 50 and 36.77 ms for 100 — per agent, 377.3 µs, 360.1 µs and 367.7 µs respectively.

Cold start vs steady state

production/cold_start/first_request (387.6 µs) creates a fresh server per iteration (sample_size=20), measuring server handler initialization + first TCP connect. steady_state (223.4 µs) reuses an existing keep-alive connection, measuring the full HTTP round-trip with connection overhead already amortized. The two benchmarks measure different things — they are complementary, not comparable.

Tenant resolver overhead

Per request, the tenant resolvers cost 131 ns (header), 89 ns (header miss), 172 ns (path), 127 ns (bearer) and 150 ns (bearer with mapper), against 215.6 µs for a full JSON-RPC round trip (transport_jsonrpc_send/single_message).

Pagination context index

At 1000 tasks, the advanced/pagination_walk filtered walk takes 325.8 µs against 637.2 µs unfiltered. The BTreeSet context index reduces the scan work by only iterating tasks matching the context_id filter.

Methodology

All benchmarks use Criterion.rs, which provides:

• Statistical significance testing — detects real regressions vs noise
• Warm-up iterations — avoids cold-start measurement artifacts
• Median ± MAD — robust central tendency resistant to outliers
• Configurable sample sizes — more iterations for noisy benchmarks

Measurement rigor

All benchmarks follow these practices for reproducibility:

• Deterministic inputs: Fixed task IDs and payloads inside iter() — no incrementing counters that change HashMap distribution across iterations
• Setup outside measurement: Store creation, server startup, and resource allocation happen before iter(), not inside it
• debug_assert! for invariants: Correctness checks inside measurement loops use debug_assert! to avoid string-formatting cost in release builds
• black_box() on inputs and outputs: Prevents the compiler from eliminating measured work through dead-code optimization
• Tolerance-based allocation assertions: Memory benchmarks use a 5% tolerance instead of exact counts to avoid spurious CI failures from serde_json/stdlib version changes
• Side-effect interceptors: The interceptor chain benchmark uses CountingInterceptor (AtomicU64) to verify interceptors are actually invoked during measurement — not just optimized away

What we benchmark

The SDK's value proposition is the A2A protocol layer and runtime efficiency, not agent logic. The bulk of the suite therefore benchmarks what the SDK owns: transport overhead, serialization cost, store operations, concurrency scaling, streaming backpressure, error handling, and memory allocation behavior.

One benchmark — coordinator_chain_under_fault — is deliberately a different shape: it measures end-to-end agent-chain latency under fault injection, not SDK-layer overhead. It is documented in its own section above with the caveats for how to interpret it (in-process only, sequential delegation only, one topology). It is not intended to substitute for a real agent-capability benchmark suite — it closes the most obvious gap in the existing suite while staying honest about what it is.

What we do NOT benchmark

• Agent intelligence — LLM quality is an eval problem, not a perf benchmark
• Real network faults — the fault-injection bench simulates synthetic ClientError::Timeout responses in-process, not real packet loss or TCP congestion control
• Network latency — all benchmarks use loopback (127.0.0.1)
• TLS handshake — benchmarks use plaintext HTTP
• Task completion quality — needs human-preference evaluation
• Multi-agent topologies beyond sequential delegation — critic loops, parallel fan-out with deadline propagation, and plan-and-execute with replanning are out of scope for this crate

Reproducing locally

# Run all benchmarks
cargo bench -p a2a-benchmarks

# Run a specific module
cargo bench -p a2a-benchmarks --bench transport_throughput

# Save baseline, make changes, then compare
./benches/scripts/run_benchmarks.sh --save
# ... make changes ...
./benches/scripts/run_benchmarks.sh --compare

Full HTML reports (with violin plots and comparison overlays) are generated in target/criterion/.