Adversarial Testing

A public, wire-facing SDK is judged by how it behaves on the inputs nobody intended to send. Unit tests prove a function rejects bad input in isolation; fuzzing proves the parsers survive generated bytes. Neither proves that a running server — one actually talking to a model — stays up and answers safely while someone tries to break it. So this project tests that directly, and writes down what it found.

The rule throughout is the one in every file header of this repository: never guess — test and verify. Nothing on this page is asserted that was not observed on the wire against a live server.

The harness

scripts/adversarial/probe.py is a black-box, over-the-wire adversarial probe. It drives a running A2A JSON-RPC server with malformed, hostile, and edge-case requests and, after every request, re-probes GET /.well-known/agent-card.json to confirm the server is still alive. A production server must answer each case with a structured JSON-RPC error (or a clean HTTP 4xx) and keep serving — never crash, hang the whole process, reset the connection, return a 5xx, leak an internal path or panic message, or silently accept what it was built to reject.

It is the third leg of the SDK's negative-input testing:

The probe needs no third-party packages and exits 0 when every case was handled safely, 1 on any danger — so it doubles as a CI gate. Its README shows the negative test that keeps it honest: against a server deliberately started with the SSRF guard off, it must exit 1 and flag every webhook case. A gate that only ever passes proves nothing.

The campaign

The probe was run against the genai-agent example in server mode, backed by the same real model used for end-to-end dogfooding — Qwen/Qwen3.5-0.8B served by llama.cpp on 127.0.0.1:11434. Six categories, 69 cases:

Result: 69 cases, zero crashes, zero process-wide hangs, zero connection resets, zero 5xx, zero path or panic leaks. The server logged not one error across the whole run. Every malformed input returned a structured JSON-RPC error; every hostile-but-valid input (an unknown field, a message carrying a right-to-left override and a NUL) was processed correctly and answered by the model. Two cases look like failures and are not — the harness is written to recognise them:

• An 8 MB body is rejected on the Content-Length fast path with a structured -32700 before the payload is buffered; the server never reads 8 MB it has already decided to refuse.
• A short body under a large Content-Length leaves the server correctly reading, bounded by its body_read_timeout; the connection waits, but the server keeps answering everyone else.

What breaking it actually found

One category did not come back clean on the first honest run, and the point of writing this down is that it did not.

The finding: an SSRF filter that missed integer-encoded IPs

The bundled HttpPushSender rejects webhook URLs that target private/loopback/link-local addresses — at registration and again at delivery. The registration-time check rejected 169.254.169.254 (the cloud metadata endpoint) and even its exotic IPv6 spellings (::ffff:169.254.169.254, the NAT64 64:ff9b::a9fe:a9fe). But it accepted the same address written as a single integer:

http://2852039166/        == 0xA9FEA9FE == 169.254.169.254
http://0xA9FEA9FE/        (hex)
http://0251.0376.0251.0376/ (octal, dotted)

Rust's Ipv4Addr::from_str only accepts dotted-quad, so these fell through the literal-IP check and were stored as webhooks. The C resolver (inet_aton, and therefore the getaddrinfo that tokio::net::lookup_host calls) does accept them — verified on this machine, all three resolve to 169.254.169.254.

Honest scope

This was a defense-in-depth gap, not a live SSRF. Delivery re-resolves the host and re-checks the resolved IP, so inet_aton-form metadata URLs resolved to link-local and were blocked before any request left the process. What was actually wrong: the registration-time filter — the documented first boundary — was inconsistent with its own threat model, storing a config it exists to reject, in exactly the attack class it already defended against for IPv6.

The fix, and the proof

validate_webhook_url now normalises the inet_aton numeric forms (single 32-bit integer; hex and octal parts; the packed 1-, 2-, and 3-part short forms) and applies the same private-range test, so the registration check now matches the address delivery would actually dial. Public numeric hosts still pass (8.8.8.8 as 134744072 is accepted), so the change blocks private targets without banning numeric hosts. Six unit tests pin the normaliser and the rejections; the probe pins it end to end:

before fix:  F category — 19/20 hostile webhook URLs rejected
             (http://2852039166/latest ACCEPTED)
after fix:   F category — 20/20 rejected, over the wire, guard on

The second finding: a template shipping the guard off

The same run surfaced a footgun in the example rather than the library. The genai-agent server mode — the one its own comment calls the "real deployment shape" — hard-coded HttpPushSender::new().allow_private_urls(), silently disabling the SSRF guard for anyone copying it as a deployment template. Server mode is now secure by default; a localhost webhook for local testing is an explicit opt-in (A2A_ALLOW_PRIVATE_WEBHOOKS=1), and the active posture is printed at startup.

Beyond JSON-RPC: gRPC and WebSocket

The same server was then made to expose all three request bindings at once (A2A_GRPC_ADDR / A2A_WS_ADDR alongside A2A_BIND_ADDR), and two companion probes attacked the other two. Both share the JSON-RPC process, so each case checks process liveness and that its own binding still accepts a fresh connection.

• probe_ws.py — WebSocket (spec §12), 32 cases. Handshake attacks (missing/bogus A2A-Version, missing key, wrong ws-version), RFC 6455 framing attacks (unmasked client frame, reserved/undefined opcodes, RSV bits, binary frame, invalid UTF-8, fragmentation, lying length, ping/pong, an 8 MB frame), and the JSON-RPC payload attacks as text frames. Zero crashes. Every protocol violation drew the correct RFC 6455 CLOSE, a ping drew a pong, the handshake was rejected (HTTP 400) without the version header, and the oversized frame was bounded.

• probe_grpc.py — gRPC (spec §10), 22 cases. Hostile field values over well-formed framing, unknown method paths, missing/bogus a2a-version metadata, malformed message bodies (truncated / garbage / wrong-wire-type protobuf sent through a passthrough serializer), an 8 MB message against the 4 MiB cap, and raw transport garbage. Zero crashes. Malformed protobuf drew a clean decode error, the oversized message OUT_OF_RANGE, the SSRF webhook INVALID_ARGUMENT (the guard holds on gRPC too), and non-HTTP/2 bytes were closed without incident.

The finding: version negotiation disagreed across bindings

The gRPC run surfaced one real gap. A request with no a2a-version was processed on gRPC, while the identical request on JSON-RPC and WebSocket was rejected — the spec (§3.6.2, §737) says an absent value is protocol 0.3, which a 1.x server must refuse with VersionNotSupported. The gRPC helper did its version check only when a value was present, and its docstring even claimed it mirrored the other bindings when it did not. It now delegates to the same shared validator the HTTP and WebSocket bindings use, so all four agree; a gRPC request with no version returns UNIMPLEMENTED / VERSION_NOT_SUPPORTED, regression-tested at the helper and through the real method path. This is the kind of cross-binding inconsistency only a probe that hits every binding with the same attack will find.

Authentication

Auth is not a transport filter here — each interceptor runs inside the transport-agnostic handler, so one interceptor guards every method on every binding, and the plain agent card stays public by design. To attack it, genai-agent server mode was configured with JWT-HS256 auth (A2A_JWT_HS256_SECRET with an issuer and audience), and probe_auth.py forged HS256 tokens in the standard library to hit it with the full matrix — 27 cases, zero bypasses:

• Signature / algorithm — alg:none, alg:NONE, an RS256 header against a server with no JWKS, HS512 (outside the allowlist), a wrong secret, a tampered payload, and a corrupted signature are all rejected. The allowlist is {HS256, RS256, ES256}; none and everything else fall through to a hard reject, and HS256 verifies only against the configured secret, so the RS256→HS256 confusion attack has nothing to grip.
• Claims — expired, not-yet-valid (nbf), missing exp, wrong issuer, and wrong audience are all rejected; an aud array that contains the accepted value is accepted, per spec.
• No oracle — missing, malformed, and wrong-credential all return one generic authentication required, so the error cannot be used to distinguish "no token" from "wrong token".
• Every binding — the same rejection holds over gRPC (metadata authorization) and WebSocket (the credential is read from the upgrade request), not only JSON-RPC.

The valid token is accepted throughout, and the process stays healthy. The credential comparison is constant-time (only token length is treated as non-secret), and rejected tokens set no caller identity, so they cannot poison a rate-limit bucket.

Sustained concurrent load

probe_load.py is a correctness-under-pressure test, not a throughput benchmark: driven by a deterministic model-free executor, it isolates the SDK's task store, dispatch, and concurrency limit from model saturation. Five phases, each a hard assertion:

• Read storm — 64 concurrent workers for several seconds: ~3,600 req/s with zero transport errors and a bounded latency tail (p99 ≈ 50 ms).
• Store integrity — hundreds of tasks created concurrently, then listed: every task the server acknowledged appears exactly once, none lost or duplicated. This is the race-condition test, and it is the one that matters most — a store that drops or double-counts under concurrency is a silent data bug no single-request probe can reach.
• Limit enforcement — a burst of permit-holding requests past max_concurrent_tasks is refused with a structured Overloaded, never dropped or crashed (16 served, 112 cleanly refused at a cap of 16).
• Recovery — after the storms, single-request latency returns to baseline (≈1 ms), so nothing stayed wedged.
• Soak / leak — moderate mixed load over tens of thousands of requests while sampling the server's memory and file descriptors: RSS stays flat (~60 MB) and the fd count does not grow, so there is no per-request leak.

Streaming (SSE)

Streaming is where a black-box probe earns its keep: a client that opens a stream and walks away, or reads too slowly, exercises cleanup and backpressure paths no request/response test reaches. probe_stream.py opens thousands of streams and abandons them mid-flight (a RST while the task is still producing), holds hundreds open at once, and stalls a reader past the write timeout.

No leak — but establishing that took care. A naive measurement (settle 1.5 s, then read RSS) looked like a 25× memory leak. It was not: draining past the 5 s write timeout and running two equal bursts showed the growth converge (burst two grew a fraction of burst one), and the fd count returned to baseline every time — the earlier figure was allocator high-water-mark from the abandoned streams' transient cleanup, not retained per-stream state. This is the discipline the whole page rests on: a scary number that isn't a finding is worth exactly as much verification as one that is.

Push delivery — the server as an outbound HTTP client

Registering a webhook turns the server into an HTTP client dialing an address the caller chose — a different threat model from every inbound probe. probe_push.py bundles hostile receivers (hang, drip, a 50 MB response, an abrupt reset, a 302 to the cloud metadata endpoint) and — crucially — counts how often the server actually dialed each, so a run that delivers nothing fails loudly instead of passing vacuously.

That counter earned its place immediately: the first run delivered nothing, because the config field is taskPushNotificationConfig and the intuitive pushNotificationConfig is silently dropped by the parser. With the field fixed, the defended properties hold — the client reads the webhook's status, not its (50 MB) body, so a flooding receiver cannot grow its memory; it does not follow the 302 to metadata; and http targets are IP-pinned against rebinding.

The finding: sync delivery blocked the response

But delivery revealed a real gap. A non-streaming SendMessage awaited webhook delivery inline: a webhook the caller registered on that same request delayed it by up to the 30 s delivery budget — 15 s measured against a hanging webhook for a three-event task. Streaming and returnImmediately=true were unaffected, and it did not starve other clients (the blocked work is an async task, not a thread) — but it coupled a caller's request latency to a client-controlled endpoint, held task-store slots for up to 30 s, and did it for a fire-and-forget notification whose delivery errors are swallowed anyway. That is the webhook anti-pattern every production system avoids, and the streaming path already delivered off-path. Delivery is now spawned off the request path on the sync path too; SendMessage against a hanging webhook returns in ~10 ms instead of 15 s, the webhook is still dialed in the background, and a regression test pins it.

What this did and did not establish

It established that all three request bindings of a real, model-backed server — JSON-RPC, gRPC, and WebSocket — hold up under a broad, deliberately hostile input set; that the authentication layer rejects every forged, expired, and malformed credential across every binding with no bypass and no oracle; that the server stays correct, bounded, and leak-free under sustained concurrency and under abandoned/stalled streams; and that it withstands a hostile webhook receiver. Three gaps the runs surfaced — the numeric-IP SSRF filter, the gRPC version negotiation, and inline push delivery on the blocking SendMessage path — are closed and regression-tested.

It did not exercise a real identity provider's rotating RSA keys under load, nor multi-node deployments behind a shared store, nor a formal throughput/latency benchmark (the numbers here characterise the SDK on one box, not a capacity guarantee). Those remain open surfaces; see Security → Known gaps. Stating the boundary is the point — a security page, or a security test, that lists only what it covered is not one.