Polygraph — prove the path before trusting the benchmark

Arm Create: AI Optimization Challenge · Cloud AI track

A fail-closed verification gate for Arm64 cloud inference.

Software can report acceleration while the intended path never runs. Polygraph checks the built binary, reads the runtime's own choice, and counts real Arm CPU kernel entries during inference. It then treats performance and promotion as separate controlled questions, turning a hopeful banner into evidence a cloud team can test, review, and use in CI.

What Polygraph proved — and what it rejected

A useful verifier must catch other software's mistakes and its own. These cards are read from the committed measurement JSON: the tested broken-build correction on one Arm system, and the later optimization candidate that a stricter paired gate refused to promote.

The Arm CPU is not the source of the claim: build/runtime software is. The first card is a broken-versus-corrected build comparison, not a new Polygraph kernel or universal speedup. The second shows why confirmed dispatch does not by itself prove better performance.

Caught a broken Arm build

–

Loading…

Rejected our own candidate

–

Loading…

Claim boundary

Loading the authoritative gate boundary…

Scope: Loading source scope…

– configurations verified (L1+L2+L3)
– silent fallbacks (advertised ≠ executed)
– hybrid dispatch (partial SME2)
– exact matches (advertised = executed)

Loading the dispatch ledger…

Advertised vs. executed

Advertised is what the startup banner and verbose log claim at model-load time (L2). Executed is what a debugger breakpoint on the real kernel entry points counted during inference (L3) — the only layer that proves anything ran. A mismatch is marked with both an icon and a text label, not colour alone.

Platform Threads Workload Advertised (L2) Executed (L3) Hits (advertised / other) Verdict
Loading dispatch-ledger JSON…

Throughput crossover — decode vs. prefill, SME2 vs. NEON, by thread count

Each chart is one workload phase on one platform: median tokens/sec (tok/s) at each measured thread count, with GGML_KLEIDIAI_SME left unset (“SME2 path”, the default) versus forced off (“NEON forced”). The x-axis is ordinal — thread counts are plotted evenly spaced and labelled, not to linear scale — because the sweep itself is sparse (1, 2, 8, 16). Series are told apart by line style and marker shape as well as colour. A numeric table backs every chart.

Loading bench JSON…

How this was verified

A timing-only benchmark cannot see a silent fallback: the run still completes, still prints a plausible tok/s number, and the banner still says the accelerator is enabled. This project's verifier (tools/verify_dispatch.py) never trusts fewer than three independent evidence layers. Those layers prove execution provenance for the measured workload; throughput, latency, memory, recovery, and deployment readiness remain separate gates:

L1 — static
Do the accelerated-kernel symbols exist in the built library at all? (nm/otool/objdump). Proves the kernel was compiled in — nothing about runtime behaviour.
L2 — selection
What does llama.cpp's own verbose log say it chose at model-load time? Still not proof of execution.
L3 — dispatch
Did the kernel's machine code actually run, counted live with a debugger breakpoint on every kai_run_matmul_* entry point during real inference? The only layer that answers the actual question, and the one every verdict on this page is based on.