# Arm Dispatch Ledger — measured results (Apple M4 Max)

> **See also: [`results/REMEASURE-2026-08-04-QUIET.md`](REMEASURE-2026-08-04-QUIET.md).**
> That file is the authoritative, quiet/interleaved re-measurement of the phase-aware
> KleidiAI *patch* comparison reported in `results/OPTIMIZATION.md` (that comparison's
> original "+57.3% decode" figure is retracted). The tables below are a **separate**
> measurement (`tools/verify_dispatch.py` + `tools/bench.py`, unpatched baseline binary
> only, SME on/off — no phase-aware patch involved) and are not themselves part of that
> retraction; they carry the same general shared-machine-contention caveat as everything
> else measured in this session, but no cross-run baseline-vs-patched comparison is made
> on this page for the retraction to invalidate.

All numbers on this page were produced by code in this repo, run for real on this
machine, in this session, on 2026-08-03/04. Nothing here is invented or
interpolated. Commands to reproduce every table are given inline. The tables in
Sections 2 and 3 are the exact output of the commands shown, run last in this
session against the checked-in artifacts in `results/` — re-running the same
commands should reproduce them within normal run-to-run noise (a second
independent run of the dispatch sweep and two independent bench sweeps are all
noted below and agree qualitatively).

Hardware: Apple M4 Max, macOS 27, 16 cores (12P+4E), Apple clang 21 (Xcode-beta),
cmake 4.4. Model: Qwen2.5-0.5B-Instruct, Q4_0 GGUF (`/tmp/ggufs/q05.gguf`). No
Q8_0 GGUF was available in this environment — every Q8_0 row below and in
`results/bench/` is explicitly `[not available]`, never fabricated or
interpolated from Q4_0. llama.cpp: `dbadb68eecdfb3ab0e86872d011738fc937f0364`,
built `-DGGML_CPU_KLEIDIAI=ON`.

## 1. Kernel correctness (`kernels/`)

```
cd kernels && mkdir build && cd build && cmake -DCMAKE_BUILD_TYPE=Release .. && cmake --build . -- -j
./kernel_test        # exit 0
ctest                # 1/1 Passed
```

Real output (this run): NEON fp32 bit-exact across 8 shapes; SME2 fp32 max-rel-diff
0 across 8 shapes; SME2 int8 bit-exact across 8 shapes; SME2 Q4 L2-rel-error
0.0036-0.0055 (tol 0.02) across 5 shapes; SVE2 kernels correctly self-report `-1`
(unavailable) on this non-SVE2 host — `ALL CHECKS PASSED`, `EXIT=0`.

Kernel microbenchmark (`./kernel_bench`, single-thread): fp32 GEMM —
NEON (tuned) 99.4/96.0/49.7 GFLOP/s, SME2 (packed) 503.6/463.6/185.2 GFLOP/s,
Apple Accelerate 1607.4/3103.3/3408.0 GFLOP/s at N=512/1024/2048 respectively
(Accelerate is still fastest, by roughly 3.2x/6.7x/18.4x — never claimed otherwise;
N=512 is noisy at this timing resolution, see caveat below). int8 GEMM:
SME2 101.8/123.2/116.0 GOP/s at N=512/1024/2048, no Accelerate column since
Accelerate has no integer GEMM at all (the honest, non-strawman gap). Raw, persisted
artifact (added during adversarial review, 2026-08-04 — these numbers previously had
no file in `results/` backing them): `results/bench/kernel-bench-apple-m4-max.log` /
`.md`, which also documents run-to-run noise observed at N=512 (Accelerate swung
1607-2509 GFLOP/s across three reruns; N=1024/2048 stayed within ~5%).

## 2. Dispatch verification — confirms Finding 1 (`tools/verify_dispatch.py`)

```
python3 tools/verify_dispatch.py --binary /tmp/llama.cpp/build/bin/llama-cli \
  --model /tmp/ggufs/q05.gguf --threads 1,2,4,8,16 --workloads all \
  --out results/dispatch-ledger-darwin-arm64.json --assert
```

Ran three times in this session (twice manually, once via `scripts/run_all.sh` ->
`results/dispatch-ledger-Darwin-arm64.json`); all three agree exactly on which
configs dispatch SME2 vs. fall back, with hit counts stable within a few percent.

| threads | workload | advertised (L2) | executed (L3) | hits (adv/other) | verdict |
|---:|---|---|---|---|---|
| 1 | decode_short | SME2 | sme2 | 996/0 | **SME2_DISPATCHED** |
| 2 | decode_short | SME2 | sme2 | 5826/0 | **SME2_DISPATCHED** |
| 4 | decode_short | SME2 | dotprod | 0/15936 | **SILENT_FALLBACK** |
| 8 | decode_short | SME2 | dotprod | 0/31871 | **SILENT_FALLBACK** |
| 16 | decode_short | SME2 | dotprod | 0/51215 | **SILENT_FALLBACK** |
| 1 | prefill_long | SME2 | sme2 | 660/0 | **SME2_DISPATCHED** |
| 2 | prefill_long | SME2 | sme2 | 3853/0 | **SME2_DISPATCHED** |
| 4 | prefill_long | SME2 | dotprod | 2232/6711 | **SME2_HYBRID_DISPATCH** |
| 8 | prefill_long | SME2 | dotprod | 1538/13702 | **SME2_HYBRID_DISPATCH** |
| 16 | prefill_long | SME2 | dotprod | 1377/21534 | **SME2_HYBRID_DISPATCH** |

`--assert` correctly exits 1 on the 3 `decode_short` `SILENT_FALLBACK` rows and
does **not** flag the 3 `prefill_long` `SME2_HYBRID_DISPATCH` rows. This
reproduces Finding 1 exactly: llama.cpp's default `n_threads` (physical core
count, 16 here) silently never dispatches SME2 for single-token decode, even
though the startup banner still advertises `SME2`. Prefill is more nuanced — a
"hybrid" split-batch path keeps some real SME2 hits even at high thread counts,
but the run's own decode step still falls back, so we label it `HYBRID_DISPATCH`
rather than a flat fallback.

Full per-config JSON with L1/L2/L3 evidence: `results/dispatch-ledger-darwin-arm64.json`.
Note: `scripts/lib/verify_dispatch.sh` derives the platform slug from `uname -s`
(`Darwin`), so `scripts/run_all.sh` writes to `dispatch-ledger-Darwin-arm64.json`
while a direct `tools/verify_dispatch.py` invocation (this repo's own `--platform`
default) writes lowercase `dispatch-ledger-darwin-arm64.json`; on this machine's
case-insensitive-but-case-preserving APFS volume these resolve to the same file,
so the latest write (from the `run_all.sh` run in Section 5, generated_at_utc
2026-08-03T22:26:56Z) is what's on disk and is what the table above reflects. On
a case-sensitive filesystem (most Linux CI runners) these would be two separate
files — worth normalizing to one casing convention in a follow-up.

## 3. Throughput sweep (`tools/bench.py`)

```
python3 tools/bench.py --threads 1,2,8,16 --sme-modes on,off \
  --phases decode,prefill_short,prefill_long --reps 5 --skip-dispatch-verify \
  --out-dir results/bench
python3 tools/plot_results.py results/bench/bench-apple-m4-max.json --out-dir results/bench/figures
```

Interleaved (A,B,C,A,B,C...), warmup-discarding, 5 reps/cell, median/stddev/min/max
(never a bare mean). `--skip-dispatch-verify` was used deliberately: the decisive
dispatch determination for this project is the standalone lldb-based
`verify_dispatch.py` sweep in Section 2 above (a second, redundant lldb pass
inside `bench.py` itself risked the OOM crash a sibling work package already
reproduced on this same 48 GB machine — see `tools/protocol.md` §6.9). "SME=on"
means `GGML_KLEIDIAI_SME` is left unset (real-world default); "SME=off" means
`GGML_KLEIDIAI_SME=0`, which forces NEON regardless of thread count. This sweep
was run twice independently in this session (numbers below are the final run,
the one that produced the checked-in `results/bench/bench-apple-m4-max.json`);
the two runs agree within normal noise (a few percent on most cells, more on the
high-variance 16-thread cells — see caveats).

Real measured tok/s, Q4_0 (Q8_0 not available — no such GGUF in this environment):

### Decode (n_gen=32)

| threads | SME on | SME off (NEON forced) | on/off ratio |
|---:|---:|---:|---:|
| 1 | 208.9 +/- 2.9 | 149.9 +/- 4.3 | 1.39x |
| 2 | **327.6 +/- 4.6** | 266.4 +/- 6.4 | 1.23x |
| 8 | 154.6 +/- 1.8 | 155.3 +/- 0.8 | 1.00x (statistical tie — matches SILENT_FALLBACK) |
| 16 | 34.9 +/- 6.6 | 28.7 +/- 7.9 | high-variance oversubscription collapse, both paths |

### Prefill, short prompt (n_prompt=64)

| threads | SME on | SME off (NEON forced) | on/off ratio |
|---:|---:|---:|---:|
| 1 | 903.3 +/- 5.6 | 422.0 +/- 8.0 | 2.14x |
| 2 | 1579.0 +/- 8.5 | 806.4 +/- 17.3 | 1.96x |
| 8 | 2136.0 +/- 91.9 | **2188.4 +/- 61.0** | 0.98x — NEON alone ties/slightly beats SME's own ceiling |
| 16 | 888.9 +/- 185.1 (unstable) | 996.7 +/- 366.7 (unstable) | both unreliable at this thread count |

### Prefill, long prompt (n_prompt=256)

| threads | SME on | SME off (NEON forced) | on/off ratio |
|---:|---:|---:|---:|
| 1 | 896.2 +/- 4.7 | 415.1 +/- 6.0 | 2.16x |
| 2 | 1629.1 +/- 8.6 | 805.2 +/- 13.1 | 2.02x |
| 8 | 1830.1 +/- 203.5 | **2676.4 +/- 30.6** | 0.68x — NEON alone beats SME's own best cell by 1.46x |
| 16 | 445.3 +/- 100.5 | 1514.1 +/- 198.8 (unstable) | NEON still clearly ahead, both degraded |

*(Note: the stddev value `198.9` in this row is this section's own `tools/bench.py`
measurement — unpatched binary, `prefill_long`, 16 threads, SME off — and is unrelated to
the retracted `results/OPTIMIZATION.md` decode figure that happens to share the same
digits. Not affected by the `REMEASURE-2026-08-04-QUIET.md` correction.)*

Raw JSON/markdown: `results/bench/bench-apple-m4-max.json`, `results/bench/bench-apple-m4-max.md`.
Figures: `results/bench/figures/*.png`.

## 4. Reconciliation — does the performance story support Finding 1?

**Short answer: yes for decode, no for prefill — and that asymmetry is itself the
most interesting result in this dataset.**

The question posed to this analysis was specifically: *is SME2-at-2-threads
faster or slower than NEON-at-16-threads, for prefill and for decode?* Measured
directly from the table above:

- **Decode:** SME2@2 (327.6 tok/s) vs. NEON@16 (28.7 tok/s) -> **SME2@2 is 11.4x
  faster.**
- **Prefill (long):** SME2@2 (1629.1 tok/s) vs. NEON@16 (1514.1 tok/s) ->
  **SME2@2 is only 1.08x faster** — nearly a tie, well within this cell's own
  noise band.
- **Prefill (short):** SME2@2 (1579.0 tok/s) vs. NEON@16 (996.7 tok/s, high
  variance) -> **SME2@2 is 1.58x faster.**

Taken at face value, SME2@2 still wins (or ties) all three literal comparisons
the question asked for. But `threads=16` is a misleading NEON baseline on this
chip for this tiny model — the measured data shows 16 threads is a genuine
**oversubscription collapse** point for *both* kernel families (decode
throughput at 16 threads is worse than at 1 thread, for SME on **and** off;
stddev on several 16-thread cells is comparable to or larger than the median
itself). Comparing against NEON's actual best-observed thread count instead of
its worst gives a much more honest — and more surprising — picture:

- **Decode: SME2 wins outright, everywhere it was measured.** NEON's own best
  decode throughput, at any thread count measured, is 266.4 tok/s (at 2
  threads — NEON does *not* get faster with more threads either, for this
  workload). SME2@2 (327.6) still beats that by **1.23x**, and beats NEON's
  more "normal-looking" 8-thread number (155.3) by **2.11x**. There is no
  thread count at which plain NEON catches SME2@2 for decode. The 2-thread SME
  cap costs nothing here, because decode on this small model degrades with more
  threads *regardless of kernel family* — the cap happens to sit right at
  NEON's own sweet spot too.

- **Prefill: the honest comparison flips once NEON is allowed its own best
  thread count (8, not 16).** For prefill_long, plain NEON at 8 threads hits
  **2676.4 tok/s** — the single highest number anywhere in this sweep. That
  beats SME2's own best (1830.1 tok/s, also at 8 threads, where the hybrid path
  is doing some but not all of the work) by **1.46x**, and beats SME2 capped at
  its 2-thread sweet spot (1629.1) by **1.64x**. The same pattern holds for
  prefill_short: NEON@8 (2188.4) statistically ties/slightly beats SME's own
  best (2136.0, also @8), and both clearly beat SME2@2 (1579.0) by roughly
  1.35-1.39x.

**The honest conclusion:** SME2's `sme_thread_cap=2` (Finding 1) is a real,
measured net throughput loss for prefill on this chip once NEON is allowed to
use its own natural thread count — not just a dispatch curiosity. The moment a
workload can use more than 2 threads (which prefill, being embarrassingly
parallel across the batch, genuinely benefits from), unconstrained NEON
overtakes SME2 and stays ahead. SME2 only wins outright, at every thread count
measured, for decode — where the workload itself doesn't parallelize well with
more threads on this small model, so the 2-thread cap isn't actually giving
anything up. **This is not the flattering story a demo would pick — it is the
one the numbers actually show**, and it is a stronger, more falsifiable finding
than a flat "SME2 is faster" claim would have been: SME2's real-world win is
conditional on phase (decode yes, prefill no once NEON can use more cores), and
that conditionality is exactly what `detect_num_smcus()`'s hardcoded,
thread-count-oblivious cap in `kleidiai.cpp` cannot express.

Caveats on this reconciliation: (1) dispatch labels for the throughput sweep
come from a *separate* lldb-verified sweep (Section 2), not from a fresh
per-cell dispatch check inside the same `bench.py` run that produced the tok/s
numbers — the phase/workload naming lines up (`decode`<->`decode_short`,
`prefill_long`<->`prefill_long`) but `prefill_short` was never independently
dispatch-verified by lldb in this session, only inferred from the on/off
throughput ratio converging at 8 threads. (2) `threads=16` numbers for both SME
on and off have stddev comparable to or exceeding a meaningful fraction of
their own median on several cells — those cells are reported for completeness
(the task explicitly asks about them) but should be read as "this regime is
unstable," not as a precise point estimate; the prefill_long@16/SME=on cell in
particular swung from 452.1 to 445.3 tok/s across two independent runs (stable)
while its own single-run min/max spanned 255.5-528.8 (unstable within-run).
(3) Single-machine, single-model (0.5B) measurement — not yet reproduced on the
DGX Spark or on GitHub-hosted `ubuntu-24.04-arm`, and not yet measured at a
larger model size where prefill/decode compute-to-memory ratios differ.

## 5. `scripts/run_all.sh` end-to-end

```
LLAMA_CPP_DIR=/tmp/llama.cpp MODEL_PATH=/tmp/ggufs/q05.gguf \
  HF_FILE_SHA256=<sha256 of your local GGUF> BENCH_REDUCED=0 \
  ./scripts/run_all.sh
```

Verified in this session end-to-end, exit code 0, every stage OK (not SKIP, not
FAIL): `build_llamacpp` (reused the pre-built `/tmp/llama.cpp`), `fetch_model`
(reused `/tmp/ggufs/q05.gguf`), `build_kernels`, `capture_hw_features`,
`correctness_tests`, `verify_dispatch` (wrote
`results/dispatch-ledger-Darwin-arm64.json`), `run_bench` (wrote
`results/bench/bench-apple-m4-max.{json,md}` + figures), `emit_ledger` (wrote
`results/LEDGER.md`). Real wall-clock breakdown from this run: bootstrap ~77s
(cached, near-instant reuse checks), correctness tests ~11s, dispatch-verify
sweep (the expensive stage — 10 lldb-attached configs) ~9m39s, bench ~67s,
ledger emission <1s. `LLAMA_CPP_DIR` / `MODEL_PATH` / `HF_FILE_SHA256` were
pointed at this session's already-verified `/tmp/llama.cpp` build and
`/tmp/ggufs/q05.gguf` model purely to avoid a redundant clone/build/409MB
re-download of assets already proven to work in this same session — a bare
`./scripts/run_all.sh` with no env overrides does the full clone+build+download
from scratch and was not separately re-timed end-to-end in this session (its
constituent stages — `build_llamacpp` cloning by pinned SHA, `fetch_model`
downloading from the Apache-2.0 HF repo — were each exercised and verified
working in isolation by the sibling `ci` work package). Full per-stage log:
`results/stage-status.tsv` and `results/logs/*.log`. Note: `scripts/lib/run_bench.sh`'s
default invocation only sweeps `--threads` at the tool's own default (1,2,8);
the richer `threads=1,2,8,16` sweep in Section 3 above was run manually
afterward (same command shown there) to answer the reconciliation question in
Section 4, and is what `results/bench/bench-apple-m4-max.json` now contains.