Authored graph quality
16.8033 pplx
Proven · WikiText-2 · vs Apple 14.79
Hand-authored 28L LUT4 · pplx 16.8033 Proven
Own the path to the Neural Engine
Measured placement. Labeled evidence. Negatives kept.
Single-stream floor
51.6 tok/s
Proven · production K=1 · 19.4 ms
Batch aggregate
102.1 tok/s
Proven · B=4 production path
Authored-graph decode
23.78 tok/s
Proven · 28L split · compute-bound
Research overview
Verified outcomes only. Strategic methods stay private.
What ANE-RE is
A native Apple Neural Engine runtime and compiler-research program that builds local intelligence through measured control of graph authorship, placement, state, memory, and runtime behavior.
It is not merely a model conversion project, and it is not merely a speculative-decoding experiment. The laboratory remains private. This page publishes verified outcomes and proof labels — not a reconstruction manual.
Strongest completed result
Hand-authored 28-layer LUT4 Qwen2.5-1.5B on the ANE
A full 28-layer LUT4 graph was authored without relying on coremltools for graph authorship, compiled through Apple’s public toolchain, and executed on the Neural Engine as two KV-sharing 14-layer ANE-resident halves (the hand-authored split path). On the full WikiText-2 validation set it measures perplexity 16.8033 versus 14.79 for the Apple-converted reference used by the project (+2.01, framed as the expected 4-bit cost, not a claim of beating Apple). Proven
Measured performance
Throughput numbers are never merged across configurations. Each row is a distinct measured path.
| Configuration | Result | Status | Notes |
|---|---|---|---|
| Production monolithic LUT4 · B=1 | 51.6 tok/s | Proven | Warm min-of-burst floor · 19.4 ms/token · stateful KV |
| Production monolithic LUT4 · B=4 | 102.1 aggregate tok/s | Proven | Equal-length cohort batching · ~99.5% ANE placement |
| Hand-authored 28L LUT4 · split decode | 23.78 tok/s | Proven | Compute-bound at 28L · separate from the 51.6 production floor |
| “Bandwidth ceiling = 57.3 tok/s” | — | Falsified | K=1 coincidence; regime at 28L is compute-bound |
| EAGLE / Medusa closed-loop speedups | below baseline | No-go / Closed | Research-complete; not shipped throughput wins |
Major engineering achievements
Public-path ANE residency
Useful results run through Apple’s public compiler / Core ML path. Private direct-execution entitlements are not required for the proven workflow.
Hand-authored production-scale graphs
Native Swift graph authorship (SwiftMIL) produces full-depth LUT4 models that load and run on the ANE.
Stateful KV on ANE
Production decode keeps key/value state on-device across steps on validated paths.
Multi-token verification loads
Real-emitter seq=K verify artifacts compile and load on ANE — opening measured speculative-decode research without treating synthetic probes as hardware law.
On-device program audit
Compiled artifacts are inspected after the public compile path — capability claim only; catalogs and recipes stay private.
Placement & toolchain science
Empirical characterization of placement, compiler-version sensitivity, and multiple distinct failure axes — with labels, not folklore.
Batch throughput lever
Measured B=4 aggregate throughput on the production path; batching survives as a real lever after bandwidth myths and several speculative paths failed fair tests.
Published negatives
LUT2 byte-cutting, fused EAGLE traps, Medusa, and prompt-injected fantasy throughput claims are retained as closed findings.
Original On-ANE Inventions
We developed and demonstrated original graph-level mechanisms around the Neural Engine’s real placement, state, and compilation constraints. Capability and proof status only — not recipes.
Associative local memory
We developed an associative-memory graph substrate whose page-selection and injection path reaches 99.5% ANE placement. Retrieval usefulness and wall-clock cost remain under measurement.
Activation firewall
We demonstrated representation-level policy enforcement inside the on-ANE path rather than only in post-hoc host code.
Representation governance
We developed governance controls that steer internal representations while preserving Neural Engine placement.
Decoy Router
We demonstrated a compiler-enforced function-selection gate: the default function reaches a benign stub, while an explicitly selected function reaches the real ANE model. This is structural gating / obfuscation — not adaptive routing or cryptographic security.
Quorum Lanes
We demonstrated a B=4 lane-agreement vote that detects one diverging lane. The model remains overwhelmingly ANE-placed; the comparison/reduction operations ride the CPU sampling tail.
Free-rider probe bank
We demonstrated parasitic probe heads that ride an existing ANE dispatch to expose safety/uncertainty signals without material added dispatch cost on the shipped subset.
State-aware rollback
We developed state-aware rollback / branch-on-KV controls with byte-exact ragged cross-lane isolation on the fp16 per-lane substrate under test.
Pack-compose primitives
We demonstrated composition of multiple control primitives in one package while retaining high ANE placement.
Hallucination / consistency signaling
We developed on-ANE confidence and consistency signaling. Plumbing is live; end-task efficacy remains explicitly UNMEASURED.
What stays private
Graph topologies, tensor layouts, operator sequences, thresholds, training recipes, weight locations, unpublished compiler workarounds, internal ledgers, and integration blueprints remain unpublished. This page is outcomes and proof labels — not a how-to.
Research methods
Evidence discipline
- Every public number is labeled Proven, Estimated, Unmeasured, Inconclusive, Falsified, or Superseded
- Toy probes are not promoted to hardware law
- Later corrections supersede earlier headlines when evidence demands it
Performance model (high level)
- Measured DRAM budget figures from compiler analytics inform the performance model
- Bandwidth saturation as a universal ceiling is falsified
- At full 28L depth, per-layer compute dominates the fixed dispatch floor
- Batching remains the practical throughput lever that survived fair tests
Research honesty
Speculative decoding
EAGLE-family work produced real research signal — acceptance, verify economics, harness bugs found and fixed. Closed-loop paths did not beat the 51.6 tok/s warm production floor as a shipped speedup.
Medusa
Fused/trimmed Medusa measured 28–36 tok/s and is closed. Plumbing worked; the algorithm did not amortize.
Evidence labels
Proven Estimated Unmeasured Falsified / No-go A modeled projection is never presented as shipped throughput.
Public artifacts
This site
Public narrative only. The ANE-RE laboratory repository remains private by design.
Related public work
Qwen2.5 ANE production-runtime site
Fugu — emit / compile / audit substrate (facts only; source private)