Qwen2.5-1.5B on the Apple Neural Engine

What this project is

A native Apple Neural Engine runtime and compiler-research program that builds local intelligence through measured control of graph authorship, placement, state, memory, and runtime behavior.

It is not merely a model conversion, and it is not merely a speculative-decoding experiment. The work covers production serving on-device, hand-authored graphs, placement science, and original on-ANE control mechanisms — with negative results kept in the public record.

Measured performance

Throughput claims are separated by configuration. Numbers from different graphs or harnesses are never merged into one headline.

Configuration	Throughput	Status	Notes
Monolithic LUT4 · ctx 512 · B=1	51.6 tok/s	Proven	Production floor · warm min-of-burst · 19.4 ms/token · stateful KV
Monolithic LUT4 · ctx 512 · B=4	102.1 aggregate tok/s	Proven	Equal-length coalescing · ~99.5% ANE placement
Chunked LUT4 · ctx 1024	37.4 tok/s	Proven	Larger-context fallback
Hand-authored 28L LUT4 · split decode	23.78 tok/s	Proven	Authored-graph path · compute-bound at 28L · separate from production 51.6
Medusa (fused/trimmed)	28–36 tok/s	Closed	Slower than production baseline
EAGLE-3 closed-loop paths	below 51.6	No-go	Research-complete; not a shipped speedup

Major engineering achievements

These are completed, measured capabilities — not roadmap language.

Native Swift stateful ANE runtime

Production serving loop in Swift with a stateful Core ML KV cache, tokenizer, daemon, HTTP/SSE bridge, and MacBrains integration.

Hand-authored 28-layer LUT4 graph

A full Qwen2.5-1.5B LUT4 graph authored without relying on coremltools for graph authorship, compiled through Apple’s public toolchain, and executed on the ANE.

Full-model ANE residency

The authored graph loads and runs resident on the Neural Engine. Production KV machinery placement is ANE-side on validated paths.

Stateful KV-cache read/write

Production decode keeps key/value state on-device across steps rather than rebuilding history on every token.

Equal-length batch coalescing

Measured B=4 aggregate throughput of 102.1 tok/s on the production monolithic path, with safe fallback when requests are ragged.

Compiler & placement characterization

Native inspection of compiled ANE programs, placement probes, and empirical characterization of undocumented compiler behavior — without requiring private execution entitlements for the useful path.

Zero-copy local IPC

Shared-memory IPC between the inference daemon and MacBrains, removing TCP/JSON overhead from the local control path.

Published negative results

Medusa, multi-call speculative paths, and fused EAGLE monoliths that lost to baseline are retained as closed findings rather than deleted from the record.

Authored-graph quality proof

On the full WikiText-2 validation set, the hand-authored 28-layer LUT4 graph measures perplexity 16.8033 versus 14.79 for the Apple-converted reference used by the project — a +2.01 gap framed as the expected 4-bit cost, not a claim of beating Apple. Proven

Original On-ANE Inventions

We developed and demonstrated original graph-level mechanisms designed around the Neural Engine’s real placement, state, and compilation constraints. This section reports what was invented, what it does, and what was proven. Implementation recipes, topologies, and private workarounds are intentionally unpublished.

Proven · ANE placement

Associative local memory

We developed an associative-memory graph substrate whose page-selection and injection path reaches 99.5% ANE placement. Retrieval usefulness and wall-clock cost remain under measurement.

Placement proven; efficacy and dispatch cost UNMEASURED.

Proven · on-device

Activation firewall

We demonstrated a representation-level firewall that enforces policy constraints inside the on-ANE compute path rather than only in post-hoc host code.

On-device gate cleared for the fp16 substrate form under test.

Proven · on-device

Representation governance

We developed governance controls that steer or constrain internal representations while preserving Neural Engine placement.

On-device evaluation marked green in the invention campaign.

Proven · structural gate

Decoy Router

We demonstrated a compiler-enforced function-selection gate: the default function reaches a benign stub, while an explicitly selected function reaches the real ANE model. Structural gating / obfuscation — not adaptive routing or cryptographic security.

Blueprint evaluation green; outcome-level disclosure only.

Proven · B=4 vote

Quorum Lanes

We demonstrated a B=4 lane-agreement vote that detects one diverging lane. The model remains overwhelmingly ANE-placed; comparison/reduction operations ride the CPU sampling tail.

Blueprint evaluation green.

Proven · Neo GO

Free-rider probe bank

We demonstrated parasitic probe heads that ride an existing ANE dispatch to expose refusal, PII, and uncertainty signals with no material added dispatch cost on the shipped subset.

Real compiled artifact validated on-device; latency delta within the pre-registered bound.

Proven · gate-0

State-aware rollback & recovery

We developed state-aware rollback / branch-on-KV controls so a lane can rewind and recover without contaminating siblings — enabling safer speculative or multi-lane local control.

Ragged cross-lane isolation passed byte-exact gates on the SwiftMIL fp16 per-lane substrate.

Proven · placement

Pack-compose control primitives

We demonstrated composition of multiple control primitives in one package while retaining high ANE placement.

Placement proven; behavioral magnitude checks called out separately where still owed.

Live · efficacy unmeasured

Hallucination / consistency signaling

We developed on-ANE confidence and consistency signaling (including a hallucination-flag combiner) so the runtime can expose self-check features without a second host model round-trip.

Plumbing live on-ANE; end-task efficacy remains explicitly UNMEASURED.

Estimated · Stage-B

Dual-tap & fused judge fits

We developed dual-tap agreement and fused judge/critique head fits intended to verify or critique inside the same efficiency envelope as decode.

Cloud Stage-B fits marked GO; on-Neo confirmatory promotion still required before calling them production-proven.

What we intentionally do not publish

Exact graph topologies, tensor layouts, operator sequences, thresholds, training recipes, weight locations, unpublished compiler workarounds, and integration blueprints remain private. The public claim is capability and proof status — not a reconstruction manual.

System architecture

Production runtime

Native Swift tokenizer and CortexEngine serving loop
Monolithic stateful LUT4 Core ML model on ANE
Daemon + HTTP/SSE bridges for local clients
Zero-copy shared-memory IPC into MacBrains
Optional equal-length B=4 request coalescing

Compiler-research companion

Hand-authored MIL / Swift-native graph emission
Public coremlcompiler path — no entitlement bypass required for the useful workflow
Placement probes, program decode, and evidence ledger discipline

Research honesty

Speculative decoding produced real research signal. It did not become the shipped throughput win.

EAGLE-3

Offline and gated acceptance work was valuable, and several closed-loop implementations were measured on-device. None beat the 51.6 tok/s warm production baseline as a shipped speedup. The public page therefore treats EAGLE as research, not a production badge.

Medusa

Fused/trimmed Medusa measured 28–36 tok/s and is closed. Plumbing worked; the algorithm did not amortize against baseline.

Evidence labels

Every public number is tagged Proven Estimated Unmeasured Falsified / No-go so a modeled projection is never mistaken for shipped throughput.

Public artifacts

This site

Public narrative only. The production source repository remains private by design.

Related public work

ANE-RE research overview
Fugu — emit / compile / audit substrate (facts only; source private)

Think on the Neural Engine