
Enigmus runs open-weight language models entirely on device, on iPhone, iPad and Mac, with MLX Swift doing the inference. There is no account, and no network traffic once a model has been downloaded. Version 1.2 shipped as three point releases, 1.2.0 through 1.2.2. This post covers the part worth writing down.

Seven changes, roughly ordered by how much they alter the app's behavior. Two are about which models run, three are about memory, two are about the chat screen. The memory ones are the least visible and took the most work.

## 1. The Qwen rows moved to 3.5 and 3.6

<IPadFrame
  src="/images/enigmus-1-2-install-list"
  alt="The model installer on an iPad with 12 GB of RAM and 7.5 GB free, sorted smallest first: Qwen 3.5 at 2B, 4B and 9B interleaved with Gemma 4 rows tagged MoE and their active parameter count, and Qwen 3.6 27B and 35B-A3B greyed out as not enough RAM"
  dark={false}
  className="relative mx-auto mb-10 mt-2 w-[300px] md:w-[420px]"
/>

The main list is now current-generation Qwen: 3.5 at 2B, 4B, 9B and 122B-A10B, and 3.6 at 27B and 35B-A3B, since 3.6 ships only those two sizes. That spans 1.7 GB to 69.6 GB of 4-bit weights, all with 256K context windows. The Qwen3 line moved to a legacy sub-menu, where the two smallest rows stay installable. The rest was removed.

This cost no dependency bump, which was the pleasant surprise. The pinned `mlx-swift-lm` 3.31.4 already registers `qwen3_5`, `qwen3_5_moe` and `qwen3_5_text`, and Qwen 3.6 reuses the 3.5 architecture outright: the `config.json` in `Qwen3.6-27B-4bit` declares `"model_type": "qwen3_5"`. Both generations load on the same stack.

There was exactly one real code change, and it was not in the loader. Qwen 3.5 and 3.6 ignore the `/no_think` suffix that Qwen3 understood. They gate reasoning on an `enable_thinking` chat-template argument instead, and default it to off. That is what forced the toggle work in item 6.

The mixture-of-experts rows carry a tag naming their active parameter count, because the size badge alone mis-sells them: decode cost tracks active parameters, not resident size. The tag deliberately says nothing about memory, and the RAM gate never reads it.

## 2. The bundled model is now Qwen3.5-0.8B

<IPhoneFrame
  src="/images/enigmus-1-2-benchmark"
  alt="The first-run benchmark screen for the bundled Qwen 3.5 0.8B-4bit model: load time, model memory, total memory, tokens per second, and a three-line generated sample"
  height={2608}
  dark={false}
  sizes="(min-width: 768px) 280px, 240px"
  className="relative mb-10 mt-2 w-[240px] md:w-[280px]"
/>

Every install ships with one model so that first launch works without a network. That used to be Qwen3-0.6B-4bit. It is now Qwen3.5-0.8B-4bit with the vision tower stripped out.

The reason for the swap is a single lineup, not a claim about output. The install list is Qwen 3.5 and 3.6 from top to bottom, so the model inside the app now belongs to the same generation as everything installable from it: the same chat template, the same reasoning switch, the same loader path. Bundling a Qwen3 model would have kept a deprecated family on every first launch, and Qwen3 is the family item 1 moved to a legacy sub-menu. The cost is about 100 MB more in the bundle and 85 MB more resident than the 0.6B it replaced.

The stripping is the interesting part. Qwen3.5's smallest size is 0.8B, every published repo for it is a vision-language checkpoint, and there is no official text-only repack. The loader discards the vision tower during weight sanitization anyway, so bundling the repo as published would put 201 MB of weights into the app that are read once and thrown away. A script removes them ahead of time.

That script's output has to be byte-deterministic, and this is a real constraint rather than tidiness: if the weights differ between builds, every App Store update re-downloads all of them. A Swift mirror of the strip exists so tests can assert against it.

## 3. Memory is decided in one place, and refusals carry numbers

No screenshot for this one, which is the correct outcome. The feature is that nothing happens.

The platform rule the app is built around: on iOS it must never be killed for memory. A jetsam kill is worse than every other failure mode, so an upfront "this will not fit" is preferred over more speed or more context. All of the arithmetic now lives in one type, `MemoryBudget`, which has no MLX dependency and so can be tested without a GPU. On iOS the ceiling is the current footprint plus the kernel's live jetsam headroom from `os_proc_available_memory()`, minus 400 MB of slack. On macOS, where swap exists, it is physical memory minus 3 GB.

The gap that needed closing was prefill. Prefill materializes the whole prompt's KV cache in fp16 in a single burst, inside a synchronous window loop with no callback, no cancellation check and no yield. The decode guard only samples between chunks, so a send with a long history could cross the per-app limit without the guard taking a single sample, and there is no way to abort mid-prefill from application code. The only available fix is to project the burst beforehand and decline.

That check sits between preparing the prompt and building the token iterator, so it runs before prefill and sees the exact prompt length without tokenizing twice. It can decline for two reasons, and both throw before any stream exists: the prompt plus the reply would exceed the context window, or the projected KV cache would push the footprint past the red line.

The reply is reserved up front, not just the prompt. Once the first token is out, a stop can only be reported in-band, after the caller has already been told the request succeeded. Paying for the whole span before starting converts a predictable mid-decode halt into a pre-flight refusal, which is the difference between an answer a caller can act on and a truncated stream it has to interpret. That shape is deliberate: the planned Mac server has to answer clients with a status code, and must never quietly trim a user's context.

Two guards sit behind the admission check, for what it cannot see. The decode loop samples the budget every chunk and stops cleanly at the red line, keeping the partial reply. A kernel memory-pressure handler covers the rest: growth while idle, and pressure caused by other processes.

## 4. Model fit is gated on measured residency, not download size

The install gate needs to know how much RAM a model will actually occupy. It used to estimate that as 0.8 times the download size, a ratio derived from Qwen3.5-2B, where the vision tower happens to be 38.5% of the repo.

The tower is roughly 0.9 GB flat, not a fixed fraction, so the rule under-predicted every row larger than the one it was derived from. Concretely, it put Qwen3.5-9B, which is 5.04 GB of text weights, inside an 8 GB iPhone's 5.0 GB budget. A model that could not run was installable.

Only measured numbers remain: 1.4 GB for the 2B, 4.77 GB for the 9B and 3.98 GB for gemma-4-e2b, all read off devices, plus a 14.09 GB floor taken from the safetensors headers for Qwen3.6-27B, acceptable only because that row is Mac-only. Five Gemma sizes were also simply wrong against their own safetensors headers, understating the storage gate by as much as 4.3 GB. Fixing them reordered two rows, since the list is sorted by size.

A related crash had the same root. Loading a model kept the outgoing one resident until the incoming one had finished, so switching from gemma-4-e2b to Qwen3.5-9B held about 8.7 GiB of weights at once on an 8 GB iPad, and jetsam killed the app in either direction. Loading now releases first, and restores the previous model if the new one fails to load. Separately, the memory readout in Settings was using binary units against a catalog that quotes decimal, so it read about 7% low.

## 5. The KV cache cap is gone

There was a `maxKVSize` setting, defaulting to 8192 tokens, with an override toggle and a slider in Settings.

It did nothing. No model family on the main list ever honored it, because Qwen3.5, Gemma and Qwen3-Next all construct their own caches. The default, the override field threaded through the generation parameters, and the Settings control were dead weight, and the context indicator was reporting the smaller of the cap and the real window, which made it wrong.

All of it came out. Conversations now run to the model's real context window, bounded by memory rather than by a number in a settings screen, and the indicator reports the true figure. The legacy Qwen3 dense rows lost their 8192-token ring-buffer eviction as a consequence, which was accepted: a token cap was only ever a proxy for a memory limit, and after item 3 the memory limit is measured directly.

One detail from the same cleanup. The context label shows a token count with no denominator, on purpose. It used to read as a fraction of the window, but a 262,144-token window reads as endless headroom on a phone that gets killed at a small fraction of it. The number's color tracks distance to the red line instead, which is the limit that actually binds.

## 6. A thinking toggle, derived from catalog metadata

<IPadFrame
  src="/images/enigmus-1-2-thinking-toggle"
  alt="The chat screen on an iPad showing a typeset derivation of Euler's formula, with the input bar underneath: the model chip, the thinking toggle set to fast, and a context counter reading 1,874 tokens"
  dark={false}
  className="relative mx-auto mb-10 mt-2 w-[300px] md:w-[420px]"
/>

Every family expresses reasoning differently. Qwen3 took a `/no_think` suffix. Qwen 3.5 and 3.6 take an `enable_thinking` argument and default it off. Gemma 4 has its own dialect again. So the app needs a per-row notion of what thinking even means before it can show a switch.

The toggle's appearance is a pure function of catalog metadata plus one resolved state bit, with three outcomes. Hidden, when a row has no reasoning to govern. Always-on and disabled, for rows that can only reason, where the disabled chip is what communicates that there is no choice. Live, for hybrid rows, the only case with a real decision in it. Keeping it pure means all three cases are tested without instantiating a view.

The state is per conversation, with an app-wide default as the seed. Both labels name the outcome rather than the mechanism, which is why the off position reads "fast" and not "thinking off". Turning it off is not an absence of thought, it is a quicker answer, and the vocabulary stays two words wide.

## 7. Two-row input bar, and fast by default on touch

The bar is two rows now, as in the screenshot at the top: the message field at full width, and the model picker, the thinking toggle and the send button underneath.

The second row never collapses, whether or not the field has text in it. It is the permanent home for the toggle and for the controls the bar is going to grow, and revealing them only once typing starts would hide them at exactly the moment a new user is looking for them. It costs a few points of idle height, which seemed the cheaper mistake.

Nothing in the bar carries decorative fill. The field and the controls sit on the page ground, and what delineates the input area is a single tinted stroke around it. Two elements do take a fill, both the same one: the model chip always, and the thinking toggle only while thinking is on. In the second case the fill is carrying state, not decoration.

The default for that toggle now differs by platform. New conversations start in fast mode on iPhone and iPad, and in thinking mode on the Mac. A phone decodes slowly enough that a reasoning preamble is most of the wait for a first reply, so on touch thinking is one tap away instead of the default. A Mac has the tokens per second to afford it. Only conversations with no preference of their own are affected.

## What is not here

The parts of 1.2 this post skips: reading a reply aloud, a per-reply decode rate, the running context counter, optional sampling overrides, the Mac menu bar work, and a pile of smaller fixes to the install flow and the message field. The changelog in the repository has the full list.

## Getting it

Enigmus 1.2.2 is a free update for the existing app on iPhone (13 or newer), iPad (M1 or newer), and Mac (Apple silicon). The inference engine is identical across all three. The difference is how much memory each has to spend on a model.

[Enigmus on the App Store, for iPhone and iPad](https://apps.apple.com/us/app/enigmus/id6771532268)

[Enigmus on the Mac App Store](https://apps.apple.com/us/app/enigmus/id6771532268?platform=mac)


Enigmus 1.2: Qwen 3.5 and 3.6, and memory handling improvements


The most quoted number about Google's [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) is a memory figure: a 2.58 GB Gemma 4 E2B model running in 607 MB. Against the 2,900 MB an MLX build of the same model occupies on the same phone, that is a 4.5x difference, and it is the number that makes LiteRT-LM look like a step change rather than an increment.

It is also the number that falls apart first. Google's own model card reports a different memory figure for every platform-and-backend row - 607 MB and 1,450 MB on the same iPhone - and a footnote on the card says Apple platforms were measured with one operating-system metric and Android with another. The two metrics do not count the same bytes, and neither counts what a phone actually has to hold in RAM.

Enigmus runs its inference on [mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm), and LiteRT-LM was evaluated as a possible replacement, so the figure mattered enough to take apart. This post is about what the memory number in an on-device LLM benchmark measures, using that evaluation as the worked example, and about what the rest of the LiteRT-LM stack looks like from a Swift project today.

One architectural fact frames it. Gemma 4's E-series uses per-layer embeddings: per the [Gemma 4 technical report](https://arxiv.org/abs/2607.02770), E2B has 2.3B effective parameters out of 5.1B in total, and the 2.8B gap is lookup tables that never enter the matrix multiplies. LiteRT-LM's model card states that "the runtime uses memory mapping to support the 1.12GB of embedding parameters", so roughly 1.12 GB of the file is paged in from disk on demand rather than loaded. Everything below follows from how that shows up when the process is measured.

## Google's own table, read properly

The [model card](https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm) publishes per-platform benchmarks. Collected into one place, with the decode speed each memory figure was measured alongside:

| Platform | Device | Backend | Decode | Memory | Metric used |
|---|---|---|---|---|---|
| iOS | iPhone 17 Pro | CPU | 25.0 tok/s | 607 MB | `phys_footprint` |
| iOS | iPhone 17 Pro | GPU | 56.5 tok/s | 1,450 MB | `phys_footprint` |
| macOS | MacBook Pro M4 Max | CPU | 41.6 tok/s | 736 MB | `phys_footprint` |
| macOS | MacBook Pro M4 Max | GPU | 160.2 tok/s | 1,623 MB | `phys_footprint` |
| Android | Galaxy S26 Ultra | CPU | 46.9 tok/s | 1,733 MB | `ru_maxrss` |
| Android | Galaxy S26 Ultra | GPU | 52.1 tok/s | 676 MB | `ru_maxrss` |
| Linux | Arm 2.3/2.8 GHz | CPU | 35.0 tok/s | 1,628 MB | `ru_maxrss` |
| Linux | RTX 4090 | GPU | 143.4 tok/s | 913 MB | `ru_maxrss` |
| Windows | Intel Lunar Lake | CPU | 29.8 tok/s | 3,505 MB | `PrivateUsage` |
| Windows | Intel Lunar Lake | GPU | 48.4 tok/s | 3,540 MB | `PrivateUsage` |

The model file is 2,583 MB in every row. Three things fall out.

**The famous memory number and the famous speed number are different rows.** 607 MB is the CPU backend, decoding at 25.0 tok/s. Google's [announcement post](https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm/) attributes the 607 MB figure to "Apple mobile CPUs utilizing XNNPACK's weight caching mechanism", XNNPACK being a CPU inference library. The same post's headline speed claim, 56 tok/s on iOS, is the Metal backend, and on that backend the card reports 1,450 MB. An Apple app shipping LiteRT-LM would use Metal, so the operative iPhone figure is 1,450 MB: 2.4x the quoted number, for the same file on the same phone.

**The direction of the CPU-to-GPU difference reverses between Apple and Android.** On iOS, moving to the GPU raises reported memory (607 -> 1,450). On Android it lowers it by a similar factor (1,733 -> 676). A property of a runtime does not invert with the operating system. A property of a *measurement* does.

**Every figure outside Windows is below the 2,583 MB file, and both Windows figures are above it.** The same file, mapped the same way, is reported anywhere from a quarter of its size to well over it depending on which counter is read.

## The footnote that explains it

The model card documents its own methodology, and this sentence resolves the table:

> CPU memory was measured using `rusage::ru_maxrss` on Android/Linux/Raspberry Pi, `task_vm_info::phys_footprint` on iOS/macOS, and `process_memory_counters::PrivateUsage` on Windows.

`ru_maxrss` is peak resident set size: physical pages the process has resident, *including* clean file-backed pages from a mapping. Map a gigabyte of embeddings and touch it, and `ru_maxrss` shows it.

`phys_footprint` is Apple's footprint metric, and it deliberately excludes those pages. Apple's definition, from [WWDC22 session 10106](https://developer.apple.com/videos/play/wwdc2022/10106/):

> For clean memory pages, they include read-only files mapped from disk, such as texture or audio assets, and frameworks loaded into the process. The system can empty or reload them from disk at any time, so they don't count towards your game's memory footprint.

So on Apple platforms the 1.12 GB of mapped embeddings is definitionally invisible to the number being reported, and on Android the equivalent pages are counted. 607 MB on iOS against 1,628-1,733 MB on Linux and Android are not measurements of different efficiency. They are measurements of different quantities.

The same session explains the reversal. Describing dirty memory, which does count:

> On devices with Apple silicon, accessed Metal resources also fall into this category - this is because CPU and GPU share the same pool of fast unified memory.

On Apple silicon the GPU path moves weights out of exempt clean mappings into fully charged Metal buffers, and the footprint goes up. On Android the GPU path moves them into OpenCL allocations outside the process's resident set, and the RSS goes down. Same underlying change, opposite accounting.

Apple attaches the necessary caveat to its own exemption in the same session: clean pages "may be resident on memory, and excessive use will slow down the system". Uncharged is not free. The pages are in RAM, competing for it.

## What is actually real

Netting the artifact out does not leave nothing.

**Mixed-bit quantization is real.** The E2B `.litertlm` is 2.6 GB at a mixture of 2-, 4- and 8-bit weights, against 4.0 GB for the uniform 4-bit MLX build in the Enigmus catalog. A smaller file is a smaller resident core on any metric. It is also a different quality point, so output is not directly comparable in either direction.

**Demand-paging a cold 1.12 GB tail is real**, not as a reduction in bytes needed but as a change in *what kind* of bytes they are. Clean file-backed pages can be evicted under pressure and re-faulted from SSD. Dirty Metal buffers cannot. On iOS they can only be compressed or swapped, or the process gets killed.

That second point cuts in LiteRT-LM's favour on iOS, because jetsam terminates apps on `phys_footprint` - the same metric that exempts clean mapped pages. What looks like a measurement trick in a benchmark table is a survival advantage in the field: under memory pressure a mapped-weights runtime gets slower, and a Metal-buffer runtime gets killed. Apple publishes no per-device limits, but a [developer report on Apple's forums](https://developer.apple.com/forums/thread/702400) puts `os_proc_available_memory` on a 4 GB iPhone 13 at about 2.2 GB. Against that ceiling a 2,900 MB MLX build does not fit and 1,450 MB does.

On the backend an Apple app would actually ship, then, the memory gain is closer to 2x than 4.5x, and the remaining advantage is as much about survivability as about size. None of this generalizes beyond the model family: the artifact is large here because the E-series puts an unusual share of its parameters into cold lookup tables. A conventional dense model has no large mmap-able tail to hide.

## Two more measurements, neither reconcilable

The most cited third-party comparison is [john-rocky's iPhone 17 Pro runtime benchmark](https://dev.to/john-rocky/on-device-llm-on-iphone-which-runtime-is-fastest-mlx-vs-llamacpp-vs-litert-lm-vs-coreml-1b42): Gemma 4 E2B, 128-token chat, median of three cold runs.

| Runtime | Decode | Peak RAM |
|---|---|---|
| LiteRT-LM | 55.4 tok/s | 641 MB |
| MLX-Swift | 47.5 tok/s | 2,900 MB |
| llama.cpp | 37.8 tok/s | 3,156 MB |
| Core ML / ANE | 33.4 tok/s | 1,187 MB |

Its two LiteRT-LM columns come from different rows of Google's table: 55.4 tok/s matches the card's GPU figure of 56.5, but 641 MB sits next to the card's *CPU* figure of 607 MB, while the card's GPU figure for the same phone is 1,450 MB. The benchmark does not state which memory API it used, so the 2.3x gap cannot be resolved from the published material.

The in-house comparison was run on one 24 GB Apple-silicon Mac with the same prompt: MLX on the 4.0 GB uniform 4-bit build, LiteRT-LM v0.13.1 on Metal with the 2.59 GB mixed-bit `.litertlm`.

| Metric | MLX 4-bit | LiteRT-LM E2B | Note |
|---|---|---|---|
| Load time | 4.54 s | 1.35 s warm | full read into Metal buffers vs mmap |
| Reported memory | 4,077 MB (`GPU.activeMemory`) | +479 MB peak (`phys_footprint`) | **different metrics - not a ratio** |
| Tokens/sec, wall | 33.30 | 33.9 | parity |
| Tokens/sec, generation only | 33.79 | 39.3 (53.6 sustained) | +16% |

The memory row shows the only figures each runtime exposes. They are not the same quantity, and the LiteRT-LM figure does not reconcile with Google's 1,623 MB for macOS GPU either: a delta from process baseline on a different machine, with a backend that logs itself as WebGPU over Metal, against an absolute figure from an undocumented setup.

The speed row is trustworthy: +16% on generation-only decode, consistent with the +17% in the iPhone benchmark. Two agreeing measurements on different hardware put the Gemma decode advantage in the mid-teens percent before multi-token prediction, well short of the ["1.8x to 3.7x faster prefill and decode than llama.cpp, MLX, Cactus, ONNX"](https://www.infoq.com/news/2026/06/google-litertlm-gemma4/) reported from Google's launch material. Prefill is where LiteRT-LM's margins are large, and that figure reads as prefill-weighted.

The load-time difference relocates cost rather than removing it. MLX pays the whole cold cost in the load bar. LiteRT-LM maps the file, so after a `sudo purge` initialization does not change (1.32 s cold against 1.35 s warm) and the cost reappears in the *first generation* as pages fault in from SSD: time-to-first-chunk of 1,145 ms against 703 ms, first-generation decode of 31.8 against 39.3 tok/s.

No comparison here holds KV-cache size constant, and it is a first-order term in footprint. The LiteRT-LM runs used `maxNumTokens: 512`, the MLX side runs uncapped with 4-bit KV on iOS, and the iPhone benchmark states no context length.

## The rest of the stack

Three practical findings matter more to adoptability than the measurement question does.

**The Swift package cannot be installed the way Google documents it.** [The Swift guide](https://developers.google.com/edge/litert-lm/swift) says to add the repository URL in Xcode. That fails for three independent reasons. The manifest compiles the wrapper with `linkerSettings: [.unsafeFlags(["-Xlinker", "-all_load"])]`, and SwiftPM refuses products with unsafe flags as *remote* dependencies; only local path references are exempt. The repository LFS-tracks around 44 prebuilt binaries, and SwiftPM checks out from a local bare mirror that never fetches LFS objects, so on any machine with git-lfs installed the checkout dies with a misleading `remote missing object` error. And v0.14.0's Swift sources call C symbols that do not exist in the v0.13.1 binaries its own manifest pins, because no v0.14.0 XCFramework was released. The newest self-consistent tag is v0.13.1, and any integration needs a committed local copy: submodule, vendored sources, or a bootstrap clone.

**The models are license-gated, and their hashes are masked.** `litert-community` Gemma repositories return 401 on `resolve/` downloads anonymously, unlike the un-gated `mlx-community` mirrors. Hugging Face also masks LFS sha256 object IDs on gated repositories, so a download cannot be pinned to a hash without a token. Shipping would mean self-hosting the files under the Gemma license's redistribution terms, at about 2.6 GB per install, or a bring-your-own-token flow.

**GPU capability is a per-conversion property, and the runtime does not check it.** `gemma-3-270m-it` at q8 is not GPU-capable; its card says GPU support is still in progress. Requested with `backend: .gpu` it initializes without complaint, streams at 110 tok/s, and emits pure special-token soup - `<pad><unused16>...` - with no error and no warning. An integration has to gate model-by-backend compatibility itself, and its tests need a coherence check rather than a non-empty check.

Two further gaps decide the question. `mlx-community` holds thousands of checkpoints loadable as plain safetensors; the `.litertlm` catalog is a short list with no Qwen3 30B or 80B MoE equivalent and no supported conversion path, so an entire product line does not run on it. And LiteRT-LM documents no equivalent of `GPU.activeMemory`: the runtime whose headline advantage is memory cannot report its own memory use to the application.

## Where Enigmus lands

LiteRT-LM is not a replacement for the MLX pipeline today. The runtime itself is good: 16-17% faster Gemma decode, faster loads, large prefill margins, and a survivability advantage on memory-constrained iPhones that outlives the deflated headline figure. What blocks it is everything around the runtime: a package that cannot be added as documented, a broken newest tag, gated models with masked hashes, a closed-source prebuilt core, a catalog that omits the models most used, and no memory introspection.

The re-check conditions are concrete, and all four are Google's to fix: a manifest without unsafe build flags or an official binary-only package; a repository without LFS-tracked build inputs; tags whose Swift sources and shipped binaries agree; and a model channel that is un-gated or authenticable from inside an app. When those land, the evaluation is worth running again - on one machine, with one metric, named.


607 MB or 1,450 MB: what LiteRT-LM's memory number measures


At WWDC 2026 Apple introduced [Core AI](https://developer.apple.com/videos/play/wwdc2026/324/), a new framework for running machine-learning models on Apple silicon — in Apple's words, "the inference framework powering on-device Apple Intelligence" — and, alongside it, an open-source repository that turns the framework into a practical local-LLM stack. For anyone shipping on-device language models on Apple platforms, this is the most consequential platform change since MLX appeared. Enigmus runs its inference on [mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm) directly today, and a transition to Core AI is under consideration — but not a quick one, for reasons covered below. This post is a technical first read: what the framework actually is, what is genuinely new, and where it beats or loses to building on MLX's LM layer directly.

## What Core AI is

Core AI is a general inference runtime, not an LLM framework. The press calls it the Core ML successor; Apple's own docs are more careful — Core ML is not deprecated, and the two are positioned side by side, with Core ML keeping classic model types and Core AI taking "the latest model architectures and inference techniques." It runs models across CPU, GPU, and Neural Engine, and its central idea is *specialization*: a `.aimodel` bundle is a device-independent source representation, exported from PyTorch via `torch.export` with first-class dynamic shapes. On first load it is compiled for the exact device in two phases — compute is segmented, planned, and optimized, then executable artifacts are generated for the compute units in use — and the result lands in an `AIModelCache` that can even be shared across apps in an app group. Ahead-of-time compilation via `coreai-build` shrinks, but does not remove, that first-load cost. There is a dedicated Core AI Debugger app plus Xcode instruments for profiling. Availability is 27.0 on every platform, and every entry is marked beta.

The LLM layer sits on top, in [apple/coreai-models](https://github.com/apple/coreai-models) (BSD-3-Clause, first released June 2026). It contains model export recipes that convert Hugging Face checkpoints into `.aimodel` bundles, a Swift runtime with four engine strategies — a pipelined GPU engine, a sequential engine, a static-shape engine aimed at the Neural Engine, and a VLM variant — plus KV-cache management, samplers, tool-call parsing, and grammar-constrained generation built on XGrammar (the same constrained-decoding engine MLX adopted, so the two stacks converged there). Export recipes currently cover Gemma 3, GPT-OSS, Mistral, Mixtral, and the Qwen families for LLMs, with a longer tail of vision, audio, and diffusion models.

## What is actually new

Three things stand out against the status quo of Core ML on one side and MLX on the other.

**A working Neural Engine path for LLM decode.** Core ML never handled autoregressive decoding well — static shapes, awkward KV-cache state, and limited ANE coverage of the decode loop. Core AI's static-shape engine targets the ANE directly, and the repository ships explicit transformer-on-ANE authoring rules: keep the whole model resident on the Neural Engine, per-head attention instead of fused SDPA, fp16 only, palettized weights. (Apple never states outright that LLM decode runs on the ANE — the claim is verified at the code level, not in marketing copy.) The one independent benchmark available ([apple-silicon-llm-bench](https://github.com/john-rocky/apple-silicon-llm-bench)) measured Qwen3-0.6B decode on an iPhone 17 Pro at 49 tok/s on the ANE via Core AI against 39 tok/s for the legacy Core ML path — not fast in absolute terms, but running decode off the GPU entirely, which matters for sustained workloads and thermals.

**A faster GPU engine for small models.** The same benchmark measured Core AI's pipelined GPU engine at 181 tok/s steady-state on Qwen3-0.6B (iPhone 17 Pro) against 112 tok/s for MLX — roughly 1.6× once warm. Apple offers no explanation for the gap; the mechanism has to be read out of the engine's source, which describes a three-stage pipeline with rotated buffers so CPU and GPU work overlap, token sampling (argmax/top-k) executed directly on the GPU via MPSGraph compute shaders instead of round-tripping logits to the CPU, and a growing KV cache with pipelined expansion. The caveats are load-bearing: the first generation pays a cold-start cost (about 71 tok/s while kernels compile and the three-stage pipeline fills), and the advantage collapses as models grow. On an M4 Max the gap is ~2.47× at 0.6B but only ~1.05× at Qwen3-8B, where memory bandwidth rather than compute is the bottleneck — and on the Mac the same source places MLX, not Core AI, on the energy-per-token Pareto frontier. These are single-author numbers with documented session-to-session variance, so they should be read as directional. Apple itself publishes no throughput figures at all.

**One generation API over swappable engines.** The Foundation Models framework shipped in OS 26, but OS 27 adds beta protocols — `LanguageModel` and `LanguageModelExecutor` — that let any third-party engine sit behind `LanguageModelSession`, the same API that drives Apple's built-in system model. `coreai-models` ships the reference implementation, and MLX ships its own bridge (`MLXFoundationModels`). An app can adopt `LanguageModelSession` as its only generation surface and choose the backend — Apple's system model, a Core AI `.aimodel`, or an arbitrary MLX checkpoint — behind one protocol, with structured output and tool calling handled by the framework.

## Against mlx-lm directly

Enigmus's current stack is MLX with mlx-swift-lm on top: the model registry, config decoding, weight loading, and the token loop, pinned to exact revisions. Measured against that, Core AI's trade-offs look like this.

In Core AI's favor:

- **First-party runtime.** The engine ships with the OS and is maintained by the platform vendor, with a debugger and profiling instruments. MLX is also Apple-run, but as an open-source project outside the OS, with the app carrying the runtime.
- **The ANE is reachable.** MLX runs LLMs on the GPU; Core AI is the only supported route to Neural Engine decode.
- **Faster small-model decode on iPhone**, per the numbers above — the regime a phone actually operates in.
- **Specialization caching and AOT compilation** — kernel compilation cost is explicit, cacheable, and can be paid at build time rather than first use.
- **Constrained decoding, tool-call parsing, and VLM support in the box**, behind the same session API.

Against it:

- **OS 27 or nothing.** Everything above requires iOS/macOS 27 — in public beta as of late July 2026, with general availability expected around September but not officially dated. Even after release, adopting Core AI as the only engine means dropping every user still on OS 26 or earlier and every device that cannot upgrade. This is the single biggest reason a shipping app cannot simply switch, the honest pain point of the whole story — and the loudest complaint in the [community discussion](https://news.ycombinator.com/item?id=48449665) too.
- **A closed model catalog.** Models must be exported to `.aimodel` bundles through the repo's recipes, which cover a short list of architectures. mlx-swift-lm loads thousands of community-quantized checkpoints straight from Hugging Face as safetensors — no export step, no per-model recipe. For an app whose model picker is a catalog of community builds, this gap is structural.
- **Export-environment sensitivity.** The benchmark author found the identical export recipe produced artifacts 2.2× slower when run on the macOS 27 beta SDK versus the macOS 26 SDK. Quantization presets also differ by platform (plain INT4 on macOS, palettized 4-bit on iOS), so an exported bundle is not one artifact but a per-platform matrix.
- **Beta protocol churn, closed development.** The `LanguageModel`/`LanguageModelExecutor` protocols are beta, and `coreai-models` is not accepting pull requests — issues only. mlx-swift-lm's development happens in the open at roughly 150 commits per quarter.
- **No first-party performance data.** Apple ships an `llm-benchmark` tool but publishes no numbers; every figure above comes from one independent benchmarker.

## Building a local-LLM CLI with Core AI

A detail worth spelling out: Apple ships Core AI LLM inference as command-line executables. `coreai-models` declares `llm-runner` and `llm-benchmark` as SwiftPM executable targets built on `swift-argument-parser` — no app bundle, no Xcode project, no entitlements dance. That makes a private, local LLM CLI on a Mac a short exercise:

```bash
# 1. Export a checkpoint to an .aimodel bundle (runs on the dev machine)
uv run coreai.model.registry --list-models
uv run coreai.llm.export Qwen/Qwen3-0.6B                 # macOS variant, 4-bit default
uv run coreai.llm.export Qwen/Qwen3-0.6B --platform iOS  # palettized 4-bit for iPhone

# 2. Run it
swift run -c release llm-runner \
  --model path/to/exported_model_folder \
  --prompt "Summarize the Bell inequality in three sentences."

# 3. Measure it (512-token prompt, 1024 generated, 5 trials by default)
swift run -c release llm-benchmark --model path/to/exported_model_folder
```

`llm-runner`'s flag set doubles as a map of what the stack can do: `--json-schema` for constrained decoding, `--kv-cache-strategy`, `--warmup` for pre-paying kernel compilation, `--image` for VLM input, and `--inference-engine-variant` to pick between the pipelined, sequential, and static-shape engines. It prints a performance summary and memory usage on exit.

A custom CLI is one SwiftPM executable that depends on `CoreAILanguageModels` and `FoundationModels`:

```swift
import FoundationModels
import CoreAILanguageModels

let model = try await CoreAILanguageModel(resourcesAt: modelURL)
let session = LanguageModelSession(model: model)
let response = try await session.respond(to: prompt)
print(response)
```

(For Apple's built-in system model there is an even shorter path — macOS 27 preinstalls an `fm` command-line tool, with a companion Python SDK — but that drives the system Foundation Model only, not custom weights.)

The privacy read here is unusually auditable. The only network code in the entire `coreai-models` repository is the Hugging Face download inside the Python export tooling, which runs on the development machine; the Swift runtime contains no networking at all — it loads a local resources folder and generates. The OS-side framework is closed source, so Apple's "zero server dependencies" claim has to be taken on its word there, but the open-source layer can be checked directly.

## Where Enigmus lands

Enigmus stays on mlx-swift-lm for now. The OS 27 floor alone settles the near term — a privacy app does not abandon its installed base for a beta OS. The catalog question matters just as much: the app's model picker is built on community MLX checkpoints fetched directly from Hugging Face, and Core AI has no equivalent supply today.

But the transition is being considered seriously, and the architecture is being shaped for it. The practical reading of the Foundation Models protocols is that the generation layer of an app should become engine-agnostic: adopt `LanguageModelSession` semantics internally, keep mlx-swift-lm as the backend that ships, and leave the seam where a `CoreAILanguageModel` could slot in once OS 27 is the floor rather than the ceiling. In the meantime, the numbers worth trusting are the ones measured locally — `llm-benchmark` exists precisely so that the decision, when it comes, rests on device-measured tokens per second rather than anyone's blog post. Including this one.


Mlx

Enigmus 1.2: Qwen 3.5 and 3.6, and memory handling improvements

607 MB or 1,450 MB: what LiteRT-LM's memory number measures

Core AI: a first look at Apple's new on-device model runtime