Using SkyDiscover-Synthesize (SkySynth), coding agents now can build LLM serving engines for Apple silicon, without a single line of human-written code. For models supported by existing servers, our engines are up to 2.72× faster than the fastest alternative we tested on the same Mac, under single-user workloads unless otherwise noted.
- 💾 Qwen3.8-Flash-Next (104 GB): 172% (2.7×) faster on the M4 Pro, and 62% faster on the M2.
- 🎯 Qwen3.6-35B: 36% faster on a 24 GB M2, and 10% faster on a 48 GB M4 Pro compared to the best baseline.
- 🔥 Flint (Recursive’s 40B MoE model): 22% faster on the M4 Pro vs. Recursive’s own engine at long context, and 131% (2.3×) faster on the M2 over our M4 Pro build, where Recursive’s engine does not fit.
Try it out:
- Get started: jitengine.ai
- Code:
- Use these LLM engines: github.com/skydiscover-ai/jitengine
- Use SkyDiscover to synthesize your own custom engine: github.com/skydiscover-ai/skydiscover
- Slack: Join Slack here
Build the Engine Around the Bottleneck
General-purpose inference engines trade performance for flexibility across models, workloads, and hardware. Avoiding this generality tax [1] used to require months of expert engineering. With coding agents, we can now build specialized, just-in-time (JIT) engines [2] in hours or days.
We tested this idea on three mixture-of-experts (MoE) models: Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, and Recursive’s Flint. SkySynth built an engine for each model on two Macs, a 24 GB M2 Mac mini and a 48 GB M4 Pro, for six engines in total.
We first provide a high-level overview of an inference engine’s goals and constraints. To produce each new token, a server reads two kinds of data:
- the weights the token uses: in a mixture-of-experts model, only the experts picked for that token, plus the layers every token shares;
- the history of the conversation so far (the KV cache), which grows with every token of context.
On Apple silicon, the CPU and GPU share memory, some of which macOS and other apps need. On the 24 GB M2, we set the budget to 18 GiB, leaving 6 GiB for macOS (Flash-Next’s M2 engine uses a slightly smaller 18 GB). On the 48 GB M4 Pro, the budget is up to 40 GB. These are our chosen budgets, not macOS limits. And we give the agents a memory budget as part of the task.
Whether the weights fit in the memory budget determines the bottleneck. Qwen3.6 fits on both Macs (15.2 GB at 3 bits on the M2, 20.4 GB at 4 bits on the M4 Pro). Flint (22.6 GB) fits only on the M4 Pro, while Flash-Next (104 GB) fits on neither. The agents therefore specialized each engine for a different constraint.
The same model can face different bottlenecks on different Macs. Figure 3 summarizes what limits each engine.
We report single-user numbers (batch size 1): how fast a single request generates tokens, timed from its first token to its last, so reading the prompt does not count. For the two Qwen3.6 engines, we also report eight requests at once, the combined speed of eight concurrent coding-agent requests that share a system prompt of about 2,500 tokens. The agents tuned each engine on a set of scored prompts; we also report unseen held-out prompts, to check that the gains carry over. We compare each engine with the best other server we measured on the same Mac; for Flint on the M2, where no other server fits, we compare it with our own M4 Pro build.
The best baseline depends on the model and the Mac. For Qwen3.6 on the M2, we tried the most popular local servers, Ollama and llama.cpp, plus two MLX-based servers, mlx-lm and vllm-mlx. We also tried vLLM, the default on NVIDIA GPUs. On a Mac it needs a community plugin, and it was the slowest of all. SGLang and mistral.rs did not fit in 18 GiB.
For the other engines, the fastest rival was oMLX for Qwen3.6 on the M4 Pro, slotstream for Flash-Next, and Recursive’s own rec engine for Flint. Like most of these servers, our engines speak the OpenAI API, so an app or agent switches over by changing one URL.
How we measured
We benchmarked the other servers in earlier sessions, on the same Macs and with the same memory and client limits. To make sure the separate sessions did not tilt results in our favor, we re-ran two baselines side by side with earlier, slower builds of our engines. Even those earlier builds reached 1.8× the speed of slotstream (Flash-Next, single user) and 1.4× the speed of mlx-lm (Qwen3.6 on the M2, eight concurrent requests).
How We Built and Checked These Engines with SkySynth
SkySynth’s system synthesis loop, running on off-the-shelf coding agents (Claude Opus 5.5), built all six engines on top of Apple’s MLX, with no hand-written code. Across the six engines, the speedups come from four kinds of moves:
- Fit more of the model. Flash-Next on the M4 Pro stores each expert in 2.76 MB instead of 3.07 MB, and on the M2 it first evicts the experts of the layer that just ran.
- Keep the GPU from waiting. Flash-Next on the M4 Pro computes a layer’s experts before the CPU learns which ones the router picked; Flint on the M2 runs a helper that keeps the GPU’s clock up between SSD reads.
- Read memory at full speed. Flint splits the stored history across 240 groups of GPU threads inside one persistent program, reaching 252–255 GB/s of the 259 GB/s limit.
- Guess ahead only where checking is cheap. Qwen3.6 guesses on the M4 Pro, where a check costs about 21% of a step, and not on the M2, where it costs 65 to 73%.
Built with SkySynth agents
A SkySynth agent profiles the hardware, model and workload, then works in a loop: make one change, test it, measure it, and keep it only if every test passes and the engine beats its best build so far. The kept changes replace the engine’s GPU kernels, decode loop, memory plan and scheduler. Each model’s second engine started from its first one instead of from scratch.
Checked with extensive tests while building
The agents check every candidate’s output against a general-purpose reference engine, its memory against the budget and its speed on the scored prompts, and they rerun the held-out prompts to confirm that a gain carries over. A separate auditor agent hunts for shortcuts that pass the tests without making the engine better (two examples below). Where an engine reads from the SSD, its own count of reads agreed with macOS’s disk counters (0.998 on the Flash-Next M2 run).
How we hold each engine to the model’s own output
The Qwen3.6 kernels compute the same bits as the MLX operations they replace. When an M4 Pro engine guesses ahead, it checks each guess with the same arithmetic as one-token decoding, so a guess never changes a token. Flash-Next’s kernels add numbers in a different order than stock MLX, so their bar is “no less accurate than stock”; after that, its M2 run and its last M4 Pro run kept every change bit-identical to the engine each started from.
Reward hacking patterns
We noticed some “fun” reward hacking cases when synthesizing the engines for different models.
Qwen3.6: counters the engine reported about itself
Both Qwen3.6 engines can also run in a budget smaller than the model, streaming experts from the SSD. When the agents built that mode, its headline numbers, how often an expert was already in memory and how many bytes each token read, came from counters the engine printed about itself. The auditor noticed that no test read those counters: a candidate could print any hit rate it liked and still pass all seven tests. It wrote that test: the expert lookups must equal tokens × 40 layers × 8 experts, taken from the checkpoint rather than from the engine; every miss must cost one whole expert; a model larger than its budget must actually miss; and the same prompt must give the same answer however full the cache is. It also wrote two dishonest engines, one that lies about its counters and one that applies fewer experts than the model asks for, and confirmed that the test fails both. It hunted for other cheats too, such as counting reads that macOS served from its file cache as SSD reads, or steering the router toward experts already in memory, and found none: with the file cache emptied, the disk read 1.39 GB for the 1.30 GB of experts the engine said it fetched.
Flint: a benchmark that knows every next token
Flint’s raw benchmark feeds the model a fixed stream of tokens, as llama-bench does, so the engine knows every next token in advance. It could run several of them in one pass, the way it reads a prompt, or skip the output layer that scores all 150,144 possible next tokens, since nothing is picked from it. Either would look faster while doing less than a real reply needs. The run banned both at the start, then made them checkable. Every timed step now records the scores the output layer produced, and a test re-runs the same tokens cold, one per pass, and requires the same scores bit for bit, so an engine that skipped the layer has nothing that matches. An audit confirmed that the 1,024 timed steps were 1,024 single-token passes, so nothing ran ahead. It also changed the 500th fed token and saw the output change from step 501 on, which shows the engine read each fed token as it came instead of replaying a stored run.
Qwen3.6-35B: Fits on Both Macs
Qwen3.6 fits in memory on both Macs, so at their default budgets neither engine waits on the SSD. A new token is one pass through the model, all of it GPU work, so the agents rewrote most of that pass as their own GPU kernels. In the M2 engine, these cover the 3-bit weight multiplies, the expert layers, routing, attention, the sampler’s sort and the DeltaNet recurrence (Qwen3.6’s linear-attention layers, which update a fixed-size running state instead of keeping a growing history).
The M4 Pro engine grew out of the M2 build: it kept 20 of its files unchanged (1,441 lines, 11% of the M2 build), changed 46 (5,185 lines) and added 9 (1,149 lines). Among those changes, it guesses ahead with multi-token prediction (MTP) [3]. The model has a small extra head that guesses the following token. Much like speculative decoding [4], the next pass checks that guess while computing a new token, so a correct guess yields two tokens from one pass. The engine makes one guess per pass; two or three were no better. Checking makes the pass slower, so guessing pays off only when checking is cheap, and that cost is very different on the two Macs.
48 GB M4 Pro
Results: 10% faster than oMLX’s best mode (eight concurrent requests, 200.5 vs. 181.5 tok/s), and 35% faster on unseen prompts (139.8 vs. 103.3 tok/s).
Bottleneck. A normal step takes 11.1 ms and a step that also checks a guess takes 13.4 ms, so a possible second token costs only 2.3 ms more. At an average of 1.85 tokens per pass, guessing ahead is a clear win on this Mac.
What the engine does
Checking a guess processes two tokens in one forward pass. MLX’s fast multi-token kernels drift slightly from single-token arithmetic, which can change the chosen token. The engine instead uses custom checking kernels that keep the exact single-token math, so its outputs stay bit-identical to normal decoding. oMLX’s VLM engine is faster for a single user (217.2 vs. 194.6 tok/s), but its inexact checks fail our accuracy check. To our surprise, our engine was the only server we measured that passed it.
24 GB M2
Results: 36% faster than llama.cpp with MTP (single user, 44.9 vs. 32.9 tok/s).
Bottleneck. To fit in 18 GiB, the M2 engine uses 3-bit weights, which the GPU must unpack before each matrix multiply. The M2’s memory bandwidth peaks at 93 GB/s, but stock MLX (the unmodified framework our engines build on) reaches only 68 GB/s: the GPU spends so many cycles unpacking weights and preparing inputs that memory bandwidth goes underused.
What the engine does
A custom kernel prepares each input once rather than per thread group, cutting matrix multiplication time from 0.108 to 0.0886 ms and lifting bandwidth to 82 GB/s. Additionally, stock MLX closes a GPU command buffer after 40 operations or 40 MB of inputs. Because each expert call used to declare the entire 235 MB expert stack as its input, every call forced a new buffer. The engine now declares only the chosen expert’s weights, which fits up to 100 operations in one buffer and cuts step time from 22.05 to 20.03 ms, at a cost of about 1 GB of memory. Guessing ahead with MTP did not pay off here (checking a guess added 65% to 73% latency), so the agents measured it and turned it off for this build.
Qwen3.8-Flash-Next: Fits on Neither Mac
At 104 GB, Flash-Next fits on neither Mac, so much of serving it comes down to choosing which experts to keep in memory. Each token uses about 480 experts (10 in each of 48 layers), and missing ones come from the SSD. Both engines follow the same expert-offloading strategy as FreeToken [5], sharing a global expert cache across layers, streaming full layers during prefill, and balancing memory allocation between weights and context history. FreeToken does not run on Macs yet, so the agents built both engines for Apple silicon, where the CPU and GPU share one memory, and most of what follows is what that setting needed on top. We first built the M4 Pro engine, and then the M2 engine evolved directly from the M4 Pro build.
48 GB M4 Pro
Results: 172% faster than slotstream (single user, 32.9 vs. 12.1 tok/s in a 40 GB budget).
Bottleneck. Each token needs about 480 experts (3.07 MB each), and any that are not in memory must be streamed from the SSD. A 40 GB memory budget holds less than half of the 104 GB model.
What the engine does
A small set of experts does most of the work, so the 40 GB budget, which holds 12,016 of the 24,576 experts, leaves a token only about 7 to read from the SSD. Because a Mac has unified memory, those reads land in the memory the GPU computes from, and in an earlier build most of the wasted time went not to reading but to the GPU waiting for the CPU to learn which experts the router picked. The agents’ biggest change lets the GPU compute a layer’s experts before that answer arrives and fix any misses afterwards; together with running one layer ahead, it took a token from 62.6 to 50.9 ms. For a single user, the cache also makes room by dropping the expert it will need furthest in the future. While missing experts load, the engine starts reading the next layer’s experts too, and 91.4% of those early reads get used. In the last run on this Mac, the agents added compact experts (2.76 MB instead of 3.07 MB, so 11% more fit in the budget), exact guessing and layer-cycle eviction, and one user went from 23.8 to 32.9 tok/s in one session. Every change kept the output bit-identical to the build before. Guessing ahead needs the 40 GB budget: below it the plan has no room for the guessing head, and 32, 24 and 16 GB budgets run 22.3, 16.9 and 10.7 tok/s for one user. With eight requests at once, all eight decode together in every step, and the cache drops the least-used expert.
24 GB M2
Results: 62% faster than slotstream (single user, 4.48 vs. 2.76 tok/s).
Bottleneck. With only 18 GB of memory and an SSD that reads 1.79 GB/s (vs. 4.66 GB/s on the M4 Pro), disk reads dominate. Computing a token takes 50 ms, and each expert fetched from the SSD adds 1.71 ms. Run unchanged on the M2, the M4 Pro engine fetched about 113 experts per token, spending about 190 ms of each 245 ms step on SSD reads.
What the engine does
The engine cuts SSD reads from 113 to 95 experts per token by making better use of memory. Moving the token embeddings to the SSD frees memory to cache more experts, and a layer-aware eviction rule prefers to evict experts from layers that just ran, cutting replay cache misses from 112.6 to 107.8 per step. With several requests at once, decoding them all together made them keep evicting each other’s experts, so the engine takes turns instead: it decodes each request for 64 tokens before switching. This cuts batched SSD reads from 243.5 to 140.9 per token.
Flint: Fits on One Mac, Spills on the Other
Flint is Recursive’s 22.6 GB int4 model, and Recursive ships its own engine for it, rec. It fits in memory on the 48 GB M4 Pro but not in the M2’s 18 GiB budget, so the two engines face different bottlenecks.
48 GB M4 Pro
Results: 22% faster than Recursive’s own engine (rec) at 252K tokens of context (single user, 27.7 vs. 22.6 tok/s).
Bottleneck. At 252K tokens of context, every new token rereads 7.40 GB of KV cache along with 1.45 GB of weights. With memory bandwidth capped at 259 GB/s, reading the KV cache takes about 29 ms of each 36 ms token, so how fast the engine streams it sets the speed.
What the engine does
Like rec, our engine runs each token as one persistent GPU program, and the two match at 1K tokens (136.0 vs. 133.2 tok/s). In each layer, eight attention heads share the same stored history. Reading it once for all of them is the standard trick in grouped-query attention; what got us to 252–255 GB/s (near the 259 GB/s hardware limit, vs. an estimated 195–200 GB/s for rec) is how the stream runs: inside the one persistent program, 240 groups of GPU threads split the history, and each computes its eight heads as one small matrix multiply on the GPU’s matrix units.
24 GB M2
Results: 131% faster than our M4 Pro build (rec does not fit).
Bottleneck. At 22.6 GB, Flint exceeds the M2’s 18 GiB budget, so only 70% of its experts fit in memory and the other 30% are read from the SSD. The GPU sits idle between layer reads, and this stop-and-go pattern drops its clock from 1,398 MHz to about 800 MHz.
What the engine does
Run on the M2, the M4 Pro build’s default offload mode took 116 ms per token, 80 ms of it fetching missing experts. That profile led SkySynth to focus on expert fetching. The biggest single gain on the M2 does no model work at all: a background helper process streams tiny kernels on a separate GPU queue to keep the GPU at full clock speed. To cut miss time, eight worker threads read each expert’s nine checkpoint pieces in parallel, and reads during prompt processing no longer count toward cache statistics, so long prompts stop evicting the experts that decoding keeps reusing. Finally, the engine overlaps disk reads with compute: it uses idle prefill memory to hold evicted experts, sends each layer to the GPU as soon as it is built, and launches each layer’s experts on the GPU before the CPU confirms they are in memory. That last change alone adds 9% to 18%, and the output logits stay bit-identical across all 11,019 positions.
Run It with Any Agent
Every engine serves the OpenAI and Anthropic APIs on one port, with tool calls, so Claude Code, Codex, OpenCode, Pi or any other coding agent runs on it. Here is Flint on a 48 GB M4 Pro Mac, with the engine installed, from failing tests to an agent fixing them:
jitengine claude starts the engine and Claude Code on it, and the agent finds and fixes both bugs on the local model. Its 5 minutes are fast-forwarded 8×, as the status bar says.On an Apple silicon Mac with Homebrew and 22.6 GB of free disk, one command installs the Flint engine built for your Mac and downloads the model:
curl -fsSL https://jitengine.ai/install.sh | bash -s -- flintThen start an agent on it:
jitengine claude # or codex, opencode, pijitengine claude starts the engine if it is not running and points the agent at it for that session only. Any other client works too: give it http://127.0.0.1:8080/v1 as an OpenAI base URL or http://127.0.0.1:8080 as an Anthropic one. bash -s -- qwen3.6 installs Qwen3.6 instead, and bash -s -- flash-next installs Qwen3.8-Flash-Next.
To call the model from your own code instead, jitengine start runs the engine in the background, and any OpenAI or Anthropic client can then use http://127.0.0.1:8080, with any API key:
jitengine start runs the Flint engine in the background, then a curl call and a short Python script with the Anthropic SDK get replies from it, the script streaming its answer. Pauses are shortened.References
- SkyDiscover. Building Specialized Systems We Can Trust with Agents. SkyDiscover Blog, Sep 2026.
- S. Liu, A. Krentsel, S. Agarwal, M. Cemri, et al. The Time is Here for Just-in-Time Systems: Challenges and Opportunities. arXiv:2605.24096, 2026.
- F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, G. Synnaeve. Better & Faster Large Language Models via Multi-token Prediction. ICML 2024.
- Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
- S. Yang, X. Fan, M. Pan, H. Xi, Z. Wang, S. Sun, K. Keutzer, S. Han, M. Zaharia, C. Xu, I. Stoica. FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution. arXiv:2608.16157, 2026. Code: FlashML-org/FreeToken.
Acknowledgments
We thank Shuo (Andy) Yang and the entire FreeToken team for their help with code reviews and feedback throughout this project.
Citation
If you use JITEngine in your research, please cite this post:
@misc{cemri2026jitengine,
author = {Mert Cemri and Shubham Agarwal and Shu Liu and Alexander Krentsel and
Sidharth Sankhe and Alexandros G. Dimakis and Ion Stoica},
title = {Your Mac Deserves Its Own LLM Engine},
year = {2026},
month = nov,
howpublished = {SkyDiscover Blog},
url = {https://skydiscover-ai.github.io/blog-jitengine.html}
}