TurboFieldfare Runs Gemma 4 26B-A4B on Apple Silicon in 2 GB RAM
Independent iOS and Metal engineer Andrey Mikhaylov has released TurboFieldfare, a custom inference runtime that enables local execution of the 26-billion-parameter Gemma 4 26B-A4B instruction-tuned model on Apple Silicon Macs with as little as 2 gigabytes of system memory. The project represents a significant optimization in on-device large language model deployment, bypassing traditional memory-bound constraints through innovative expert streaming and hardware-specific acceleration. TurboFieldfare operates by keeping a 1.35 gigabyte shared model core and FP16 key-value cache in RAM while streaming only the necessary mixture-of-experts weights directly from the solid-state drive for each generated token. The architecture utilizes a custom Swift and Metal runtime, complete with bespoke kernels for quantized matrix operations, attention mechanisms, and routing logic. Rather than relying on existing inference wrappers, the project implements its own memory management and compute pipeline. An 8-bit router dynamically selects top experts, which are cached using a bounded least-frequently-used strategy and processed in parallel with resident shared-expert computations. Prompt prefilling handles up to 128 tokens per chunk, while token-by-token decoding follows a strict single-file generation loop. Benchmarking across Apple Silicon hardware demonstrates the runtime's practical viability. On an 8 gigabyte M2 MacBook Air, the system achieves approximately 5.1 to 6.3 tokens per second during decoding. More powerful configurations, such as a 24 gigabyte M5 Pro, scale to 31 to 35 tokens per second. These figures serve as baseline references, as actual throughput fluctuates based on prompt length, context window size, and page-cache states. The developer has published 103 audited benchmark entries detailing kernel optimizations, caching strategies, and I/O improvements that contributed to the final memory footprint. The distribution package includes a native Mac application, a command-line interface, and an experimental loopback server compatible with the OpenAI Chat Completions API. The installation process streams approximately 15 gigabytes of model data from Hugging Face, repacking the checkpoint into a compact 14.3 gigabyte layout without materializing the full source model in memory. The runtime strictly enforces text-only generation with instruction-tuned chat formatting, supporting configurable sampling parameters, context limits, and tool-call declarations for external client authorization. Images, audio, and video modalities remain outside the current scope. TurboFieldfare is distributed as an independent research project under the Apache 2.0 license, with model weights governed by their original terms. The project explicitly maintains no affiliation with Google or the official Gemma development teams. Mikhaylov developed the runtime using Swift 6.2 and Metal 4 targeting macOS 14 and later versions on ARM64 architecture only. By decoupling expert routing from RAM capacity and leveraging native Mac GPU acceleration, the release establishes a new reference point for resource-constrained on-device language model inference, inviting further community benchmarking and kernel optimization.
