HyperAIHyperAI

Command Palette

Search for a command to run...

18 hours ago
LLM
Text Generation

Colibri Runs 744B GLM-5.2 on 25GB RAM Consumer Hardware

Developer JustVugg has released colibri, a lightweight, dependency-free inference engine capable of running the 744-billion-parameter GLM-5.2 Mixture-of-Experts model on consumer-grade hardware. Built entirely in C, the engine operates without GPUs, Python, or external libraries, leveraging a twenty-five-gigabyte RAM budget and standard NVMe storage to deliver functional frontier-model inference on a personal laptop. The architecture addresses the primary memory bottleneck of large language models by streaming MoE experts directly from disk while retaining only approximately forty billion active parameters per token. Colibri implements MLA attention with compressed key-value caching, reducing memory overhead by fifty-seven times compared to standard implementations. The engine utilizes native multi-token prediction speculative decoding, which validates drafted tokens in a single forward pass to achieve a two-to-two-point-eight token yield per inference cycle. To maintain efficiency, the system employs an adaptive expert top-p routing mechanism and a learning cache that automatically pins frequently accessed experts in available RAM, progressively reducing disk read operations over time. Performance benchmarks confirm that the engine is strictly bound by storage input-output latency rather than computational throughput. On a baseline setup featuring a twelve-core CPU and a standard virtualized NVMe drive, cold inference yields approximately 0.05 to 0.1 tokens per second. Community testing indicates that upgrading to native PCIe storage and expanding RAM to thirty-two gigabytes can increase throughput to half a token per second, while unified memory architectures have successfully pushed performance to one token per second. The developers caution that SSD endurance remains a consideration under heavy workloads, recommending conservative RAM allocation to prevent swap traffic. Quantization is handled through integer-dot kernels supporting int8 and packed int4 formats, with the default configuration tuned to minimize sampling artifacts. A companion offline converter allows users to transform 756-gigabyte FP8 checkpoints into optimized int4 containers without requiring full model residency in memory. The project currently lacks formal accuracy benchmarks for its quantized weights, and the developer community has been invited to test the integrated evaluation harness on higher-end hardware to validate inference quality. Colibri operates under the Apache 2.0 license, with GLM-5.2 weights released by Z.ai under MIT. The repository is available on GitHub, with pre-converted model weights hosted on Hugging Face. The project demonstrates that frontier-class language models can be operationalized on modest consumer hardware through careful system design, though practical deployment will continue to depend heavily on storage speed and RAM capacity.

Related Links