Vocabulary Specialization Cuts Small Language Model Inference Cost by 77%

Researchers at Dharma AI previewed a method called VocabX that rewrites a specialized language model's tokenizer around measured serving cost rather than token frequency, reducing inference cost by 77% in a production Portuguese essay-grading system without changing the base model or training recipe. The system is based on Qwen3-4B and grades essays in Portuguese, returning structured JSON assessments. Although the model was already fine-tuned for the task, it retained the original Qwen3 tokenizer, designed for many languages and text types. This created two inefficiencies: repetitive domain output such as JSON envelopes and evaluation phrases was split into many tokens, and the model continued scoring 151,669 vocabulary entries, many of which the task never used. The briefing notes that for small models running under memory-bound decode, generating tokens is much more expensive than reading them. In the measured grader, one generated token cost 7.4 times one input token. The output layer also became costly because the language-model head evaluates every vocabulary row for each generated token. In Qwen3-4B, that matrix accounts for almost 10% of parameters. VocabX selects tokens by the actual cost saved in system prompt, user message or response fields. It rebuilds sub-word segmentation and adds macro tokens for repeated spans such as phrases, lines, sentences and JSON structures, using five specialized miners. To avoid losing rare but useful candidates when evaluating on samples, it estimates their corpus-wide benefit instead of assigning them zero gain. In the grader, average model responses fell from 350 to 101 tokens. Measured inference cost dropped 77.0%, while grading quality, measured by a benchmark combining accuracy and rule-breaking detection, improved from 0.703 to 0.810. The team said the first vocabulary version overly compressed grade decisions into macro tokens, hurting quality; giving grades their own tokens preserved more flexibility and was cheaper than expected. Testing across contracts, medical papers, scientific papers, news and Python code, using Qwen3, Llama-3.1 and Gemma-3, VocabX compressed more than fast vocabulary transfer, multi-word tokenization and in-place BPE methods in all fifteen combinations. Estimated inference-cost reductions also led all comparisons, including margins of 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. The findings suggest that domain specialization can begin with the tokenizer, making small language models cheaper to serve by cutting repeated output fragmentation and scoring only the vocabulary a task needs.