One-sentence Summary

Researchers from the University of Waterloo, Cornell University, and Harvard University introduce fuzzy-function programming with Program-as-Weights (PAW), where a $4\mathrm{B}$4B compiler trained on the $10\mathrm{M}$10M-example FuzzyBench dataset compiles natural-language specifications into parameter-efficient adapters for a frozen $0.6\mathrm{B}$0.6B Qwen3 interpreter, matching the performance of directly prompting Qwen3-32B while using $\frac{1}{50}$501​ of the inference memory and running at 30 tokens per second on a MacBook M3, reframing foundation models from per-input solvers into tool builders.

Key Contributions

• Program-as-Weights (PAW) compiles a natural-language specification into parameter-efficient adapter weights for a frozen 0.6B Qwen3 interpreter, converting the foundation model from a per-input problem solver into a one-time tool builder.
• The FuzzyBench-10M dataset, generated via 29 task-class-specific pipelines, provides 10 million fuzzy-task examples with a verified test split filtered by agreement between two strong LLMs.
• A PAW adapter on the 0.6B interpreter matches the prompting performance of Qwen3-32B on FuzzyBench while using roughly 50× less inference memory and running at 30 tokens per second on a MacBook M3.

Introduction

Many real-world programming tasks involve “fuzzy functions” that resist precise symbolic specification, such as filtering critical log lines, repairing malformed JSON, or writing regexes that handle edge cases. Developers commonly outsource these tasks to large language model APIs, but that approach is expensive, fragile (providers may silently update models), and prevents software from being self-contained. Handwritten rules and regexes are brittle in the face of noisy, drifting inputs. The authors propose Program-as-Weights (PAW), a paradigm in which a neural compiler transforms a natural-language task description into a compact LoRA adapter, and a frozen lightweight interpreter runs that adapter locally on the user’s device. By shifting the heavy computation to a one-time compilation step, PAW enables a small on-device model to match or exceed the accuracy of a much larger prompted model while keeping software reproducible and self-contained.

Dataset

The authors construct FuzzyBench, a 10 million-example dataset of triples (specification, input, target output) for the task of compiling fuzzy functions. Here are the key details:

• Source and generation: All triples are generated by gpt‑5.2 in a two-stage pipeline. First, the model produces natural‑language specifications under diverse category constraints (covering breadth of real‑world fuzzy tasks). Second, for each specification it generates input/output pairs. The dataset is built incrementally across 29 versions, each adding 100 K–500 K examples from a new task family.
• Scale and thematic coverage: The final 10 M examples span 7 high‑level task families (text processing/parsing, agentic tool use, web intelligence, code/command generation, safety/verification, etc.) and over 800 fine‑grained sub‑categories.
• Train/validation/test split: Data is split 80/10/10 by specification so that test specifications never appear during training. A verified test set is further curated by keeping only examples where gpt‑5.2 and an independent strong model (gpt‑5‑mini) agree on the output, eliminating ambiguous targets.
• Noise‑augmented test sets: For robustness evaluation, the test set is perturbed along eight axes (typos, grammar errors, ambiguity, formatting drift, “all noise,” terse phrasing, casual phrasing, and paraphrase), each at three intensity levels (light, medium, heavy).
• How the data is used: The dataset serves as the primary training resource for PAW‑style methods that compile fuzzy functions from specifications. Training uses the train split, validation uses the val split, and final performance is reported on the main test set, the verified test set, and the noise‑variant test sets.
• Empirical ceiling: The generating model gpt‑5.2 achieves 96.09% on the test set, and gpt‑5‑mini reaches 91.87%, providing an upper bound for any model trained on FuzzyBench.

Method

The authors propose a "Programs as Weights" (PAW) framework, which compiles a user specification into a neural program that specializes a fixed model to implement a target fuzzy function. Instead of repeatedly querying a large language model, the system divides the workload into a compiler and an interpreter. Formally, a neural compiler maps a user specification $s$s to a program $p$p, and a small fixed neural interpreter executes $p$p on inputs $x$x to produce outputs $\hat{y}$y^​:

$p = \operatorname{Compiler}(s), \quad \hat{y} = \operatorname{Interpreter}(p, x) \approx f(x).$p=Compiler(s),y^​=Interpreter(p,x)≈f(x).

The compiled program $p$p is a hybrid of a discrete and a continuous component, $p = (p_{\text{discrete}}, p_{\text{continuous}})$p=(pdiscrete​,pcontinuous​). The discrete component acts as a self-contained pseudo-program presented to the interpreter as part of its input, while the continuous component is implemented using a parameter-efficient fine-tuning method, such as LoRA.

The PAW pipeline comprises three main components. First, a pseudo compiler reads the specification $s$s and produces the discrete pseudo-program $p_{\text{discrete}}$pdiscrete​. The authors utilize an off-the-shelf model for this step, prompting it to generate a clean restatement of the task alongside representative input-output examples. Second, a PEFT compiler reads the specification together with $p_{\text{discrete}}$pdiscrete​ and emits a small parameter-efficient module $p_{\text{continuous}}$pcontinuous​. Finally, the frozen interpreter ingests $p_{\text{continuous}}$pcontinuous​ at runtime by attaching it to the appropriate target modules and running the user input through it.

The authors' current best instantiation of this pipeline is Text-to-LoRA. The LoRA compiler is a trained model that processes the concatenation of the specification, the pseudo-program, an end-of-sequence token, and a fixed sequence of learned prefix tokens. It extracts prefix-position hidden states from layers spaced uniformly by depth ratio, stacking them into a tensor that is fed to the LoRA mapper.

As shown in the figure below:

The LoRA mapper converts the compiler's hidden states into per-example LoRA weights. For each target module type in the interpreter, the mapper maintains shared learnable bases. The hidden states are mean-pooled over the depth-aligned layers and prefix positions, and passed through a linear head to produce mixing coefficients. The final LoRA matrices for a specific layer and module are computed as linear combinations of these shared bases:

$A_{l, m}^{\text{ex}} = \sum_{n=1}^{N} \alpha_{l, m, n}^{A} A_{n}^{(m)}, \quad B_{l, m}^{\text{ex}} = \sum_{n=1}^{N} \alpha_{l, m, n}^{B} B_{n}^{(m)}.$Al,mex​=∑n=1N​αl,m,nA​An(m)​,Bl,mex​=∑n=1N​αl,m,nB​Bn(m)​.

During execution, the interpreter attaches the generated LoRA weights to its target modules, prepends the discrete pseudo-program to the user input, and generates the output autoregressively. Because the interpreter remains frozen and the LoRA adapters are hot-swappable, a single device-resident interpreter can serve an unbounded number of compiled programs.

Regarding the training process, only the PEFT compiler and the LoRA mapper are trained, while the pseudo compiler and the interpreter remain frozen. For each training triple, the system looks up a pre-generated pseudo-program, runs a forward pass through the LoRA compiler to obtain prefix-position hidden states, and passes them through the LoRA mapper to generate the adapter weights. The loss is the negative mean-token log-likelihood of the target output under the frozen interpreter:

$\mathcal{L}(\theta) = \mathbb{E}_{(s, x, y)} \left[ - \log P_{\phi}(y \mid p_{\text{discrete}}, p_{\text{LoRA}}(\theta; s, p_{\text{discrete}}), x) \right],$L(θ)=E(s,x,y)​[−logPϕ​(y∣pdiscrete​,pLoRA​(θ;s,pdiscrete​),x)],

where $\theta$θ represents the parameters of the LoRA compiler and mapper, and $\phi$ϕ denotes the frozen interpreter parameters. Gradients flow back through the frozen interpreter into the LoRA mapper and subsequently into the compiler's hidden states.

Experiment

The experiments evaluate the PAW framework, where a large compiler model generates pseudo-programs and LoRA adapters for a small fixed interpreter, across fuzzy text tasks, multimodal conditioning, and realistic deployment scenarios. A 0.6B interpreter with compiler-generated LoRA matches or exceeds far larger prompted models while using much less memory, and the compiler's generated pseudo-program effectively denoises noisy specifications, making the system robust. Ablations confirm that the gain comes specifically from the compiler, and the simplest mapper design works best. Local execution with quantization maintains accuracy and enables economical, on-device deployment without per-request LLM calls.

On FuzzyBench, both prefix-tuning and LoRA compiler–interpreter instantiations dramatically outperform a prompting baseline. Prefix-tuning reaches 50.4% exact match, while LoRA at rank 18 (matching prefix-tuning’s program size) reaches 56.5% and at rank 64 achieves 65.7%, making LoRA the stronger method. Prefix-tuning delivers 50.4% accuracy, far above the no-compiler prompting baseline of 9.8%. LoRA at rank 18 improves to 56.5%, and at rank 64 reaches 65.7%, outperforming prefix-tuning at all tested sizes.

A 0.6B interpreter executing PAW-compiled programs achieves 73.78% exact match on FuzzyBench, surpassing a directly prompted 32B model (68.70%) with roughly 50x less inference memory. Even a 124M GPT-2 interpreter without instruction tuning reaches 54%, demonstrating that the compiler-generated LoRA injects useful task knowledge into very small models. PAW programs are self-contained and shipped as lightweight quantized adapters, enabling local execution without internet access. PAW with a 0.6B interpreter outperforms Qwen3-32B prompting on FuzzyBench (73.78% vs 68.70%) while using ~50× less memory. A 124M GPT-2 interpreter, despite lacking instruction tuning, attains 54% on FuzzyBench, showing that the compiler’s LoRA adapter can adapt even tiny base models. Among the tested interpreters, Qwen3 0.6B is the strongest, slightly ahead of the hybrid Qwen3.5 0.8B.

PAW with small interpreters (0.6B–0.8B) outperforms dedicated vision-language models up to 4B on most image-conditioned fuzzy tasks, while using far fewer parameters. The LoRA adaptation is consistently stronger than prefix-tuning, and the 0.8B interpreter offers modest gains over the 0.6B. The exception is Im2LaTeX, where the specialized models still lead. PAW’s tiny interpreters surpass the much larger Qwen3-VL models on Circuit, Chemical, Music, and Im2SMILES. LoRA achieves better accuracy than prefix-tuning on every task, with the gap most pronounced on TextVQA. On Im2LaTeX, dedicated vision models outperform PAW, reversing the trend seen on other benchmarks. Scaling the interpreter from 0.6B to 0.8B yields consistent but slight improvements across all six tasks. AndesVL scores zero on Im2SMILES, while PAW LoRA reaches a usable accuracy, highlighting the compiler’s ability to handle niche image queries.

Simpler LoRA mapper designs outperformed more expressive ones. The default—mean-pooling prefix tokens, shared bases, and a single residual MLP—achieved the highest accuracy. Adding per-position aggregation, per-layer bases, or a parallel prefix-tuning pathway all reduced performance, with the combination of per-position and per-layer bases causing the largest drop. The default mapper was the most accurate variant. Per-position aggregation lowered accuracy, and adding per-layer bases hurt further. A parallel prefix-tuning pathway or per-layer bases alone matched each other but still underperformed the default.

Removing the compiler and training the same 0.6B base model with full fine-tuning or fixed LoRAs leads to large accuracy drops. PAW with its compiler-generated LoRA reaches 73.8% accuracy, surpassing full fine-tuning by 15.4 points and the strongest fixed-rank LoRA by 21.7 points. PAW’s compiler-generated LoRA outperforms full fine-tuning of the same base model by 15.4 percentage points. The best fixed LoRA (r=64) only achieves 52.1% accuracy compared to PAW’s 73.8%.

Evaluated on FuzzyBench, the PAW compiler-interpreter framework with LoRA adapters significantly outperforms prompting baselines, full fine-tuning, and fixed-rank LoRA, enabling tiny 0.6B interpreters to surpass much larger models while using substantially less memory. LoRA consistently outperforms prefix-tuning across text and image-conditioned fuzzy tasks, often exceeding dedicated vision-language models on niche domains, and the resulting programs are self-contained lightweight adapters. Ablations confirm that simple mapper designs are most effective and that the compiler is essential for injecting reusable task knowledge into base models.