One-sentence Summary

Researchers from the University of Waterloo and Harvard University introduce compile by training, a method that converts natural-language specifications into reusable neural functions by training a small adapter on teacher-generated examples, achieving $83.6\%$83.6% semantic accuracy on FuzzyBench-Hard and enabling offline composable functions for website helpers and language-controlled avatars.

Key Contributions

• Compile by training compiles a natural-language function specification into a reusable neural program by using teacher models to synthesize task-specific examples and fine-tuning a LoRA adapter on a compact frozen interpreter.
• On FuzzyBench-Hard, where the prior fast compiler produced no exact matches, compile by training reaches 83.6% semantic accuracy at a compile cost of roughly a minute.
• The compiler is deployed in a public interactive service, and compiled functions are demonstrated in a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English-Claudish translator.

Introduction

Many real-world text-processing tasks, like triaging emails or filtering messages, sit in a difficult middle ground: they are too fuzzy for traditional rules but too narrow and frequent to justify calling a large remote model on every input. The authors build on Program-as-Weights (PAW), which introduced an amortized compiler that predicts a compact neural function in a single forward pass. While fast, that approach spends a fixed amount of computation per function and often fails on harder specifications. The key contribution is compile by training, which treats adaptation as a build step. A developer writes a natural-language specification, teacher models generate example behaviors, and gradient descent fine-tunes a LoRA adapter for a shared interpreter, starting from the amortized prediction. This minute-scale compilation yields substantially higher accuracy while preserving the same lightweight runtime interface, enabling neural functions to be versioned, cached, and composed like ordinary software.

Dataset

The authors synthesize a task-specific dataset $D_s$Ds​ from a natural-language specification $s$s using one or more teacher models. The dataset consists of input–output pairs $\{(x_i, y_i)\}${(xi​,yi​)} that illustrate the desired mapping, and it is generated on demand for each specification.

• Composition and sources

  • Each example is an $(x, y)$(x,y) pair produced by a teacher model that interprets the specification.
  • The teachers operate through a structured JSON request/response format, enabling automated ingestion into the training pipeline.
  • A public service combines a lower-cost teacher (which supplies the majority of examples) with a larger teacher that contributes complementary supervision.

• Subset details and filtering

  • No fixed subsets are defined; the dataset is created per specification.
  • A compiler validates every teacher response and rejects malformed or incomplete batches before the examples are accepted for training.
  • The paper does not report a predetermined dataset size; the number of examples depends on the specification and the teacher outputs.

• Usage in the model

  • The accepted examples flow directly into the training pipeline to “compile” the specification into a model.
  • The entire validated set $D_s$Ds​ is used for training; no explicit training/validation split or mixture ratios are mentioned.

• Processing and metadata

  • No cropping or complex preprocessing is applied. The data is the raw input–output text pairs.
  • Metadata is minimal, consisting only of the validated JSON structure that pairs each input with its expected output.

Method

The authors propose a system that separates the construction of a neural function from its execution. A developer describes the desired behavior in natural language, which is then compiled into a reusable program. This program can be invoked repeatedly on new inputs without requiring further calls to teacher models. Formally, the system exposes the interface:

where $s$s is a natural-language function specification, $p_s$ps​ is the compiled program, $x$x is a new input, and $\hat{y}$y^​ is the output. A shared frozen language model acts as the interpreter, while each compiled program provides the adapter and prompt to specialize it.

The user workflow consists of three stages: specifying a text-to-text function in natural language, submitting a build job to monitor training progress, and invoking the completed program on new inputs via an SDK. The compiled artifact packages the specification and the components needed to specialize the shared interpreter, allowing it to be stored, versioned, and reused.

The compilation process transforms a natural-language specification into a reusable neural program through two main stages. First, teacher models synthesize examples of the desired function. Since a specification alone lacks sufficient labeled examples, the system uses one or more teacher models to generate a task-specific dataset:

where each pair illustrates the mapping from input $x_i$xi​ to output $y_i$yi​. Teacher requests utilize a structured JSON format, and the compiler validates each response to reject malformed batches before training.

To avoid the prohibitive cost of training a separate full model for every specification, all programs share a frozen Qwen3-0.6B interpreter. Each function is represented by a lightweight LoRA adapter and a run-time scaffold, which is a compiler-generated prompt template encoding the specification as structured instructions. The compiler provides initial adapter parameters $\theta_s^{(0)}$θs(0)​ and scaffold $r_s$rs​, which are then refined using the synthesized dataset by minimizing:

Following optimization, the compiler packages the adapter $\theta_s$θs​, scaffold $r_s$rs​, original specification, and interpreter metadata into the program $p_s$ps​.

To make the compilation process usable interactively, the system shortens the critical path and allows users to continue working while compilation proceeds. In a sequential pipeline, the compiler would wait for all teacher-generated examples before starting optimization, leaving the GPU idle. Instead, the deployed streaming compile path initiates teacher requests, model loading, and training concurrently. Training begins as soon as the first batch of examples is available, blocking only if it catches up with synthesis.

The service architecture separates job coordination from the compilation workers. The API maintains a persistent record of each job, while a shared queue dispatches pending jobs to available GPU workers. Workers check a cache to reuse matching teacher outputs before requesting new examples. Completed programs are written to a shared artifact store and linked back to the persistent job record.

The authors demonstrate the system capabilities by building a live bidirectional translation service between plain English and Claudish, a distinctive prose style. They write a natural-language specification for each direction. For English-to-Claudish, the specification instructs the program to adopt characteristic vocabulary, compounds, and rhetorical patterns. For the reverse direction, it asks the program to rewrite the input in direct English, removing redundant contrasts. Compile by training synthesizes examples from these specifications and finetunes one adapter per direction for the shared interpreter.

The resulting programs power the live service, allowing users to translate text back and forth. Both programs can be downloaded and run locally.

Experiment

The evaluation uses a semantic exact-match metric (LEM) on FuzzyBench-Hard, where exact-match compilers fail. Training the compiler substantially improves correctness, though at the cost of longer compilation that still supports interactive use. Supervision quality and data scaling further boost accuracy, while deployed applications show that compiled functions can be composed for fuzzy decision-making in tasks like website assistance, motion generation, and stylistic translation.

Mixing a stronger teacher model (GPT-5.5) with a weaker one (GPT-5.4-mini) in a 2:1 ratio substantially raises mean LEM compared to using the weaker model alone. In a data-scaling sweep, increasing unique training pairs from 1440 to 7200 yields a modest overall improvement, with a plateau where 2400 and 3600 pairs produce identical mean LEM. A 2:1 mixture of GPT-5.4-mini and GPT-5.5 supervision improves mean LEM by 0.105 over using GPT-5.4-mini alone. Scaling unique training pairs from 1440 to 7200 increases mean LEM by 0.045, but no gain is observed between 2400 and 3600 pairs.

The experiments evaluate how mixing supervision from a stronger teacher model with a weaker one and scaling training data affect mean LEM. A 2:1 mixture of stronger and weaker teachers substantially boosts performance compared to using the weaker model alone. Increasing the number of unique training pairs provides only a modest overall gain and plateaus, with no further improvement beyond a moderate dataset size.