One-sentence Summary

UKP Lab at TU Darmstadt and University of Würzburg propose IRCoder, a family of code language models trained via continued causal language modeling on SLTrans, a nearly 4M parallel dataset pairing self-contained source code files with their compiler intermediate representations, to align IR constructs across programming languages for improved multilingual code generation, cross-lingual transfer, and prompt robustness.

Key Contributions

• Introduces SLTrans, a 26.2B-token parallel dataset containing nearly 4 million self-contained source code files and their compiler intermediate representations across 12 programming languages, extending source-to-IR alignment from short function snippets to full file-level programs.

• Develops IRCoder by continued pretraining of base Code-LMs ranging from 1.1B to 7.3B parameters on parallel source code and intermediate representation data, forcing the models to learn IR generation and align IR constructs with constructs in multiple programming languages without architectural changes or auxiliary objectives.

• Shows that IRCoder yields consistent gains in prompt robustness, multilingual code completion, code understanding, and instruction following, indicating that shared intermediate representations can improve multilingual code-model performance even with relatively small continued-pretraining datasets.

Introduction

The authors address a core gap in code language models: despite their potential to automate software development, benchmarking and performance have centered on high resource languages such as Python, while real-world codebases include many popular but under represented languages. This imbalance is significant because programming language popularity changes quickly, and some critical languages may be scarce or missing from pretraining corpora, making models less reliable across heterogeneous software ecosystems.

Prior work partly exposes these gaps through multilingual or transpilation oriented benchmarks, but cross language performance remains uneven, and prior research on programming language alignment is limited. Existing efforts using compiler artifacts often focus on narrow tasks, short contexts, or specific toolchains, leaving open whether shared code representations can broadly improve robustness and multilingual transfer without model architecture changes.

The authors contribute a language agnostic approach by using compiler intermediate representations as an interlingua for source code understanding. They introduce SLTrans, a large parallel source code and IR dataset, and create the IRCoder family of models through continued pretraining on mixed parallel and monolingual data. Their results show that IR grounding can improve prompt robustness, multilingual code completion, code understanding, and instruction following.

Dataset

• Dataset: SLTrans is a parallel source-code to LLVM IR translation dataset. The authors use it to learn and evaluate translation between programming-language source code and LLVM intermediate representation.

• Source composition: The data is sourced from accepted solutions to programming contest problems. The authors choose these because they are usually self-contained compilation units and avoid the dependency-tracking problems that make free-form GitHub code difficult to compile.

• Language coverage: The dataset covers a mixture of low-, medium-, and high-resource programming languages. In total, it includes about 4 million samples across 12 programming languages and totals 26.2 billion tokens.

• IR targets: Each source file is compiled into two LLVM IR variants: size-optimized IR, equivalent to the -Oz optimization level, and performance-optimized IR, equivalent to the -O3 optimization level. The size-optimized variant is described as more suitable for larger inference context windows and more uniform across languages, while the performance-optimized variant is more common in open-domain code corpora.

• Filtering rules: The authors keep only samples whose IR is shorter than 2500 code lines. This limits sequence length and makes the data easier to process with language models.

• Deduplication: Because programming contest solutions often contain near-duplicates, the authors apply MinHash-based deduplication to reduce redundancy before constructing the final dataset.

• Processing focus: The pipeline emphasizes complete, compilable, dependency-free source units and parallel IR extraction. This avoids file-level or repository-level dependency issues and allows the authors to collect full compilation units rather than only snippets or functions.

• Dataset usage: The authors use SLTrans as parallel translation pairs for source code to LLVM IR modeling. By collecting both size-optimized and performance-optimized IR, they can study and trade off different IR styles during language modeling.

Method

The method constructs a parallel source code and LLVM IR corpus by turning compilation into a structured data generation pipeline. The authors choose LLVM IR as the target representation because it is broadly supported by compiler frontends, has a stable and readable structure, and can be treated as a learnable sequence for language modeling. Rather than sampling arbitrary repository files, the pipeline focuses on self-contained programs that can be compiled reliably without extensive dependency resolution.

The first module of the pipeline collects complete compilation units from accepted programming contest solutions. These units are useful because they usually operate independently and avoid cross-file imports or external package dependencies, which are common obstacles when compiling code scraped from large software repositories. This selection step reduces the need for language-specific dependency tracking and enables whole-program compilation instead of relying only on isolated functions or snippets.

The second module compiles each selected source file into two variants of LLVM IR: a size-optimized variant corresponding to an optimization level similar to -Oz, and a performance-optimized variant corresponding to an optimization level similar to -O3. This dual compilation strategy captures two complementary views of the same program. The size-optimized IR is more compact and can be better suited for long-context modeling, while the performance-optimized IR is more representative of the IR commonly encountered in real-world compiler workflows.

The third module applies filtering to keep examples that are manageable for training and inference. IR outputs longer than 2500 code lines are discarded. This step prevents excessively long compilation results from dominating the dataset and helps maintain sequence lengths that are easier to process during model training and deployment.

The final stage removes near-duplicate programs using MinHash-based deduplication. Programming contest solutions often contain many structurally similar submissions, so this filtering step reduces redundant examples while preserving useful coverage of language constructs and coding patterns. Together, these steps form a repeatable workflow that converts executable source files into aligned source-to-IR training pairs through controlled compilation, optimization profiling, length filtering, and near-duplicate suppression.

Experiment

The authors continued pretraining several code language models on paired source code and LLVM IR, then compared them with models trained on source-only or unpaired IR data across code completion, prompt perturbation, multilingual understanding, and instruction-following tasks. The experiments show that aligning code across languages through shared IR is more beneficial than merely exposing models to IR, with gains extending to robust prompt handling, multilingual code generation and understanding, and instruction following, especially for low-resource languages. Overall, IR grounding appears to help models capture semantic and data-flow structure rather than superficial cues, leading to consistent improvements across evaluations.

The table shows that SLTrans pairs source code with LLVM IR across several programming languages using different compiler frontends. C++ and C provide the largest sample collections, while Rust, Haskell, and Python contribute smaller but still substantial sets. The relative length of generated IR varies strongly by language, with Rust producing the longest IR per source unit and C or C++ producing much shorter IR. C++ has the largest number of parallel samples, followed by C. Rust has the largest source to IR length multiplier while contributing far fewer samples than C++. Changing optimization level usually preserves sample counts, but Go shows the clearest reduction between levels.

The table compares three training mixes of equal total size while shifting their contents across code, text, unpaired IR, and paired source-IR data. The Paired mix is distinctive because it includes aligned source-IR examples and reduces other data components to accommodate them. The Unpaired mix tests exposure to IR without pairing, while the CodeText mix contains no IR data. Only the Paired mix includes source-IR pairs. The Unpaired mix adds unpaired IR but keeps no aligned source-IR examples. The CodeText mix relies solely on code and text sources with no IR component.

Across the tested base code models, adding paired source code and IR data gives the largest pass@1 gains on the multilingual code completion benchmark. Adding unpaired IR data also helps, but the improvements are smaller, while continuing to train only on additional source-code data provides little benefit and can slightly hurt some models. This suggests that aligning source code with IR is more useful than simply exposing the model to IR or more code. Paired source code and IR training improves performance over base and unpaired settings for all tested model variants. Unpaired IR training gives modest gains compared with base models, indicating exposure to IR alone is not enough. Source-code-only continued training performs similarly to or slightly worse than the base setting on several models.

The ReCode repair results compare average pass@1 across format, syntax, and function perturbations for IRCoder and its base models. The base model scores show that format perturbations are easiest to handle, syntax is harder, and function-level repair is most challenging. The reported comparison indicates that IR grounding improves IRCoder over corresponding base models. Format perturbations yield the highest pass@1, followed by syntax perturbations, while function perturbations are the most difficult. Larger base variants within the same model family tend to achieve stronger repair performance. IR-grounded IRCoder is evaluated against base models and shows benefits on ReCode repair tasks.

The table shows that IRCoder improves multilingual code performance over corresponding base Code-LMs across code completion, code-to-text, and instruction-following evaluations. The gains appear across model families and sizes, with the strongest improvements for larger or stronger base models. The results support the view that grounding source code in IR helps models generalize across programming languages and higher-level code concepts. IRCoder consistently improves over corresponding base Code-LMs in multilingual code generation and code understanding tasks. The largest performance gains are observed for stronger base Code-LMs. The improvements extend across multiple multilingual benchmarks, including code completion, code-to-text, and instruction following.

The experiments build SLTrans pairs across multiple languages and compare equally sized training mixtures to assess whether aligned source-IR data, unpaired IR exposure, or additional source code and text data most benefits code models. Results show that paired source-IR training consistently yields the largest qualitative gains on multilingual completion, repair, and understanding tasks, while unpaired IR provides smaller benefits and extra source-only training offers limited or negative effects. Overall, grounding source code in IR improves generalization and robustness, with especially strong benefits for larger or stronger base models.