One-sentence Summary

Cohere For AI, Brown University, and Carnegie Mellon University et al. introduce Aya, an open-access instruction-tuned multilingual language model covering 101 languages that outperforms mT0 and BLOOMZ with double the language coverage, and present new evaluation suites across 99 languages alongside detailed analyses of finetuning mixture composition, data pruning, and safety.

Key Contributions

• The paper introduces Aya, an open-source instruction-finetuned multilingual language model that follows instructions in 101 languages, including 51 lower-resourced languages, and is trained on a 203M-example mixture with 21.5% English, compared with 46 languages, 81M examples, and 39% English in xP3.

• The paper develops a multilingual evaluation suite spanning 99 languages across discriminative tasks, generative tasks, LLM-as-judge simulated win rates, human evaluations, and safety tests, where Aya achieves 13.1% and 11.7% relative gains over mT0x on discriminative and generative tasks and 77% human preference win rates across seven languages.

• The paper studies data weighting and pruning to rebalance permissively licensed academic-style multilingual data, removing 19.66% of English instances and 18.25% of multilingual instances based on human annotations and varying weights for translated, templated, and human-annotated data. It also applies multilingual safety context distillation, which reduces harmful generations for adversarial prompts by 78 to 89% as judged by human experts, and evaluates toxicity, social bias, and gender bias across 18 languages.

Introduction

The authors focus on multilingual instruction tuning for large language models, which helps systems respond usefully to user requests written in different languages and is central to making AI assistants more accessible worldwide. Prior open-source instruction-following models remain heavily English-centric; 73% of popular instruction-tuning datasets are primarily English, and existing multilingual approaches either lack broad language coverage or still expect speakers to use English prompts. The authors contribute Aya, an open-source instruction-finetuned model that expands training and evaluation to 101 languages, uses a more balanced 203M-example mix with only 21.5% English, broadens multilingual evaluation to 99 languages, and adds safety-oriented pruning and distillation.

Dataset

• The authors construct a multilingual instruction finetuning mixture by turning existing datasets, human annotations, translations, and synthetic generations into instruction and response pairs, focusing primarily on the 101 languages used for mT5 pretraining.

• xP3x is an extended version of xP3, expanding from 86M examples across 46 languages and 13 tasks to 680M examples across 277 languages and 16 tasks. For training, the authors use a subset restricted to the 101 mT5 languages and prune it to 168M examples across 101 languages and 56 datasets to improve quality and generation length.

• The Data Provenance Collection uses filtering tools to select publicly available supervised instruction datasets with self-reported commercially permissive licenses. It includes OctoPack's cleaned version of Open Assistant, Open Instruction Generalist, a subset of the Flan Collection, and Tasksource Instruct. After filtering, it contains 1.6M examples, of which 550K are few-shot and the rest are zero-shot, covering 14 languages and 161 datasets. The authors remove datasets derived from evaluation benchmarks or evaluation task categories such as textual entailment, co-reference resolution, and sentence comparison, and exclude code data because the base mT5 model was not pretrained on code.

• The templated portion of the Aya Collection is drawn from a larger collection of 513M instances and templates in 74 languages. For training, the authors filter it to the 101 target languages and remove evaluation contamination, leaving 18.9M samples across 51 languages, split as 21 high-resource, 11 medium-resource, and 19 low-resource languages, and spanning 34 datasets.

• The Aya Dataset provides human-curated open-ended prompt and response pairs written by native speakers. The full set contains 204K examples in 65 languages. After restricting to the target languages, the authors use 199.5K examples covering 64 languages, with 22 high-resource, 12 medium-resource, and 30 low-resource languages.

• Translation is used as augmentation to broaden language coverage and improve crosslingual generalization. The authors use the Aya collection's NLLB translations of widely used English instruction finetuning datasets into 101 languages. Translation priority favors datasets with rich task diversity and longer completions, and only languages overlapping with the 101 mT5 training languages are retained.

• ShareGPT-Custom is a synthetic, machine-translated dataset of 6.8M examples in 93 languages. It combines human-annotated prompts from ShareGPT with synthetic English completions generated by in-house LLMs. The authors do not use the original ShareGPT completions, because those come from user-shared ChatGPT conversations and are excluded to comply with usage terms. Prompts are filtered before generation and translation.

• Across all sources, the combined pool contains over 203M instances, but the volume is heavily skewed. Human annotations are especially small, at 199.5K samples, or about 0.7% of the full training budget.

• The authors sample from the combined pool under a fixed budget of 25M training instances. They assign sampling weights to major data sources to balance task and language coverage, then distribute each source weight proportionally across datasets within that source. Ablation settings include human annotation heavy, which upweights the Aya Dataset; translation heavy, which upweights Aya translations and ShareGPT-Custom; and template heavy, which upweights Aya Collection templates, xP3x, and the Data Provenance Collection.

Method

The authors build the method around a multilingual instruction tuning pipeline that starts from human curated prompts and expands them into large scale, machine translated training data. They first take annotated prompts from ShareGPT and remove the original synthetic completions. English completions are then generated for these prompts using in house language models. The resulting prompt completion pairs are machine translated into many languages, producing a synthetic data set called ShareGPT Custom with 6.8M examples across 93 languages. Filtering is applied both to the original prompts and to the translated outputs, so the training mixture is derived from controlled data generation and translation rather than directly reused user conversations.

The tuned model pipeline centers on mT5, a 13 billion parameter encoder-decoder transformer that was pretrained on one trillion tokens of natural language text spanning 101 languages. The authors choose this model because it provides unusually broad language coverage, which is important for instruction fine-tuning when embeddings are adapted to many target languages. Although newer models may be stronger in other respects, the objective here is to reduce the risk of poor adaptation to languages absent from earlier pretraining stages.

During fine-tuning, the mT5 model is trained with the Adafactor optimizer. The learning rate is set to $3 \times 10^{-4}$3×10−4, and the batch size is 256 with sample packing enabled. Training runs for 30,000 steps, which corresponds to a total budget of approximately 25M training samples. In parallel, the authors also use Okapi style models, where pretrained BLOOM-7B and LLaMA-7B models are fine-tuned on a mixture of translated prompts and synthetic data to produce language-specific variants.

Experiment

The experiments fine-tune a large multilingual mT5 model on a broad instruction mixture, ablate sampling weights for human-annotated, translated, and templated data, and compare variants against multilingual baselines across unseen discriminative tasks, general knowledge evaluation, generative tasks, preference studies, and safety tests. The discriminative findings show that a well-balanced instruction mixture can preserve strong zero-shot performance despite broader language coverage, while template-heavy weighting is most effective for benchmark-style tasks and appears to benefit from cross-lingual transfer from English. Generative and preference results reveal that translated-heavy weighting produces better open-ended responses and translation quality, indicating that benchmark accuracy does not necessarily translate into preferred conversational behavior. Safety analyses show mixed harms, with the model generating less toxic but still biased and adversarially vulnerable outputs, which safety distillation reduces substantially at a modest performance cost.

The training mixture combines many multilingual instruction sources that differ widely in size, language coverage, resource representation, and sequence length. Because human-annotated data is very small compared with templated, translated, and synthetic sources, sampling weights strongly shape which task strengths emerge during finetuning. Balancing instruction following requires prioritizing sources so no single language or data type dominates the mixture. Templated data supports discriminative tasks, while translated data supports machine translation and more natural open-ended outputs. Overweighting the small human-annotated dataset can reduce average performance because repeated exposure risks overfitting.

The variants compare allocating a fixed finetuning budget across human annotated, templated, and translated sources. Translated heavy weighting is favored because it gives stronger generative and machine translation behavior, while templated heavy weighting helps discriminative tasks through stronger English transfer. Human annotated heavy weighting performs less well overall, likely because repeated exposure to a small dataset can cause overfitting. Upweighting translated data supports open ended and translation oriented performance. Upweighting templated data favors discriminative tasks despite a smaller English share in the translated variant. Upweighting human annotated data gives the lowest average performance among the schemes.

The evaluation setup covers unseen discriminative tasks, unseen general language understanding, and generative tasks with held-out evaluation splits. Discriminative benchmarks are measured by accuracy across multiple languages and task categories, while generative tasks include translation, summarization, and question answering. Open-ended generation is evaluated using human and simulated preference-based win rates. Discriminative evaluation includes unseen coreference, natural language inference, sentence completion, and story cloze tasks. Multilingual MMLU is used as an unseen general language understanding benchmark. Generative evaluation uses held-out data for translation, summarization, and question answering. Open-ended outputs are compared with human and LLM-simulated win rates. Translation and open-ended generation favor translated data emphasis, while discriminative tasks favor templated data emphasis.

Aya obtains the strongest average held out accuracy on unseen multilingual discriminative tasks while covering substantially more languages than the compared baselines. The results indicate that a broad and balanced instruction finetuning mixture can preserve performance as language coverage expands. Aya is the best overall model on the held out discriminative tasks, with gains on XCOPA, XNLI, and XStoryCloze relative to the listed baselines. The 101 language mT0x setup falls behind the 46 language mT0 baseline, suggesting that data diversity and balanced sampling matter more than language count alone.

Across the multilingual MMLU scores, Aya shows the strongest overall average performance and is highest in the high-resource and mid-resource categories. MT0 and MT0X remain close to one another and outperform the reported Okapi baseline on average. Okapi achieves the best low-resource score, but its low-resource strength does not carry over to a competitive overall result. Aya leads the average multilingual MMLU score Okapi is strongest in low-resource performance but weakest on average MT0 and MT0X have similar results, with MT0 slightly ahead overall

The experiments evaluate how different training data mixtures affect multilingual instruction following across unseen discriminative tasks, general language understanding, and generative tasks. They show that translated data favors translation and open-ended generation, templated data favors discriminative tasks and English transfer, while excessive weighting of small human annotated data can overfit and reduce average performance. The comparisons also indicate that broad and balanced coverage is more important than language count alone, with the strongest overall held out discriminative and multilingual MMLU performance obtained by the more balanced setup.