One-sentence Summary

Researchers from the University of Zurich and collaborating institutions introduce PARME, the first parallel corpora for eight severely underresourced Middle Eastern language varieties, built via community-driven efforts with over $36{,}000$36,000 translated sentences, and show through zero-shot and fine-tuning experiments that existing machine translation technologies fail these languages, exposing critical gaps for improvement.

Key Contributions

• PARME introduces the first parallel corpora for eight severely underresourced Middle Eastern language varieties, built through a community-driven effort with over 40 native speaker volunteers and comprising 36,384 trilingual translation pairs, making publicly available datasets for languages that previously lacked them.
• The evaluation uses zero-shot approaches and fine-tuning experiments with pretrained models such as NLLB across various multilingual configurations, providing a systematic analysis of translation performance and limitations for these low-resource languages.
• Baseline X-English BLEU scores fall below 1 for multiple languages, while the best fine-tuned model achieves a mean X-English BLEU score of 7.50, with Hawrami reaching the highest performance at 16.54, demonstrating measurable improvements and exposing critical gaps in existing language technology for the region.

Introduction

The Middle East hosts a rich linguistic landscape across over 400 million people, yet many of its languages remain severely under-represented in digital tools and resources. UNESCO identifies 60 varieties in the region as endangered, with many facing political marginalization, competing orthographies, limited educational materials, and declining intergenerational transmission. Prior NLP work for these languages is nascent, since most have predominantly oral traditions, no standardized writing systems, and few parallel corpora, which are essential for modern machine translation (MT) systems. To address this gap, the authors focus on eight under-represented languages, including Luri Bakhtiari, Gilaki, Hawrami, Laki Kurdish, Mazandarani, Southern Kurdish, Talysh, and Zazaki, and implement a community-driven initiative to build parallel corpora for evaluating and creating MT systems. They carry out experimental evaluations of pretrained models in both zero-shot and fine-tuning scenarios, showing that substantial work remains to develop effective translation tools for these varieties and opening new research avenues for low-resource languages more broadly.

Dataset

The authors introduce PARME, a new parallel corpus designed to address the data scarcity for low-resource Middle Eastern languages in machine translation (MT). The dataset's construction was driven by both technical needs and practical constraints, such as the lack of standardized language varieties and reliance on volunteer contributors.

Dataset Composition and Sources

• PARME is a parallel corpus containing 36,384 translation pairs for eight under-resourced languages: Luri Bakhtiari (BQI), Gilaki (GLK), Hawrami (HAC), Laki (LKI), Mazandarani (MZN), Southern Kurdish (SDH), Talysh (TLY), and Zazaki (ZZA).
• The source sentences were selected from the Mizan Farsi-English bilingual corpus. Additional reference translations were also provided, with English and Persian as the primary reference languages, except for Zazaki, which uses English and Northern Kurdish.

Translation Initiative

• The corpus was created with the help of 45 volunteer translators who are native speakers of the target languages, recruited through social media and personal networks.
• Contributors were given source sentences in English and a second language they were fluent in, such as Persian or Turkish. Translation work was primarily done using spreadsheets.
• To address resource limitations, volunteers who made substantial contributions were offered co-authorship on the paper.

Data Selection and Processing

• The authors used a specific algorithm to select sentences from the source corpus to maximize lexical diversity, sentence length variety, and semantic richness.
• The selection process involved cleaning data to remove sentences with named entities, ellipses, code-switching, URLs, and other less informative cases.
• A score was calculated for each candidate sentence pair based on the ratio of its edit distance to its semantic similarity with previously selected sentences. This score was designed to reward novel sentences with higher edit distances while penalizing those that are semantically too similar to the existing selection.
• The selection process was iterative, using batches of 3,000 sentences, and was repeated until a final corpus size of 15,000 sentences was reached for the source language. The similarity was computed using Sentence-BERT embeddings.

Corpus Statistics and Quality Control

• The final corpus has 36,384 sentence pairs across all languages. In terms of size, Luri Bakhtiari (BQI) has the fewest sentences, and Southern Kurdish (SDH) has the most.
• Quality control measures included allowing translators to skip problematic sentences, marking empty translations for exclusion, and requiring consistent orthography. The final translations were also checked by at least two other translators.

Evaluation Set Construction

• For testing, a distinct evaluation set of approximately 1,000 instances per language was created. The selection criteria for the test set were designed to:
  • Avoid data contamination, ensuring sentences translated into multiple dialects are in the same split.
  • Ensure consistent orthography.
  • Guarantee a balanced dialect distribution for languages without a standard form.
  • Prioritize sentences that were translated into the highest number of languages in PARME.

Data Usage in the Model

• The paper explores two experimental setups for training:
  • A base setup using official train sets with English translations. For languages lacking dedicated training data (BQI and TLY), their validation sets were repurposed for training.
  • An augmented setup that expands the training data by using NLLB to translate the additional reference sentences (from Northern Kurdish or Farsi) into English.
• For training, all parallel sentences are merged and converted into JSON Lines format, with each entry containing the source sentence and its English translation. The NLLB model is fine-tuned on this data with English (eng_Latn) set as the target language, while the source language token is set dynamically based on the data.

Method

The authors address the scarcity of parallel data for low-resourced Middle Eastern languages by constructing PARME, a parallel corpus, and by adapting a pretrained multilingual translation model to these languages through a transfer learning pipeline. The overall approach combines a carefully controlled data collection effort with a token-based extension of the NLLB model, followed by fine-tuning under two experimental configurations.

Corpus Construction

Given the absence of widely accepted standard varieties and the substantial dialectal variation in the selected languages, the authors adopt a pragmatic data collection strategy. Volunteer translators, who are native speakers with strong writing skills, are recruited through social media and personal networks. Each translator receives source sentences in English and in another language they are fluent in (typically Farsi or Turkish), which accommodates varying English proficiency and enables potential cross-lingual learning. The translation task is carried out primarily through spreadsheets, with quality control mechanisms in place. Translators are permitted to skip sentences containing inconsistencies, spelling errors, code-switching, or named entities, and they are required to maintain consistent orthography and preserve the source text as-is. At least two additional translators verify the translations at the end of the campaign.

Data Selection

To maximize the utility of limited human translation effort, the authors introduce a selection algorithm aimed at increasing lexical diversity and semantic coverage. Given a large bilingual corpus $P$P containing source sentences $s$s (in a high-resource language such as Farsi) and their English translations $t$t, the method proceeds in batches. In the first batch, a random valid subset $C_1$C1​ is sampled, where validity is determined by filtering out named entities, ellipses, code-switching, URLs, and other less informative content. For each subsequent batch, candidate pairs are scored based on two criteria. The average edit distance between each new source sentence $s'_j$sj′​ and all previously selected sentences is computed as

and the average semantic similarity between the English translations of the candidate and the previously selected pairs is computed in the embedding space as

where $\mathcal{E}$E is a sentence embedding function based on Sentence-BERT (all-MiniLM-L6-v2). The final diversity score for each candidate is the ratio of edit distance to semantic similarity:

Candidates with higher edit distances are rewarded, while those with high semantic similarities are penalized. The top-scoring $n$n pairs are selected per batch, and the process iterates until the corpus reaches a target size of 15,000 sentences. The source corpus is the Mizan Farsi-English corpus, chosen because the volunteer translators are familiar with Farsi. Punctuation marks are removed from both sentences when computing edit distances.

Translation Model Fine-Tuning

For training, the authors adopt a transfer learning approach centered on the NLLB model, a multilingual encoder-decoder system supporting 203 languages. Since the target languages are not natively supported, they extend the model's language capability through a token-based embedding initialization strategy. New language-specific tokens (e.g., sdh_Arab for Southern Kurdish) are added to the tokenizer, and each new token's embedding is cloned from a linguistically related language already present in NLLB. For example, Southern Kurdish is initialized using Central Kurdish embeddings, and similar mappings are applied for the other languages based on language proximities. This initialization leverages the model's existing knowledge of related languages and writing systems without requiring architectural changes or full retraining.

Baselines and Transliteration

The authors evaluate NLLB's zero-shot performance using two model variants: the 3.3B model and its 600M distilled variant. Because NLLB's training data does not include indicator tokens for the target languages, zero-shot performance is assessed by feeding sentences with proximal language tokens such as Arabic, Turkish, Central Kurdish, Northern Kurdish, Farsi, and English. To mitigate the impact of divergent writing systems (Perso-Arabic versus Latin scripts), a transliteration pipeline is also explored. For Southern Kurdish, Laki, Hawrami, and Zazaki, a rule-based transliteration system is extended with variety-specific character mappings; for the remaining languages, a phonetic-based mapping converts each character to its closest Latin phonological equivalent.

Fine-Tuning Setup and Configurations

Fine-tuning focuses on the X-to-English translation direction. The training data is merged and converted to JSON Lines format, with each entry specifying the newly introduced source language token paired with English translations indicated by eng_Latn. The base setup uses the training sets paired with English, and for languages lacking dedicated training data (BQI and TLY), validation sets are repurposed as training data. The augmented setup expands the training data by using NLLB to translate additional available translations into English, leveraging Northern Kurdish for Zazaki and Farsi for all other languages. Three fine-tuning configurations are explored: an initial setup with batch size 8, learning rate 3e-5, and 20 epochs; an enhanced setup with batch size 16, learning rate 5e-4, and 50 epochs; and a third configuration that retains the enhanced parameters but trains for 100 epochs. The multilingual tokenization explicitly sets the source language token during preprocessing while fixing English as the target.

Experiment

The study evaluates low-resource dialect translation by fine-tuning NLLB on curated evaluation sets, with zero-shot baselines showing poor performance. Fine-tuning yields substantial gains, with the base setup generally outperforming the augmented setup, indicating data quality outweighs quantity. Ablation studies reveal that training data size and cross-lingual transfer impact quality, but hyperparameters and data quality also play crucial roles, while excluding certain languages occasionally benefits others, suggesting interference among closely related varieties.

The table shows that major regional languages such as Arabic, Hebrew, Turkish, and Persian have full resource and tool coverage, while minority languages like Northern Kurdish and Central Kurdish lack certain resources, notably WordNet and Universal Dependencies respectively. This contrasts starkly with the dominant languages, highlighting the uneven digital readiness across the selected varieties. Arabic, Hebrew, Turkish, and Persian have complete availability across all listed resources and tools. Northern Kurdish lacks WordNet support, while Central Kurdish lacks Universal Dependencies annotations. All selected languages have language identification, machine translation, and spell checkers except for a few gaps in tools like ASR for some varieties.

The initialization strategy adds new language-specific tokens to NLLB, cloning embeddings from closely related languages: Farsi for Luri Bakhtiari, Gilaki, Mazandarani, and Talysh, and Central Kurdish for Hawrami, Laki, and Southern Kurdish. This approach leverages pre-existing linguistic knowledge, particularly writing systems, to efficiently extend the model's multilingual capabilities. New tokens are initialized using embeddings from Farsi for four languages and from Central Kurdish for three others. The initialization choice follows linguistic similarity, aiming to benefit from the model's prior knowledge of the source languages' writing systems.

Fine-tuning substantially improves BLEU scores over zero-shot baselines, with the base data setup generally outperforming the augmented setup. Among configurations, CONFIG II yields the best average performance, and further training modestly improves it. The augmented setup is particularly helpful for languages closely related to Farsi, while data quality appears more influential than quantity. The base data setup mostly surpasses the augmented setup across configurations, indicating that data quality matters more than volume. CONFIG II achieves the highest average BLEU score, and additional fine-tuning yields only a slight further improvement. Languages closely related to Farsi benefit from the augmented setup, with some exceeding 10 BLEU points.

Excluding a language's fine-tuning data typically reduces that language's BLEU score compared to the model trained on all 1,000-sample languages, but the effect varies by language. Cross-lingual transfer is evident because some languages score higher when another language is excluded, while others suffer. The fully fine-tuned model consistently outperforms all ablation variants and zero-shot baselines. Removing a language's training data often lowers its own BLEU score, such as Gilaki dropping from 2.40 to 1.44 when excluded. Some exclusions improve other languages, for instance Gilaki reaches 2.99 when Zazaki is excluded, surpassing its score in the full 1,000-sample model. The full fine-tuned model achieves the highest BLEU scores for every language, while zero-shot baselines remain far lower.

The evaluation highlights uneven digital readiness among the studied languages: major regional languages have full resource coverage, while minority varieties face gaps in resources like WordNet and Universal Dependencies. Initializing new tokens from closely related languages (e.g., Farsi for Luri Bakhtiari, Central Kurdish for Hawrami) effectively extends the multilingual model. Fine-tuning substantially outperforms zero-shot baselines, with data quality proving more influential than quantity, and the best configuration yielding modest gains from further training. Ablation studies reveal that excluding a language's training data often lowers its own performance, yet cross-lingual transfer can benefit other languages, while the fully fine-tuned model consistently achieves the highest scores.