One-sentence Summary

University of Edinburgh researchers show that Arabic Level of Dialectness (ALDi) scores strongly predict interannotator agreement across 15 multi-dialect Arabic sentence-classification datasets, and recommend routing high-ALDi samples preferentially to native speakers of their respective dialects to improve annotation quality.

Key Contributions

• The Sentence-ALDi model is used to estimate Arabic Level of Dialectness (ALDi) as a factor related to annotator disagreement, and this relation is analyzed on 15 public datasets with individual annotations for six Arabic sentence-classification tasks.
• The analysis finds that higher ALDi scores correlate with lower full annotator agreement for 8 of 12 non-dialect-identification datasets, while the 3 dialect-identification datasets show higher agreement for higher-ALDi samples.
• The results support a routing workflow that estimates ALDi first, sends low-ALDi samples to speakers of any Arabic dialect, and routes high-ALDi samples through dialect identification to native speakers of the relevant dialect.

Introduction

Arabic NLP must handle a mix of Modern Standard Arabic and Dialectal Arabic, which is important because user-generated text often combines standardized and regional variants. Prior work showed that sending dialectal content to annotators from the wrong dialect can harm accuracy and fairness, but routing data by dialect is difficult because automatic dialect identification is imperfect and some dialects may have limited annotator availability. The authors leverage an Arabic Level of Dialectness estimate to prioritize effort: low-dialectness sentences can often be annotated by any Arabic speaker, while high-dialectness sentences should be routed to native speakers of the relevant dialect.

Dataset

• The authors build their analysis dataset from publicly released Arabic datasets that mix Modern Standard Arabic and dialectal Arabic, target multiple dialect variants, use sentence-level classification tasks, and expose either individual annotator labels or the percentage of annotators agreeing on the majority label.

• They source candidate datasets from Masader, a community-curated catalog of Arabic datasets, and use each dataset's metadata field for Arabic variants to discard datasets containing only MSA samples.

• They manually inspect the remaining 151 candidate datasets, identify 28 potential matches, contact dataset authors when individual annotations are not publicly released, and ultimately retain 15 datasets for analysis.

• The final 15 datasets cover offensive text classification, hate speech detection, sarcasm detection, sentiment analysis, speech act detection, stance detection, and dialect identification.

• The datasets are composed of sentence-level samples with annotator labels assigned to each sentence. The annotators are speakers of different dialectal Arabic variants and are randomly assigned to samples.

• The authors attach derived metadata to each sample by computing an Arabic Level of Dialectness score with the Sentence-ALDi model. The score ranges from 0 for MSA or non-dialectal sentences to 1 for strongly dialectal sentences.

• They use this derived ALDi score to group samples into ten bins with a width of 0.1. For each bin, they compute the percentage of samples where all annotators agree on a single label.

• The processed data are then used to examine whether higher dialectness scores are associated with lower full annotator agreement.

Method

The authors leverage a filtering, scoring, and correlation pipeline to examine how the Arabic Level of Dialectness, abbreviated as ALDi, relates to annotator agreement. The workflow begins with candidate sentence-level classification datasets that contain a mixture of Modern Standard Arabic and dialectal Arabic. Eligible resources must target multiple dialect variants, include annotators from different dialect backgrounds, and expose individual labels or a majority agreement percentage. The method first removes datasets containing only Modern Standard Arabic material, then screens the remaining candidates for dialect coverage and annotation format. When available labels are incomplete, the authors collect individual annotations from dataset authors to enable agreement measurement.

For each retained sample, the method obtains a dialectal intensity score from the Sentence-ALDi model. This score is a normalized scalar in the interval $0$0 to $1$1, where $0$0 denotes a sample that is judged non-dialectal or compatible with Modern Standard Arabic, and $1$1 denotes a strongly dialectal sample. The score is computed at the sentence level, matching the annotation granularity of the datasets.

The agreement analysis then discretizes the continuous ALDi scores into ten bins of width $0.1$0.1. For bin $b$b, the method defines $x_b$xb​ as the bin midpoint and computes $y_b$yb​ as the percentage of samples in that bin for which all annotators selected the same label. This produces a paired sequence of dialectal intensity and full agreement percentages across the dialectness spectrum.

Finally, the pipeline quantifies the association between dialectal intensity and agreement. It computes Pearson's correlation coefficient $r_{x,y}$rx,y​ over the bin midpoints and the corresponding full agreement percentages. It also fits a linear regression of the form

where $\alpha$α is the intercept, $\epsilon_b$ϵb​ is the residual, and $\beta$β is the slope of the best-fitting line. The reported slope serves as the effect size for the direction of the relationship, while the correlation coefficient summarizes its strength across the binned ALDi range.

Experiment

The experiments examine how annotator agreement varies with predicted dialectality across Arabic datasets, using scatter plots, histograms, and majority-label breakdowns. For non-dialect-identification tasks, higher dialectality generally lowers full agreement, especially for neutral or negative labels that require understanding the whole sentence, whereas salient cues can sustain agreement on more obvious positive or offensive labels. For dialect-identification tasks, agreement is highest when sentences are clearly standard Arabic or strongly dialectal, and lower in intermediate ranges or when standard Arabic is not an option but low dialectality signals remain ambiguous. Overall, agreement often reflects ease of recognition or label cues rather than guaranteed annotation accuracy.

The listed datasets span Arabic offensive, sentiment, sarcasm, stance, speech act, and dialect identification tasks, with different sample sizes, label sets, and annotation counts. They also differ in dialectal content, as several contain a large share of samples expected to be close to Modern Standard Arabic while others are more dialect heavy. The associated results indicate that increasing dialectality usually lowers full agreement for non dialect tasks, while dialect identification shows a more nuanced pattern. The datasets cover multiple Arabic NLP tasks beyond dialect identification. Dialectal content varies across datasets, with different shares of low ALDi samples. For non dialect tasks, full annotator agreement generally decreases as ALDi scores rise. For dialect identification, agreement is often strongest at very low or very high ALDi levels and weaker for intermediate levels.

The samples show that salient words or cues can help annotators assign labels to high ALDi sentences even when other parts are hard to understand. This cue-based labeling appears strongest for positive or overtly marked classes, such as hate speech, but agreement still varies on sentiment examples. The pattern suggests that agreement may reflect shortcut cues rather than full comprehension. Annotators can sometimes identify offensive or hateful samples from brief lexical cues despite low intelligibility elsewhere. High ALDi does not guarantee agreement, because some sentiment samples still receive split labels. Label agreement from cues may not imply correct or fully understood annotation.

The experiments evaluate annotator agreement across multiple Arabic NLP datasets covering offense, sentiment, sarcasm, stance, speech acts, and dialect identification, with varying levels of dialectal content. Higher dialectality generally reduces full agreement for non-dialect tasks, while dialect identification shows a less uniform pattern with stronger agreement at the extremes of dialectality. Further analysis of high-dialectality samples suggests that visible lexical cues can guide annotation, especially for clearly marked classes, indicating that agreement may partly reflect shortcut cues rather than complete understanding.