One-sentence Summary

Researchers at Saarland University and the University of Toronto propose a neural-network measure of complexity based on the learnability of meaning-to-form mappings that captures fine-grained systematicity in verb and pronoun forms across typologically diverse languages, bridging efficient communication theory and systematicity in natural language.

Key Contributions

• A unified framework integrates systematicity between grammatical meanings and word forms into efficient communication theory, modeling verb and pronoun form-meaning mappings as shaped by competing pressures for simplicity and accuracy across typologically diverse languages.

• A novel neural-network-based complexity measure, defined as the inverse of simplicity, quantifies the learnability of meaning-to-form mappings and captures fine-grained systematic regularities, including partial systematic correspondences that extend beyond syncretism.

• Evaluation on hundreds of languages shows that attested paradigms are more efficient than counterfactual paradigms, that more natural syncretism patterns are more learnable, and that the approach outperforms the prior Information Bottleneck method in discriminating attested from unattested systems.

Introduction

Languages map many possible meanings onto a limited set of forms, and cross-linguistic differences in these form-meaning mappings affect how precisely listeners can infer intended meanings. Prior work on efficient communication shows that lexical systems often balance simplicity and accuracy, but it lacks a general framework for modeling systematic structure inside word forms. Laboratory studies of language evolution show that systematic form-meaning mappings can emerge from pressures toward simplicity and accuracy, yet this insight is not directly integrated into models of natural-language efficiency. The authors address this gap by proposing an information-theoretic framework that incorporates systematicity into efficient communication theory, using a new complexity measure based on how learnable meaning-to-form mappings are. They evaluate the framework on verbs and pronouns across languages, showing that explicit modeling of systematicity better distinguishes attested linguistic systems from unattested ones.

Dataset

• The authors use evaluation data organized into three language morphology domains, summarized in Table 3, with additional source details in Appendix D. The paper does not report explicit training splits, mixture ratios, or dataset sizes for all domains.

• Domain 1: Verbal Morphology (VERB)

  • Collected by the authors.
  • Covers verbal inflection in Semitic, non-Semitic Afro-Asiatic, Germanic, and Romance languages.
  • Includes both concatenative and nonconcatenative morphology.
  • Form and feature representations are illustrated in Table 1 and described further in Appendix A.

• Domain 2a: Pronouns (PPD)

  • Uses the Pronoun Paradigms Database from Zaslavsky et al.
  • Includes hundreds of languages across families.
  • Follows a simplified feature scheme for direct comparison.
  • Uses only masculine forms and omits dual number.

• Domain 2b: Detailed Pronouns (PRON)

  • Collected by the authors for selected language families, specifically Semitic and Germanic.
  • Tests full pronoun paradigms, including case and gender distinctions.
  • Unlike Domain 2a, the feature set varies by language to capture language-specific distinctions.

• Need Distribution

  • The authors estimate the need distribution $p_{cog}(t)$pcog​(t) by combining the need distribution from Zaslavsky et al. with additional corpus frequency estimates for gender and dual/plural distinctions.
  • Other categories are assumed to be uniformly distributed.
  • The authors note that results are robust to this choice.

• Processing and Metadata

  • Dataset construction involves form and feature representation, language-specific feature schemes, and pronoun paradigm organization.
  • No cropping strategy or additional metadata pipeline is described beyond these feature and distribution specifications.
  • The data are used for evaluation across verbal morphology and pronoun paradigms rather than for reporting a training mixture.

Method

The authors leverage an information-theoretic view of morphological paradigms to combine two pressures: communicative accuracy and learnability based complexity reduction. The basic pipeline samples a target from a need distribution, encodes it into a surface form, and decodes that form back into a distribution over meanings. Refer to the framework diagram below.

The complexity route operates from meanings to forms. It asks how much information the encoder must transmit to convey the intended grammatical meaning. The accuracy route operates from a received form to reconstructed meanings. The listener applies Bayesian inference, using the need distribution and a semantic geometry, to infer which meaning the form likely represents. The authors keep the accuracy side close to the established information bottleneck approach, but modify the encoding side so that forms are not atomic labels. Instead, each form is represented as a character sequence. This makes the encoder sensitive to shared prefixes, shared suffixes, and other systematic form-level regularities. A conventional mutual information measure based on atomic forms is

where

Because the form $w$w is treated as a discrete outcome, that measure can register reductions in complexity when forms are reused, but it cannot directly register reductions caused by systematic segmentation within forms.

The proposed complexity measure treats a paradigm as a supervised sequence generation problem. The input is the set of morphosyntactic features for a target meaning, and the output is the corresponding surface form. As shown in the training example below:

The encoder models the mapping $q_{\mathrm{enc}}(w|m_t)$qenc​(w∣mt​). It takes categorical grammatical features $m_t$mt​ and generates the surface form $w$w as a character sequence. The implementation uses an LSTM based sequence to sequence architecture, with two stacked LSTM layers in both the encoder and the decoder. By generating characters rather than atomic symbols, the encoder can exploit regularities across forms. For example, shared prefixes, shared suffixes, or other stable morphological patterns can make the mapping from features to forms easier to learn.

The need distribution $p_{\mathrm{cog}}(t)$pcog​(t) weights target meanings according to communicative need. The authors estimate it from corpus frequency, which assigns greater importance to meanings that are more frequently encountered. This gives the model the same intuitive weighting used in efficient communication accounts: frequently needed meanings receive more influence in the overall complexity score.

Complexity is then measured through the cross entropy training loss, or CETL. If $q_{\mathrm{enc}}^{(T)}(w_t|m_t)$qenc(T)​(wt​∣mt​) is the encoder probability of the correct form after $T$T training epochs, the measure is

Here, $w_t$wt​ is the correct surface form for meaning $m_t$mt​, and $T_{\max}$Tmax​ is the maximum number of training epochs. The quantity $\log q_{\mathrm{enc}}^{(T)}(w_t|m_t)$logqenc(T)​(wt​∣mt​) reflects how strongly the network predicts the correct form at each stage of learning. If a paradigm contains stable form meaning regularities, the network is expected to converge faster, producing lower cumulative training loss. Thus, lower CETL corresponds to lower cognitive complexity under the authors’ learnability criterion.

In the implemented training pipeline, the model uses batch size one, a dropout rate of 0.5, and $T_{\max}=50$Tmax​=50 epochs. The total cross entropy loss is weighted by the need distribution. For each original paradigm, the authors train ten separate networks and average the results. For each counterfactual paradigm, they train five separate networks and average those results as well.

Accuracy is retained from the information bottleneck view. A listener receives a form $w$w and constructs a probability distribution $\hat{m}_w$m^w​ over possible meanings. The listener uses Bayesian inference together with the need distribution and the semantic geometry of the target domain. The accuracy score is defined as the expected negative Kullback-Leibler divergence between the reconstructed meaning distribution and the intended meaning distribution:

Because speakers are assumed to produce the correct form for a target meaning, $m_t$mt​ is treated as a one-hot vector.

To reconstruct meanings, the model represents grammatical meanings as categorical features. Meanings that share more feature values are treated as more similar. One simple distance is the number of differing feature dimensions:

The corresponding target distribution is specified up to normalization by

This captures the intuition that meanings such as first person singular past and first person plural past are closer than first person singular past and third person singular future, because they share more grammatical features.

The evaluation design generates counterfactual paradigms from existing ones by permuting paradigm cells. Structural permutations swap meanings across cells, which changes which meanings are expressed by syncretic forms. These changes affect both the naturalness of the paradigm and, in some cases, the accuracy complexity tradeoff. Form only permutations swap surface forms while preserving the original syncretism pattern. These permutations do not change the syncretic structure, so they leave atomic form complexity unchanged. However, they can disrupt systematic regularities in the actual surface strings, such as a shared prefix or suffix pattern. The authors then train the same sequence generation models on the original and counterfactual paradigms and compare CETL and accuracy. This makes it possible to test whether the proposed learnability measure captures systematic covariation across forms, not merely repetition of whole forms.

Experiment

The experiments test whether natural language verb and pronoun paradigms are optimized for communicative efficiency and whether systematic syncretism patterns improve learnability, by comparing attested paradigms with structurally and formally permuted counterfactuals. Results show that attested paradigms achieve better complexity-accuracy trade-offs than most counterfactuals and that paradigms with more natural syncretism are simpler and more learnable. Compared with the IB framework, the proposed learnability-based measure better identifies attested paradigms as efficient and captures fine-grained systematic form-meaning regularities that the IB measure cannot detect.

The paradigm shows how Arabic verb forms vary with person, gender, and number through prefixes, stem changes, and suffixes. Several distinct feature combinations share the same form, while recurring prefixes such as ta- mark second person. This structured variation supports a model that scores natural paradigms as more learnable and more efficient than randomized alternatives. Second person forms commonly use the prefix ta- across the paradigm. Syncretism appears when different feature combinations share identical forms. Structured form-meaning mappings are linked to lower complexity and higher accuracy.

The evaluation datasets span two main morphological settings: a very large pronoun collection and smaller family-specific verbal paradigm datasets. The pronoun collection covers many languages and uses a simplified person and number scheme, while the family-specific subsets focus on verbal morphology and include richer feature categories such as gender and case. This structure supports comparisons across language families and between simplified pronoun paradigms and more feature-rich morphological paradigms. The pronoun dataset is the broadest subset, covering many languages with a simplified person and number representation. Family-specific subsets are smaller but emphasize verbal morphology, with feature sets that can include person, number, gender, and case.

Across structural and surface permutation tests, CETL shows consistently higher performance than IB in identifying attested paradigms as more efficient than counterfactuals. The advantage is especially clear for surface form permutations, where IB cannot distinguish counterfactuals from attested paradigms. This indicates that CETL captures systematic surface structure beyond syncretism patterns. CETL more reliably identifies attested paradigms as more efficient than counterfactuals than IB. The relative advantage extends to surface form permutations that share syncretism patterns with attested paradigms.

The experiments evaluate morphological paradigms using broad pronoun datasets and smaller family-specific verbal datasets, testing whether structured mappings are scored as more efficient than randomized or permuted alternatives. Results indicate that systematic person, gender, number, prefix, and syncretism patterns support learnability and efficiency. CETL consistently outperforms IB in recognizing attested paradigms, especially under surface form permutations, showing that it captures systematic surface structure beyond shared syncretism.