One-sentence Summary

Researchers at Nanjing University find that LLMs establish human-like learning preferences for data with conflicting knowledge, favoring formal and correctly-spelled texts whose features signal consistency with the majority of training data, and show that these preferences can be instilled or erased by manipulating the degree of consistency with majority data.

Key Contributions

• Construct an evaluation framework for conflicting-knowledge learning on synthetic biography data by forming template-filled statement pairs and defining a pairwise preference score from the relative probabilities assigned by pretrained LLMs.

• The evaluation shows that pretrained LLMs learn knowledge faster from formal texts and texts with fewer spelling errors, assign such conflicting statements higher probability, and display this preference across models and languages, with stronger effects in larger models.

• The analysis identifies consistency with the majority of training data as a driver of these preferences and shows that changing this consistency can instill new learning preferences and erase existing ones.

Introduction

Large language models acquire much of their knowledge from large web-scraped pretraining corpora, but these corpora can contain misleading or contradictory facts. This matters because it affects how reliable knowledge is learned, retained, and recalled. Humans often resolve such conflicts by considering source quality and agreement with other information. Prior work has mainly examined how knowledge is stored in model parameters or how formats such as paraphrasing improve knowledge acquisition, leaving limited insight into how textual features influence learning when conflicting facts compete.

The authors leverage synthetic biographical data with controlled conflicting knowledge and different text styles and spelling conditions. They find that pretrained LLMs show learning preferences: they learn and assign higher probability to facts presented in formal, news or scientific styles and to texts with fewer spelling errors. Their main contribution is a Consistency-driven Feature Preference Hypothesis, arguing that LLMs learn from features that signal consistency with other data, and they show that these preferences can be introduced, neutralized, or reversed.

Dataset

• The authors build a synthetic preference dataset by attaching invented source information to knowledge items used for training and testing LLM preference behavior. The original knowledge set is partitioned into an evidence knowledge set and a test knowledge set. The evidence set is used to construct training examples that teach consistency patterns, while the test set is reserved for measuring preference scores at inference time.

• Two synthetic feature types are injected into the data as source metadata:

  • Source-name feature: each example is prefixed with a synthetic newspaper name generated by GPT-4. Two separate name sets are created, one for feature A and one for feature B, using the template “According to , T.”
  • Source-time feature: all examples use the same source name, “Global News,” but include a volume number. Feature A uses random volume numbers below 1000, while feature B uses random volume numbers above 1000, using the template “According to Global News (Vol. ), T.” This makes the model infer consistency from the numeric relationship rather than from a fixed surface name.

• The evidence subset uses conflicting knowledge pairs. For each evidence item k_A, the authors generate a conflicting item k_B and create a group of m + n + 2 biographies. This includes one example with feature A and k_A, one example with feature B and k_B, m neutral-template support examples for k_A, and n neutral-template support examples for k_B. The values of m and n are adjustable, allowing the authors to control the consistency ratio between feature A and feature B.

• The test subset is smaller in structure for each knowledge item. For each test item k_A, the authors generate a conflicting item k_B and create only two records, one with feature A and one with feature B. These examples do not include neutral-template support records.

• For training, the authors combine all evidence-subset biography groups and test-subset pair groups into the fine-tuning data for LLaMA-2-7B. The paper does not provide fixed numeric dataset sizes; scale depends on the number of original knowledge items and the selected values of m and n. No external corpus, filtering rules, cropping strategy, or detailed metadata schema are described beyond these synthetic prefixes and template-based construction rules.

Method

The authors formulate the method as a data driven probing framework for identifying the textual features that influence knowledge learning in large language models. They assume that during training the model can implicitly associate style related cues, such as genre, source formatting, spelling correctness, or synthetic source identifiers, with the reliability of the contained knowledge. This assumption leads to a consistency driven preference hypothesis: if two features are attached to competing statements, the model should prefer the statement whose feature appears more consistently with other data.

To make this idea testable, the authors design controlled synthetic features that are unlikely to be present in the original pretraining corpus. Two feature families are introduced. The first is a source name feature, where a neutral knowledge sentence $T$T is prefixed with a synthetic source marker. One version uses a generated newspaper name:

$$\tilde{T}=\text{According to }\langle\text{newspaper}\rangle\text{, }+T$$T~=According to ⟨newspaper⟩, +T

The second is a source time feature. It uses the same newspaper name but varies a publication volume token, making the model infer consistency from a numeric relationship rather than a fixed surface string. Text associated with feature A receives volumes smaller than $1000$1000, while text associated with feature B receives volumes larger than $1000$1000:

$$\tilde{T}=\text{According to Global News }\left(\text{Vol. }\langle\text{vol}\rangle\right)\text{, }+T$$T~=According to Global News (Vol. ⟨vol⟩), +T

Refer to the causal graph below.

The core of the training workflow is a consistency ratio control pipeline. Given two features A and B, the authors partition the knowledge set into an evidence set and a test set. The evidence knowledge set $\mathcal{K}^{e}$Ke is used to construct training biographies that provide the model with clues about which feature is more consistent with other data. The test knowledge set $\mathcal{K}^{t}$Kt is reserved for inference and is not used to provide corroborating evidence during training.

For each evidence knowledge item $k_{A}$kA​ in $\mathcal{K}^{e}$Ke, the authors generate a conflicting item $k_{B}$kB​ and then compose a biography group containing one feature conditioned claim for each side plus several neutral claims that support one side or the other:

$$I^{e}=\{\tilde{T}_{A}(k_{A}),\tilde{T}_{B}(k_{B})\} \cup \{T_{i}(k_{A})\}_{i=1}^{m} \cup \{T_{j}(k_{B})\}_{j=1}^{n}$$Ie={T~A​(kA​),T~B​(kB​)}∪{Ti​(kA​)}i=1m​∪{Tj​(kB​)}j=1n​

Here $\tilde{T}_{A}$T~A​ and $\tilde{T}_{B}$T~B​ are templates carrying feature A and feature B respectively. The terms $\{T_{i}(k_{A})\}_{i=1}^{m}${Ti​(kA​)}i=1m​ and $\{T_{j}(k_{B})\}_{j=1}^{n}${Tj​(kB​)}j=1n​ are neutral support sets that do not carry the experimental feature. By varying $m$m and $n$n, the authors manipulate the consistency ratio of the evidence, effectively changing how often knowledge associated with A appears in agreement with other training examples relative to knowledge associated with B.

For each test item, the training data is reduced to a minimal conflict pair:

$$I^{t}=\{\tilde{T}_{A}(k_{A}),\tilde{T}_{B}(k_{B})\}$$It={T~A​(kA​),T~B​(kB​)}

The full fine-tuning corpus is then the union of all evidence biographies and all test conflict pairs:

$$\mathcal{D} = \bigcup_{k_{A}\in\mathcal{K}^{e}}I^{e}(k_{A}) \cup \bigcup_{k_{A}\in\mathcal{K}^{t}}I^{t}(k_{A})$$D=kA​∈Ke⋃​Ie(kA​)∪kA​∈Kt⋃​It(kA​)

The authors fine-tune LLaMA-2-7B on this constructed corpus. After training, they evaluate on the test knowledge set and compute an observed preference score $\Pr(A,B)$Pr(A,B), which indicates whether the model tends to adopt the knowledge paired with feature A or the competing knowledge paired with feature B. This gives a measurable readout of preference formation.

In parallel, the method also considers nonconflicting settings, where preference is inferred indirectly from convergence speed and training accuracy. When conflicts are present, the same experimental logic applies to feature families such as text style and spelling correctness. Correctly spelled text is denoted as $T_{\mathrm{GoodSpelling}}$TGoodSpelling​, and corrupted variants are compared against it to test whether orthographic quality influences learning preference. The central design principle across all variants is the same: control the consistency signal attached to different feature pairs, train under that controlled condition, and then quantify the preference that emerges at inference time.

Experiment

The experiments evaluate whether language models develop fine-tuning preferences from textual features such as style, spelling, information source, and consistency with other data. Learning-speed and conflicting-knowledge tests show that models favor formal styles and correctly spelled text, and this bias becomes stronger with model scale and appears across English and Chinese models. Follow-up consistency, representation, and counterfactual experiments further indicate that models identify features associated with majority-aligned knowledge, and that artificially changing this consistency can erase or reverse their learned preferences.

The examples show synthetic biography passages written with different textual features, including general, poor-spelling, newspaper-style, and novel-style forms, all describing the same person with conflicting facts. These variants illustrate how text style and spelling can be used as signals for information credibility in the analysis. The surrounding context indicates that newspaper-style biographies are associated with stronger learned preference than novel-style biographies. Biography examples encode the same person name with different attributes across text features. Poor spelling and narrative style are treated as distinguishable features that may affect model preference. Newspaper-style examples are presented as more credible than novel-style examples in the preference analysis.

The fine-tuned LLaMA-2-7B comparisons show a consistent preference for formal or standard text features over less formal alternatives. Newspapers and scientific reports are generally favored compared with novels and social media, while good spelling is favored over poor spelling. The relative scores vary by factual attribute, but the overall trend indicates stronger knowledge activation from formal and correctly spelled sources. Formal text styles receive higher preference than novels and social media. Good spelling is preferred over poor spelling across tested biographical attributes.

The finetuned LLaMA-2-7B model generally shows stronger pairwise preference for formal writing styles than for informal ones. Newspaper and scientific report styles usually receive higher scores than novels and social media styles across attribute categories. Comparisons between the two formal styles are close, as are comparisons between social media and novels, indicating that the clearest preference gap is formal versus informal text. Formal styles receive higher pairwise preference than novels and social media Newspaper and scientific report styles are nearly balanced against each other

Across English and Chinese large language models, newspaper style is consistently preferred over social-media style, and good spelling is generally preferred over poor spelling. The preference is not equally strong for every model, with Pythia showing the strongest newspaper-style preference among the tested models. This pattern suggests a broadly shared learned preference, while differences in strength likely reflect model-specific training data and training methods. Newspaper style is preferred over social-media style across English and Chinese models. Good spelling is favored over poor spelling across the tested models. Pythia shows the strongest newspaper-style preference compared with the other models.

Experiments use synthetic biography passages that encode the same person with conflicting facts across different text styles and spelling conditions, examining how such features act as credibility signals. Across comparisons with fine-tuned LLaMA-2-7B and multiple English and Chinese models, formal styles such as newspaper and scientific reports are generally preferred over informal styles such as novels and social media, and correctly spelled text is favored over poor spelling. The preference is strongest between formal and informal text, while specific comparisons and model behaviors vary, indicating a broadly shared but model-dependent learned preference.