One-sentence Summary

The authors introduce a counterfactual ablation framework that detects scientific revolutions by measuring geometric perturbations in document embedding spaces when removing concept-related documents before and after a concept's emergence, and validate it on five historical case studies spanning physics, mathematics, and machine learning, including special relativity, Gödel's incompleteness theorems, the Higgs mechanism, deep learning, and the attention mechanism underlying transformer architectures, where the framework identifies measurable geometric signatures of conceptual reorganization but also reveals limitations from document assignment and sparse historical data.

Key Contributions

• Document embedding geometry is introduced as a quantitative observable for conceptual reorganization, with reorganization measured through geometric perturbations in embedding spaces when documents associated with a concept are removed.
• A counterfactual ablation framework is developed that quantifies how individual concepts influence knowledge organization by comparing embedding geometry before and after a concept's historical emergence.
• Validation across five historical case studies (special relativity, Gödel's incompleteness theorems, the Higgs mechanism, deep learning, and the attention mechanism) shows the framework detects geometric signatures of conceptual change and distinguishes localized breakthroughs from broader paradigm shifts, while also revealing limitations from document assignment and sparse data.

Introduction

Understanding how conceptual knowledge reorganizes during scientific breakthroughs has traditionally relied on qualitative historical analysis. Prior computational approaches, such as citation network analysis and topic modeling, capture external patterns of scientific change but do not directly measure the internal structural role of concepts within the semantic organization of literature. The authors introduce a framework that treats document embedding geometry as a quantitative observable of conceptual reorganization. By counterfactually removing documents associated with a concept and measuring the resulting perturbation in embedding space before and after its historical emergence, they quantify the concept’s structural influence without relying on citation graphs or external metadata.

Method

The authors leverage a six-step counterfactual ablation framework to quantify the structural impact of a target concept on the organization of scientific literature. The complete pipeline is illustrated in the framework diagram below.

The first phase constructs a geometric representation of the corpus. Scientific documents are divided into overlapping chunks, which are independently embedded using sentence-transformer models. The final document representation is the $\ell_2$ℓ2​-normalized mean of its chunk embeddings. Rather than using unsupervised clustering, the authors define a fixed set of concepts represented by short natural-language descriptions. For each concept, five paraphrases of its description are embedded, and the concept anchor is computed as the $\ell_2$ℓ2​-normalized mean of these embeddings. Documents are assigned to the concept with the highest cosine similarity, provided the similarity exceeds a per-concept baseline threshold and the margin to the second-best concept exceeds a confusion-aware threshold.

The second phase implements the counterfactual framework. For a given target concept, all associated documents are removed from the corpus. Concept centroids are then recomputed within rolling time windows. The geometry of the concept space is characterized using two complementary observables: total inertia and mean pairwise cosine distance. Total inertia $d_I(t)$dI​(t) measures the overall spread of the concept space and is defined as:

$$d_I(t) = \sum_{c=1}^{N(t)} \left\| \mathbf{c}_c(t) - \bar{\mathbf{c}}(t) \right\|^2$$dI​(t)=c=1∑N(t)​∥cc​(t)−cˉ(t)∥2

where $\mathbf{c}_c(t)$cc​(t) is the centroid of concept $c$c and $\bar{\mathbf{c}}(t)$cˉ(t) is the average centroid. The mean pairwise cosine distance $d_P(t)$dP​(t) measures the average angular separation between concepts:

$$d_P(t) = \binom{N(t)}{2}^{-1} \sum_{i < j} \left(1 - \cos(\mathbf{c}_i(t), \mathbf{c}_j(t))\right)$$dP​(t)=(2N(t)​)−1i<j∑​(1−cos(ci​(t),cj​(t)))

The yearly geometric perturbation induced by the ablation is calculated as the difference between the baseline and ablated geometries, $\Delta(t) = g^{\text{base}}(t) - g^{\text{abl}}(t)$Δ(t)=gbase(t)−gabl(t), where $g$g denotes either $d_I$dI​ or $d_P$dP​. To quantify the asymmetry of this perturbation around a candidate pivot year $t^*$t∗, the authors compute Cohen's $D$D, which standardizes the difference between the average post-pivot and pre-pivot perturbations by their pooled standard deviation.

To localize conceptual reorganizations in a data-driven manner, the authors evaluate Cohen's $D$D over all combinations of candidate concepts and pivot years, producing a concept pivot-year grid. This grid acts as a two-dimensional map of the standardized response.

The heatmap above illustrates the resulting grid for a specific historical case, highlighting the pivot year where the target concept produces the strongest geometric perturbation. The corresponding per-metric decomposition into $D_I$DI​ and $D_P$DP​ is shown in the figure below.

Finally, the framework employs a suite of statistical validation tests to ensure the detected signals are not artifacts of corpus composition or individual influential publications. One such test is a leave-one-out jackknife, which evaluates the stability of the standardized response against the deletion of individual documents.

This validation confirms that the measured geometric perturbation is a collective effect driven by the structural reorganization of the field, rather than being dominated by a single publication.

Experiment

The evaluation applies counterfactual ablation to a special relativity corpus and four additional historical revolutions (Gödel, Higgs, deep learning, attention mechanism) to measure geometric perturbations in embedding space. Statistical tests show the signals are concept-specific, robust to individual documents, and not artifacts of multiple comparisons, while document-assignment and cross-encoder analyses confirm methodological stability. The framework consistently identifies the primary conceptual reorganization in most cases, with the attention mechanism yielding a weaker but still significant signal, and the Higgs case highlighting a vulnerability to sparse pre-pivot assignments.

The geometric signatures of conceptual reorganization are robust across five diverse embedding models, confirming that the findings do not depend on a single encoder. While transition year localization is largely consistent across models, the specific ranking of concepts varies with the embedding representation, indicating that semantic representation quality influences the measurement of conceptual organization. The framework recovers the geometrical signal across encoders differing in architecture, training objective, and corpus coverage. Different encoders independently select pivot years that agree with the primary model exactly in about half of cases and to within a few years in the large majority. Concept ranking depends on the specific embedding model, suggesting that semantic representation quality is an ingredient in measuring conceptual reorganization.

At the pivot year 1902, special relativity produces a counterfactual perturbation (max |D| = 7.61) that is six times larger than any other concept, dominating the embedding geometry. Concepts such as aether optics and thermodynamics show negative total-inertia responses, indicating their geometric influence is concentrated before 1902, consistent with their declining structural role after special relativity's emergence. Special relativity achieves a max(|D_I|, |D_P|) of 7.61, far exceeding the second-ranked concept spectroscopy at 1.26. Aether optics and thermodynamics have negative D_I values (-0.78 and -0.67), meaning their removal perturbs the earlier embedding geometry more than the later one.

The five case studies vary widely in corpus size, from 599 to 37,598 documents, and span different historical periods. Smaller corpora use a two-year rolling window to accumulate enough papers per window, while the larger machine-learning corpora use a one-year window for finer temporal resolution. Target concept papers represent a very small fraction of each corpus, and the share of documents excluded by the confusion-aware margin is substantial, particularly for the physics cases. Smaller corpora (special relativity, Gödel incompleteness, Higgs mechanism) use a rolling-window radius of 2 years, while the large deep learning and attention corpora use a radius of 1 year. Target concept papers account for only 0.2% to 6.5% of total documents, with the attention mechanism case having the smallest share (0.2%) and deep learning the largest (6.5%).

Across five historical case studies, counterfactual removal of the target concept produced the largest geometric perturbation in four cases, with the attention mechanism ranking fourth. The total-inertia and pairwise-distance observables captured complementary reorganization patterns: special relativity and attention were dominated by changes in overall concept-space spread, while Gödel incompleteness, the Higgs mechanism, and deep learning were dominated by changes in pairwise separation. Null tests using random removal, scrambled assignments, and a look-elsewhere correction confirm the perturbations are statistically significant. In four of five case studies, the target concept's ablation caused the strongest structural reorganization among all candidate concepts. The attention mechanism was the only exception, ranking fourth within its corpus. Special relativity and the attention mechanism showed dominant total-inertia response, indicating the target contributed more to concept-space spread after the pivot than before. Gödel incompleteness, the Higgs mechanism, and deep learning showed dominant pairwise-distance response, reflecting changes in how distinguishable the target concept became from others. Null tests with 5,000 permutations each (random removal, scrambled assignment, look-elsewhere effect) confirm the observed perturbations are unlikely under random chance.

Across five historical case studies, the target concept's rank remains largely stable across different embedding models, with special relativity and the Higgs mechanism ranked first by a majority of encoders. Deep learning and attention mechanism exhibit greater inter-model spread, and the citation-prediction model SPECTER2 yields a null result for Gödel incompleteness due to corpus homogeneity. Independently selected pivot years agree with the primary model in about half of cases and within a few years for most, confirming that transition-year localization is not an artifact of a single encoder. For special relativity, four of five encoders rank the target concept first, and for the Higgs mechanism, three do, while for Gödel incompleteness, four place it among the top five (SPECTER2 returns a null result). Pivot years selected independently by different encoders agree exactly with the primary model in roughly half of all cases and to within a few years in the large majority, demonstrating robust localization of the transition year.

The evaluation applies a geometric framework across five diverse embedding models and five historical case studies to detect conceptual reorganization in scientific literature. The method robustly localizes pivot years across encoders and confirms that counterfactual removal of the target concept produces the largest structural perturbation in most cases, with reorganization manifesting either as changes in overall concept-space spread or pairwise separation. The findings demonstrate that semantic representation quality influences concept ranking but does not undermine the framework's ability to recover consistent geometric signals of intellectual shifts.