One-sentence Summary

Researchers from the University of Science and Technology of China and the National University of Singapore propose AlphaEdit, a method that projects parameter perturbations into the null space of preserved knowledge to prevent disruption during sequential LLM editing, theoretically guaranteeing unchanged outputs for preserved queries and boosting existing locating-then-editing methods by an average of $36.7\%$36.7%.

Key Contributions

• The paper introduces AlphaEdit, which projects parameter perturbations into the null space of preserved-knowledge key matrices before applying them to large language models, enabling targeted knowledge updates while reducing disruption to existing knowledge.
• The method provides a theoretical formulation showing that this null-space projection keeps outputs unchanged for queries about preserved knowledge while still permitting updates to edited knowledge, and the projection can be added with a single line of code.
• Experiments on LLaMA3, GPT2-XL, and GPT-J show that AlphaEdit improves most locating-then-editing methods by an average of 36.7%, demonstrating a practical gain in editing performance with preserved knowledge.

Introduction

Large language models can store and recall extensive knowledge, but they often produce stale or incorrect facts, and full fine-tuning is frequently too costly for resource-limited settings. Model-editing methods address this by updating specific stored knowledge with minimal changes, especially through parameter-modifying locate-then-edit approaches that adjust feed-forward layer weights. A key limitation is that prior methods struggle to balance updating target knowledge and preserving existing knowledge: aggressively minimizing update error can shift hidden representations, overfit to the new fact, and cause forgetting or model collapse during sequential edits. The authors introduce AlphaEdit, which removes the preserved-knowledge loss from the objective and projects the parameter update into the null space of preserved key matrices. This preserves the model’s hidden representation distribution while updating facts, and works as a simple plug-and-play enhancement that improved existing editing methods by an average of 36.7 percent.

Method

The authors present AlphaEdit as a projection-based model editing method. Its central idea is to constrain a parameter update to a subspace that is inactive on preserved knowledge, rather than balancing knowledge updating and knowledge preservation through separate loss terms. Let $\boldsymbol{W}$W denote the selected weight matrix to be edited, let $(\boldsymbol{K}_0,\boldsymbol{V}_0)$(K0​,V0​) denote preserved key-value associations, and let $(\boldsymbol{K}_1,\boldsymbol{V}_1)$(K1​,V1​) denote the current knowledge that should be updated.

The method begins with the definition of a left null space. A matrix $\boldsymbol{B}$B lies in the left null space of $\boldsymbol{A}$A if and only if

$$\boldsymbol{B}\boldsymbol{A}=\boldsymbol{0}.$$BA=0.

If an additive perturbation $\boldsymbol{\Delta}'$Δ′ satisfies $\boldsymbol{\Delta}'\boldsymbol{K}_0=\boldsymbol{0}$Δ′K0​=0, then applying it to the original parameters gives

$$(\boldsymbol{W}+\boldsymbol{\Delta}')\boldsymbol{K}_0 = \boldsymbol{W}\boldsymbol{K}_0+\boldsymbol{\Delta}'\boldsymbol{K}_0 = \boldsymbol{W}\boldsymbol{K}_0 = \boldsymbol{V}_0.$$(W+Δ′)K0​=WK0​+Δ′K0​=WK0​=V0​.

Thus, the projected perturbation does not change the output associated with preserved knowledge. AlphaEdit therefore first constructs a projection matrix that maps candidate perturbations into the null space of the preserved knowledge before they are applied to the model.

Direct projection into the null space of $\boldsymbol{K}_0$K0​ is expensive because $\boldsymbol{K}_0$K0​ can have a very large number of columns. The authors instead use the null space of the covariance matrix $\boldsymbol{K}_0\boldsymbol{K}_0^{T}\in\mathbb{R}^{d_0\times d_0}$K0​K0T​∈Rd0​×d0​, since this smaller matrix shares the same null space as $\boldsymbol{K}_0$K0​ while being much cheaper to handle when $d_0$d0​ is much smaller than the number of preserved samples. They compute

$$\{\boldsymbol{U},\boldsymbol{\Lambda},\boldsymbol{U}^T\} = \operatorname{SVD}(\boldsymbol{K}_0\boldsymbol{K}_0^{T}),$${U,Λ,UT}=SVD(K0​K0T​),

where the columns of $\boldsymbol{U}$U are eigenvectors of $\boldsymbol{K}_0\boldsymbol{K}_0^{T}$K0​K0T​. The eigenvectors corresponding to nonzero eigenvalues are removed, and the remaining submatrix $\hat{\boldsymbol{U}}$U^ is used to define the projection matrix

$$\boldsymbol{P}=\hat{\boldsymbol{U}}\hat{\boldsymbol{U}}^{T}.$$P=U^U^T.

This matrix maps the column vectors of a perturbation into the null space of $\boldsymbol{K}_0$K0​. As a result, for a projected perturbation $\boldsymbol{\Delta}\boldsymbol{P}$ΔP, the update preserves the original key-value association:

$$(\boldsymbol{W}+\boldsymbol{\Delta}\boldsymbol{P})\boldsymbol{K}_0 = \boldsymbol{W}\boldsymbol{K}_0 = \boldsymbol{V}_0.$$(W+ΔP)K0​=WK0​=V0​.

With this projection available, the authors modify the usual model editing objective. Instead of minimizing both the error on the knowledge being updated and the error on preserved knowledge, they replace the raw perturbation $\tilde{\boldsymbol{\Delta}}$Δ~ with the projected perturbation $\tilde{\boldsymbol{\Delta}}\boldsymbol{P}$Δ~P. Because the projection already guarantees preservation of $\boldsymbol{K}_0$K0​, the explicit preservation term can be removed. They also add a regularization term $\|\tilde{\boldsymbol{\Delta}}\boldsymbol{P}\|^{2}$∥Δ~P∥2 to stabilize the solution, yielding

$$\boldsymbol{\Delta} = \underset{\tilde{\boldsymbol{\Delta}}}{\arg\min} \left( \|(\boldsymbol{W}+\tilde{\boldsymbol{\Delta}}\boldsymbol{P})\boldsymbol{K}_1-\boldsymbol{V}_1\|^{2} + \|\tilde{\boldsymbol{\Delta}}\boldsymbol{P}\|^{2} \right).$$Δ=Δ~argmin​(∥(W+Δ~P)K1​−V1​∥2+∥Δ~P∥2).

For sequential editing, the method additionally protects knowledge that has already been updated by previous edits. Let $(\boldsymbol{K}_p,\boldsymbol{V}_p)$(Kp​,Vp​) denote the previously updated key-value associations. The corresponding protection term is

$$\|(\boldsymbol{W}+\tilde{\boldsymbol{\Delta}}\boldsymbol{P})\boldsymbol{K}_p-\boldsymbol{V}_p\|^{2}.$$∥(W+Δ~P)Kp​−Vp​∥2.

Because previous edits have already made $\boldsymbol{W}\boldsymbol{K}_p=\boldsymbol{V}_p$WKp​=Vp​, this term reduces to

$$\|\tilde{\boldsymbol{\Delta}}\boldsymbol{P}\boldsymbol{K}_p\|^{2}.$$∥Δ~PKp​∥2.

Therefore, the sequential editing objective becomes

$$\boldsymbol{\Delta} = \underset{\tilde{\boldsymbol{\Delta}}}{\arg\min} \left( \|(\boldsymbol{W}+\tilde{\boldsymbol{\Delta}}\boldsymbol{P})\boldsymbol{K}_1-\boldsymbol{V}_1\|^{2} + \|\tilde{\boldsymbol{\Delta}}\boldsymbol{P}\|^{2} + \|\tilde{\boldsymbol{\Delta}}\boldsymbol{P}\boldsymbol{K}_p\|^{2} \right).$$Δ=Δ~argmin​(∥(W+Δ~P)K1​−V1​∥2+∥Δ~P∥2+∥Δ~PKp​∥2).

The authors define the current edit residual as

$$\boldsymbol{R}=\boldsymbol{V}_1-\boldsymbol{W}\boldsymbol{K}_1.$$R=V1​−WK1​.

By solving the corresponding normal equation, they obtain a closed-form projected perturbation:

$$\boldsymbol{\Delta}_{\text{AlphaEdit}} = \boldsymbol{R}\boldsymbol{K}_1^{T}\boldsymbol{P} \left( \boldsymbol{K}_p\boldsymbol{K}_p^{T}\boldsymbol{P} + \boldsymbol{K}_1\boldsymbol{K}_1^{T}\boldsymbol{P} + \boldsymbol{I} \right)^{-1}.$$ΔAlphaEdit​=RK1T​P(Kp​KpT​P+K1​K1T​P+I)−1.

This update is designed to incorporate the current knowledge into the model while keeping both preserved knowledge and previously updated knowledge stable. The key difference from common closed-form editing solutions such as MEMIT is the insertion of the projection matrix $\boldsymbol{P}$P into the update pathway. For comparison, a standard unprojected solution has the form

$$\boldsymbol{\Delta}_{\text{MEMIT}} = \boldsymbol{R}\boldsymbol{K}_1^{T} \left( \boldsymbol{K}_p\boldsymbol{K}_p^{T} + \boldsymbol{K}_1\boldsymbol{K}_1^{T} + \boldsymbol{K}_0\boldsymbol{K}_0^{T} \right)^{-1}.$$ΔMEMIT​=RK1T​(Kp​KpT​+K1​K1T​+K0​K0T​)−1.

The AlphaEdit variant replaces the explicit preservation term with the projected null-space constraint. Refer to the framework diagram.

The projection matrix $\boldsymbol{P}$P depends only on preserved knowledge and not on the specific update being performed. Therefore, it can be computed once and reused across many editing operations. This makes the additional computational overhead small relative to existing editing pipelines. As shown in the figure below, this simple projection-oriented modification can lead to stronger editing performance while maintaining general capabilities in the evaluated settings.

Experiment

Experiments evaluate AlphaEdit on sequential knowledge editing, general capability retention, and hidden representation stability across multiple LLMs and editing baselines. Results show that AlphaEdit better balances knowledge update with knowledge preservation and fluency, while baseline methods often degrade general abilities and distort internal representations. The projection strategy also improves existing editing methods, indicating broad applicability and effectiveness.

AlphaEdit delivers the strongest overall results on sequential knowledge editing across multiple base models, with especially large gains in updating effectiveness and generalization. It also preserves generation quality better than existing approaches, improving fluency and consistency while avoiding the severe degradation seen in several baselines. These patterns suggest that better balancing knowledge updates with preservation is central to its advantage. AlphaEdit outperforms baselines across nearly all measured editing metrics and base models The method narrows the trade-off between editing effectiveness and generalization, with especially strong gains on LLaMA3 Several baselines lose generalization or consistency dramatically during sequential editing, while AlphaEdit maintains stronger overall retention

Experiments evaluate sequential knowledge editing across multiple base models and show that AlphaEdit achieves the strongest overall performance by improving update effectiveness and generalization while preserving generation quality. Compared with baselines, it better balances knowledge updates and preservation, with especially strong gains on LLaMA3. The results indicate that maintaining fluency and consistency is central to its advantage, whereas several baselines degrade sharply during sequential editing.