HyperAIHyperAI

Command Palette

Search for a command to run...

LLM
Transformer

가중치 공간 프로그래밍에는 트랜스포머가 필요하지 않다

Hyunjong Ok Jaeho Lee

초록

트랜스포머와 같은 표준 신경망 구조는 경사 하강법으로 잘 학습되고 GPU에서 빠르게 실행되기 때문에 선택되었다. 가중치 공간 프로그래밍에서는 대규모 언어 모델(LLM)이 경사 기반 학습 없이 목표 가중치를 직접 작성하므로, 그러한 이유는 더 이상 적용되지 않는다. 그렇다면 트랜스포머를 유지해야 하는가? 최소한의 지침, 10개의 레이블된 예시, 그리고 주로 객관식 문제로 구성된 샌드박스 환경에서, 세 개의 최첨단 LLM에게 고정된 구조의 가중치를 생성하거나 구조 자체를 설계하도록 과제를 부여했다. 자유롭게 설계할 수 있게 하자, 세 모델 모두 일반 텍스트 사실을 고정된 절차로 조회하는 희소 테이블로 컴파일하여 최대 54.4%의 테스트 정확도에 도달했다. LLM이 설계한 모델은 동일한 LLM이 작성한 가장 정확한 공개 트랜스포머보다 약 1,800개 적은 파라미터를 사용하면서 더 높은 테스트 정확도를 달성했다. 경사법이 없을 때 이들은 테이블을 선택하는데, 이 테이블에서 LLM은 각 가중치의 의미를 명시할 수 있고 잘못된 가중치를 찾아 다시 작성할 수 있다. 각각의 오답은 하나의 행을 가리키며, 한 행을 제거하면 약 하나의 다른 답이 변경된다. 이로부터 우리는 동일한 크기의 트랜스포머보다 GPU에서 최대 17배 빠른 연상 메모리 기반 모델을 제안한다. 또한 사전 학습된 가중치, 다른 벤치마크, 모델이 답을 생성하는 과제를 테스트했다. 사전 학습된 LLM이 여전히 더 정확하며, 본 주장은 가중치를 작성할 때 어떤 구조를 구축할 것인지에 관한 것이다.

One-sentence Summary

Researchers at Pohang University of Science and Technology (POSTECH) show that in weight-space programming without gradients, frontier LLMs favor sparse associative-memory tables over Transformers, with an LLM-designed model reaching up to 54.4% test accuracy and using about 1,800 fewer parameters than the most accurate released Transformer it wrote, while a proposed associative-memory model runs up to 17 times faster on GPUs than a same-sized Transformer.

Key Contributions

  • Frontier LLMs allowed to design models without gradient training converge on sparse tables queried by a fixed procedure, reaching up to 54.4% test accuracy; an LLM-designed table outperforms the most accurate released Transformer written by the same LLM while using about 1,800 fewer parameters.
  • The observed table structure motivates an associative-memory-based model that runs up to 17 times faster on GPUs than a same-sized Transformer, with row-wise interpretability where each wrong answer points to one row and removing a row changes about one other answer.
  • Additional evaluations with pretrained weights, other benchmarks, answer-generation tasks, and low-label warm-starting show that a written network initialized with 100 labeled AG News articles reaches up to 84.7% accuracy versus 63.6% from random initialization, while pretrained LLMs remain more accurate overall.

Introduction

The authors study weight-space programming (WSP), where a large language model writes the weights of a separate model directly, without gradient-based training or data fitting. This matters because Transformers are highly effective for gradient optimization on GPUs, but that advantage may not transfer when weights must be written explicitly. Prior work on writing or compiling weights typically fixes a target architecture and asks what can be written into it, leaving open whether the Transformer is the right structure for WSP. The authors let frontier LLMs either write weights for released architectures or design a target architecture from scratch. They find that LLMs consistently avoid Transformers and instead build sparse table lookup models that store facts as rows and score answers through fixed retrieval. The authors analyze why this design is easier to write without gradients, and they introduce an associative-memory-based variant that reads only a fixed number of rows per question, retaining accuracy while improving GPU inference speed.

Method

The authors investigate Weight-Space Programming (WSP), a paradigm where Large Language Models (LLMs) write model weights directly without gradient descent. In this setup, the LLM operates in a loop: it creates practice questions, runs the model, inspects failures, and adds rules or fact rows. This process requires traceable answers and local edits, properties naturally satisfied by sparse structures rather than dense neural networks.

When allowed to design architectures from scratch, the LLMs converge on a specific design: a sparse fact table queried by a fixed procedure. This model maps words into concepts and matches the question and each candidate option against a sparse fact table. It retains the top-matching rows per option, runs a second pass expanded with concepts from the top facts, and applies rules for negation and word order.

As shown in the figure above, the authors compare a standard two-layer Transformer, the LLM-designed model, and their proposed associative-memory-based model. While the Transformer relies on dense matrices in its attention and MLP stacks, the LLM-designed model utilizes sparse concept vectors. The authors frame this through the lens of associative memory. A Transformer recalls values viv_ivi​ whose keys kik_iki​ resemble a query xxx via:

f(x)=∑i=1Mviσ(ki⊤x)f(x) = \sum_{i=1}^{M} v_i \sigma(k_i^\top x)f(x)=i=1∑M​vi​σ(ki⊤​x)

where σ\sigmaσ is a selection function like softmax. In contrast, the LLM-designed model instantiates this memory with sparser choices. Each memory iii is a written fact with a sparse concept vector kik_iki​ serving as both key and value. The score for an option ccc is computed by matching the option concepts oco_coc​ against the best-supported fact:

sc=max⁡i(ki⊤q)(ki⊤oc)s_c = \max_i (k_i^\top q)(k_i^\top o_c)sc​=imax​(ki⊤​q)(ki⊤​oc​)

where qqq is the question's concept vector.

The figure above illustrates this process with an example question. In the LLM-designed model, the answer traces directly to specific written facts, such as "Hollow bones make birds light so they can fly," which yields a high support score. However, this design scores every stored fact for every question, causing inference time to grow with the number of facts and launching many GPU kernels.

As demonstrated in the figure above, most LLM-designed models run slower on a GPU than a same-sized Transformer due to the high number of kernel launches per question. To address this, the authors propose an associative-memory-based model that preserves line-by-line writing while fixing the forward pass to read only a fixed number of rows per question.

This model compiles written lines into two tables: a Concept Table AAA (first-order memory) and a Pair Table (second-order memory). The Pair Table stores concepts recalled by a pair of concepts (a,b)(a, b)(a,b), mapping directly to a row address h(a,b)=(ap+b)(modH)h(a, b) = (ap + b) \pmod Hh(a,b)=(ap+b)(modH) for a fixed prime ppp and table height HHH. From the question's concept vector qqq, the J=16J=16J=16 strongest concepts form a set of 120 pairs. The recall vector rrr and option score scs_csc​ are computed as:

r=∑(a,b)∈P(q)qaqbth(a,b)+μA⊤q,sc=oc⊤rr = \sum_{(a, b) \in P(q)} q_a q_b t_{h(a, b)} + \mu A^\top q, \quad s_c = o_c^\top rr=(a,b)∈P(q)∑​qa​qb​th(a,b)​+μA⊤q,sc​=oc⊤​r

where tht_hth​ is the row of the pair table and μ\muμ is a mixing weight. This design keeps the per-question cost independent of the fact count, allowing the model to answer significantly faster than a same-sized Transformer while maintaining high accuracy.

Experiment

The experiments evaluate weight-space programming on 3,530 ARC multiple-choice science questions, where three frontier LLMs write models under either fixed architectures or LLM-designed structures, with lineage refinement in some runs. Across all three LLMs, LLM-designed models outperform fixed architectures and gradient descent from random initialization, and they rely on interpretable key-value lookup or associative-memory tables whose predictions trace mainly to single facts and whose edits remain local. Further tests show these written models transfer to similar science questions, text generation, classification, and multi-task settings, while associative-memory designs improve GPU efficiency and pretrained checkpoints are usually edited lightly or overwritten only when near chance. Pretrained fine-tuning still remains far more accurate, and LLM-designed models are generally slower than same-sized Transformers.

Gradient descent sets weights jointly with an optimizer on a GPU, requiring a differentiable model, an architecture built for gradient flow, and a large training set, while weight-space programming writes weights line by line on a CPU using an LLM and none of those requirements. The written weights are traceable to readable text, and the reported LLM-written ARC accuracies are around 50 to 54 percent across generations and designs. Results on other tasks show similar mid-range performance on SciQ, near chance on raw-pixel CIFAR-10, and stronger performance when meaningful semantic token features are available. Weight-space programming removes the need for differentiability, gradient-flow architectures, and large training sets, and it makes weights traceable to human-readable text. LLM-written models reach ARC accuracy around 50 to 54 percent and stay near chance on raw-pixel CIFAR-10, but improve sharply when using semantic token features.

The comparison frames trained Transformers, LLM-designed models, and the associative-memory-based model as different instantiations of the same associative memory equation. They differ mainly in memory unit, key space, selection mechanism, how memory is set, and per-question cost. The written models use sparser and more interpretable memory units than the dense learned vectors of a trained Transformer. The trained Transformer uses dense learned vectors and smooth selection, with per-question cost fixed by model width. The LLM-designed model uses written sentences as sparse memories with top-k row selection, while the associative-memory-based model uses concept and pair indices with direct addressing.

On ARC, letting the LLM design the model in weight space yields better accuracy than writing a fixed Transformer or released architecture, while using far fewer parameters. The associative-memory-based model also outperforms fixed architectures, and iterative refinement improves accuracy further. Pretrained fine-tuned models remain substantially more accurate, leaving a wide gap for weight-space programming. LLM-designed models achieve higher ARC accuracy than fixed architectures written from random initialization, despite having far fewer parameters. Gradient descent from random initialization leaves released architectures near random guessing, while pretrained fine-tuned models remain far ahead.

Forbidding practice questions costs only about one point of ARC accuracy, so the models are not relying heavily on self-written test-like questions. Forbidding fact rows costs about five points and forces over 100 times more non-zero values, showing the row-table structure is more important. Scoring each option from its single best-matching fact nearly reproduces full accuracy, indicating answers trace mainly to one stored fact. Practice questions forbidden: accuracy drops only about one point. Fact rows forbidden: accuracy drops about five points while non-zero values increase by over 100 times. Using only the single best-matching fact per option nearly matches the full model.

Removing one written line causes only small, localized answer changes in the table-style models, averaging roughly 0.5 to 1.8 changed answers across the LLM-designed and associative-memory-based runs. Written Transformer runs are more sensitive, with one run changing several times more answers and another changing nearly all weights and a large share of answers because its vector codes draw from one random stream. Test accuracy remains largely stable after line removal, and changed answers in the table-style models tend to share wording with the removed line. LLM-designed and associative-memory-based table models remain relatively stable after a line is removed, with only about 0.5 to 1.8 answers changing on average. Written Transformer runs show much larger disruption, including one run where a removed line changes nearly all weights and a large fraction of answers. In table-style models, most changed answers share a word with the removed line, while written Transformer changed answers share wording less consistently. Overall test accuracy before and after line removal stays essentially unchanged across most runs.

The paper evaluates weight-space programming, where an LLM writes model weights directly, against gradient-trained and fixed architectures on ARC and auxiliary tasks. LLM-written and associative-memory table models reach mid-range ARC accuracy with far fewer parameters and interpretable sparse memories, while raw-pixel CIFAR-10 stays near chance unless semantic token features are available. Ablations show performance depends primarily on stored fact rows rather than practice questions or multi-fact scoring, and removing individual lines causes only small localized answer changes in table models, unlike more brittle written Transformers. Pretrained fine-tuned models remain substantially ahead, leaving a wide gap for weight-space programming.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp