One-sentence Summary

NVIDIA researchers introduce Agora, a Git-based append-only directed acyclic graph (DAG) that serves as shared memory for multiple autonomous language-model agents, recording every claim, insight, hypothesis, verification, and report as immutable, checkable commits with a diversity-aware selection rule to prevent monoculture; in a nearly 12-day run with 13 workers and no central planner, it improved a frozen $119.6\text{M}$119.6M attention-SSM hybrid's evaluator from $3.39$3.39 to $1.899$1.899 bits per byte, closing $62\%$62% of the gap to GPT-2 124M, with all $165$165 independent reproductions succeeding.

Key Contributions

• Agora is introduced as a shared memory system that stores autonomous research outputs in an append-only directed acyclic graph in Git, where each claim, result, insight, hypothesis, verification, and report is an immutable commit with parent edges, enabling any participant to check out and rerun every contribution.
• Agora provides a derived index exposing the research frontier, neglected branches, and verification status of each claim, plus a diversity-aware selection rule that prevents the community from collapsing onto a single leader, converting parallel duplicated search into coordinated exploration.
• In a sustained 12-day deployment with 13 language-model workers and no central planner, the system produced 1,703 contributions that improved a weight-transfer task from 3.39 to 1.899 bits per byte, closing 62% of the gap to a trained GPT-2 124M, with 165 independent reproductions and none failing; the winning recipe’s 145-commit ancestry spanned 15 accounts.

Introduction

AI research agents often operate in isolation: a session runs code, reads papers, and launches experiments, but its insights remain trapped in transcripts or temporary worktrees. The next session lacks knowledge of which learning rates diverged, which branches failed, or which results still await independent reproduction. Scaling up with more agents worsens this, causing duplicate searches, premature convergence, and wasted effort in reconstructing past actions. Existing multi-agent frameworks organize conversations or role-specific workflows for single tasks, but they lack a durable state that outlives any worker, such as a public frontier, immutable lineage, negative results, and independent verification.

To address this, the authors introduce Agora, a coordination layer that frames research as a directed acyclic graph (DAG) where every contribution is a Git commit and each parent edge signifies "builds on." Git provides immutable, content-addressed artifacts, while a database offers searchable views, making claims, dependencies, verification status, and untried alternatives visible. This graph serves as the sole state for coordination, enabling a mixed community of humans and agents to align without dictating a single workflow. In a 12-day experiment with 13 coding-agent sessions, Agora supported work on initializing a frozen hybrid language model from pretrained donors, achieving 1.899 bpb from a random baseline of 3.39, with 165 reproductions and a quick recovery from a five-day monoculture after a single human-provided map. The system separates immutable storage from downstream evidence and diversity-aware attention, offering a new approach to scaling research communities.

Dataset

The authors construct a dataset primarily from a "donor zoo" of 141 open-weight models, totaling 534 GB, sourced from 32 architecture families. These families include GPT-2, LLaMA, Mistral, Qwen, Gemma, Pythia, RWKV, and Mamba. The target model for weight transfer is a 14-layer hybrid architecture that alternates multi-head attention blocks with simplified Mamba-style state-space model (SSM) blocks, featuring a hidden size of 672, seven attention heads, untied embeddings, and 119,572,320 parameters. The authors deliberately chose these dimensions so that no donor model matches any of them, ensuring a novel target configuration.

For evaluation, the authors use 200 texts from the FineWeb-Edu dataset, which are scored in non-overlapping 512-token chunks under the GPT-2 tokenizer. The evaluation metric is summed next-token loss divided by UTF-8 byte count, reported as bits per byte (bpb). The dataset handling includes a strict loader that raises an error if called from inside the transfer function, preventing direct use of the training corpus during weight transfer. Random initialization yields a baseline score of 3.3923 bpb, while a conventionally trained GPT-2 124M model scores approximately 1.0 bpb, setting the scale for comparison; the aspirational target is below 2.5 bpb.

The data is used in a weight-transfer task where a participant submits a Python file with a transfer(model, config) function. This function receives the randomly initialized target and returns it with new weights derived from the donors, using only the donors' weights and forward passes. The evaluator seeds all random-number generators with 42, runs the transfer, and then scores the target on the FineWeb-Edu texts. No gradient updates on the target are permitted, and the rules forbid pretraining, fine-tuning, or editing the evaluator or target configuration. Two runs of the same code on identical hardware are bit-identical, though scores may vary in the third decimal place across different GPU types.

Method

The authors design Agora as a coordination substrate that models project state as a directed acyclic graph $G = (V, E)$G=(V,E). In this framework, an edge $(u, v) \in E$(u,v)∈E indicates that contribution $v$v builds upon $u$u. Each node $v$v stores a tuple $(h, a, T, d, x, m, P, \tau)$(h,a,T,d,x,m,P,τ), representing the canonical commit hash, publishing account, tags, description, structured metadata, optional project metric, parent set, and server timestamp. Because identity relies on hashes and parentage follows Git semantics, the history remains append-only and acyclic by construction.

Participants publish through two distinct paths. Metadata-only work uses a light path where the client sends JSON and the server creates the commit. Code-bearing work uses a heavy path where the participant commits locally, uploads a Git bundle, and the server validates it before creating a canonical commit. Both paths yield identical node types for lineage queries.

The prototype implementation consists of a Go service with a command-line client and a Next.js web interface. Each project owns a bare repository, and a SQLite index derived from the Git history supports queries for agents, projects, contributions, and embeddings. The system exposes HTTP routes and CLI commands for reading lineage, DAG structure, and analysis views, while writes require bearer authentication and adhere to rate limits.

To evaluate contributions, the authors define an evidence score $S(u)$S(u) as the weighted count of downstream work built by other accounts:

where $w(v)$w(v) is a tag-dependent weight. This score emphasizes work that others have reproduced or extended, excluding self-citations to prevent manufactured impact.

To prevent the community from collapsing into a monoculture around a single leaderboard, the analyze call returns multiple views, including metric leaders, leaves, and underexplored results. Once embedding coverage is sufficient, the service performs single-link clustering over descriptions. Candidates are then ranked using a diversity-aware upper-confidence bound:

where $Q(v)$Q(v) is a quality percentile, $n(v)$n(v) counts follow-on work, and $\rho(v)$ρ(v) counts near-duplicate descriptions. This ranking splits candidates into exploit, explore known, and explore novel slots, guiding participants toward diverse research directions.

The authors applied this framework to a weight-transfer task involving 141 open-weight donor models and a 14-layer hybrid target architecture. The workers were coding-agent sessions running frontier language models. Each session operated in a container with GPU access, following a loop: reading the analyze output, picking a parent, fetching the commit, making a change, evaluating, and pushing the result.

The run sustained roughly 170 contributions per day once all workers were active. The 1,703 contributions included scored results, insights, hypotheses, and verifications. The winning recipe emerged from this iterative process, utilizing a two-stage algorithm. Stage A builds initialization from donor predictions by blending next-token log-softmaxes into a context-averaged bigram table, which is then factorized via randomized SVD to populate the input embedding and output head. Stage B re-enables sublayers with sparse deterministic edits on 96-dimensional bands of the hidden state, configuring attention layers as uniform causal mean-pools and SSM blocks as gated depthwise causal convolutions. Each constant in this architecture was introduced as a single change and retained based on evaluator improvement.

Experiment

The evaluation involved a community of agents collaboratively improving a frozen hybrid model's weight initialization without training data, achieving a substantial score improvement that closed 62% of the gap to a trained reference. The run demonstrated fast initial gains, a narrow lineage of successful contributions, and effective cross-worker reproduction, though it also showed duplicate work and an exploitation bias. Human intervention added diversity-aware tools that broke a leaderboard stall. The authors verified all claims by recomputing statistics from the raw trace and inspecting code at every step of the winning method, confirming no evaluation data was touched.

The table outlines key design goals for a research platform and maps each to a specific mechanism in Agora, such as durable memory via append-only Git commits and auditability via content-addressed artifacts. The context explains that Agora's mechanisms aim to address failures like duplicate work and popularity-biased rewards by promoting frontier visibility, evidence quality, and search diversity. Agora uses append-only Git commits with explicit parent lineage to preserve negative results and session discoveries. Frontier visibility is supported through leaf, hypothesis, verification, cluster, and metric-landscape views to reduce duplication. Evidence quality is enforced by excluding self-citations and requiring independent follow-on work for verification. Search diversity is encouraged via semantic clusters and separate exploit/explore slots to avoid concentration on one local basin. Auditability relies on content-addressed artifacts and immutable revisions to trace results back to their code and configuration.

Reserved contribution types assign specific weights to tags based on their role in the research graph, with higher weights for reproducible experiments and lower or zero for endorsements. Verification receives differential weights depending on confirmation status, while endorsed contributions are visible but do not affect fitness scores. Setup and result contributions carry equal base weights, while verification can range from strongly positive to negative depending on outcome. Endorsed contributions are explicitly excluded from fitness calculations, making them non-scoring acknowledgments. Hypothesis contributions are restricted from presenting metric values as if already tested, preserving the distinction between proposed and confirmed results.

Stage-B re-enables sublayers by applying sparse, fixed edits to 96-dimensional bands of the hidden state, where the first band stores the leading singular directions from a factorization. Attention layers write back to their own band at small scales, while SSM layers use either a sign-alternating or uniform kernel and often write to additional bands. All edits are deterministic and were verified directly from the committed code. Attention layers read and write the same band with small scalar weights that vary by layer. SSM layer 1 uses a sign-alternating kernel emphasizing recent positions, while other SSM layers use a uniform kernel. Six SSM layers write filtered results into shared secondary bands, suggesting cross-layer information aggregation. All constants were traced through an import chain and confirmed to involve no gradient updates.

The best transfer initialization for a frozen 119.6M hybrid model achieved 1.899044 bpb, greatly improving over random initialization (3.3923) and closing 62% of the gap to a trained GPT-2. The search progressed through stages: initial worse-than-random copying, then unigram and bigram priors, with most gains coming from the bigram recipe under various prefixes. Later refinements and diverse strategies contributed small improvements, and the winning contribution is a community effort with many cross-account parent edges. Initial copying of GPT-2 and Mamba parameters scored worse than random (4.68 vs 3.39 bpb), and was published as a negative result. Using a unigram prior from GPT-2 predictions improved to 2.52 bpb, and extending to bigram statistics with up to 24 prefixes reached 1.93 bpb. Most of the score reduction (about 98%) came from the bigram-based contributions, with later changes adding only 0.03 bpb. Adding a second donor, widening contexts, and a power iteration in SVD contributed small gains, but some changes like doubling prefixes to 48 or using Pythia regressed and were published as negative results. The winning recipe was assembled by the community: the best contribution has 145 ancestors, with 115 of 144 parent edges crossing account boundaries, and 40 ancestors were independently reproduced.

The evaluation setup validates Agora's design goals, including durable memory, frontier visibility, evidence quality, search diversity, and auditability, through specific mechanisms like append-only Git commits, content-addressed artifacts, and reserved contribution weights that exclude endorsements from fitness scores. Stage-B re-enables sublayers via fixed, band-limited edits that were verified from committed code, with attention and SSM layers writing deterministically to specified bands. The transfer experiments show that a community-assembled bigram-based initialization for a frozen hybrid model dramatically improved over random initialization, closing 62% of the gap to a trained GPT-2, while copying parameters outright scored worse than random and was published as a negative result; the winning recipe, with 145 ancestors and many cross-account edges, was independently reproduced in 40 cases, with later refinements contributing only marginal gains.