One-sentence Summary

Researchers from Zhongguancun Academy and Zhongguancun Institute of Artificial Intelligence present ZGCM-1, a fully open $7\text{B}$7B dense foundation model trained from scratch with extreme efficiency that couples deliberate internal thinking with external tool use via interleaved gated sliding-window attention, $\text{FP8}$FP8 Muon, and MDP-based mid-training, achieving competitive mathematical reasoning and agentic search performance against frontier models orders of magnitude larger while offering a $\sim 4.2\times$∼4.2× pre-training efficiency gain.

Key Contributions

• Introduces ZGCM-1, a fully open 7B dense foundation model trained from scratch, coupling chain-of-thought reasoning with external tool use to overcome parametric capacity limits; achieves first-place average performance across 14 reasoning benchmarks at the 7B–8B scale, including 75.0% on AIME 2026 and 97.1% on MATH-500.
• Presents an efficient training recipe with a hybrid gated sliding-window and global attention architecture (5:1 ratio), reducing KV-cache footprint by 6.4× and delivering a 3.94× throughput speedup at 256K context, plus a stable FP8 Muon optimizer and TWEO outlier regularization yielding a ~4.2× pre-training time-to-loss speedup over an AdamW/BF16 baseline.
• Develops a progressive curriculum with MDP-based mid-training, scaling context from 16K to 256K across 600B tokens while reformulating interaction traces into Markov Decision Process transitions; the model remains competitive with frontier models orders of magnitude larger on agentic search tasks, achieving 63.1% on WebWalkerQA and 19.4% on BrowseComp.

Introduction

Foundation models are advancing rapidly, but two practical bottlenecks hinder research: the scale barrier, where frontier-grade intelligence is assumed to require hundred-billion-parameter systems, and the opacity barrier, where competitive models are released as open-weight rather than fully open-source, leaving training recipes and agent traces as proprietary black boxes. Prior work offers limited transparency and compute accessibility, preventing systematic study of training dynamics and capacity limits.

The authors introduce ZGCM-1, a 7.39B dense foundation model trained from scratch under an open-science paradigm, challenging the notion that advanced intelligence requires massive parameter counts. Their core thesis is that compact models, though bounded by static parametric capacity, can transcend this via a dual engine of deliberate internal thinking and active external seeking, coupling long-horizon chain-of-thought reasoning with autonomous tool use for web evidence gathering, terminal interaction, and binary program analysis. To make training tractable on academic budgets, they build an efficient full-stack pipeline with hybrid attention (gated sliding-window and global attention at a 5:1 ratio, reducing KV-cache footprint by 6.4x and delivering 3.94x throughput speedup at 256K context), system-algorithm co-design (Muon optimizer, hybrid FP8 precision, TWEO outlier regularization, achieving 4.2x pre-training time-to-loss speedup), curriculum mid-training with MDP supervision across 600B tokens, and execution-grounded alignment with mixed think/no-think fine-tuning. Evaluations across 20 benchmarks show ZGCM-1-7B ranks first on average across 14 reasoning benchmarks at the 7B-8B scale, scoring 75.0% on AIME 2026, 97.1% on MATH-500, and 70.4% on HMMT 2025, while also contending with frontier models on agentic tasks like WebWalkerQA (63.1%), BrowseComp (19.4%), and Binary Function Search (62.0%). The authors also integrate AI-native R&D agents throughout the development lifecycle, releasing model weights, intermediate checkpoints, training code, data recipes, and evaluation harnesses to support community research.

Dataset

The authors assemble a multi-stage training corpus with distinct data recipes for general pre-training, mid-training, and post-training. Below is a breakdown of the dataset composition, sources, processing, and usage.

General Pre-Training (Stage 1 and Stage 2)

• The authors first search for optimal data mixtures using a 0.3B-parameter proxy model trained on roughly 30B tokens, iterating based on loss and evaluation signals for knowledge, code, and mathematics.
• The final General Pre-Training corpus contains approximately 0.99T tokens in Stage 1 and 3.20T tokens in Stage 2. Web data is the largest source in both stages; Stage 2 increases the relative share of code and mathematics, introduces a specialized reasoning component, and retains academic, OCR, and LaTeX content.
• Web data: Curated English web corpora are the main source, supplemented by a controlled allocation of Chinese web data. The authors retain upstream document-level quality scores and quality strata from validation-driven filtering, then normalize text and language metadata into a common document representation.
• Academic and OCR data: The authors combine educationally filtered PDF views with OCR-derived scientific content from Ai2, produced via the olmOCR pipeline. For arXiv papers, an in-house pipeline extracts and cleans text while preserving technical content.
• Code data: The code mixture includes code-rich web pages and open-source repository files. Structured releases are converted into text views, file-level metadata is normalized, and repository artifacts and non-source content are cleaned while preserving programs, technical documentation, and explanatory code text.
• Mathematics data: Higher-quality tiers from mathematical corpora are prioritized, along with classifier-filtered web mathematics and textbook collections following the SmolLM2 math recipe. Processing combines heuristic cleaning, quality-model selection, and formula-preserving normalization so LaTeX expressions remain embedded in their reasoning context, as in Proof-Pile-2.
• LaTeX papers: The arXiv LaTeX-source slice of RedPajama-1T is combined with filtered recent TeX sources. Accepted views recover the main textual stream, filter malformed or content-poor documents, and preserve equations and scientific structure.
• Specialized reasoning data: Reasoning-oriented material is kept as a separate source family. The authors draw on Nemotron-Pretraining-Specialized-v1, validate source schemas and versions, select designated high-quality reasoning views, and admit them to Stage 2 as an independently controlled mixture component.
• Each source family undergoes source-specific language and quality filtering, text extraction, or repository cleaning. Accepted documents are normalized into a common record schema, materialized as versioned shards, and registered in source manifests. The shards are tokenized and indexed with the GLM-5.1 tokenizer. Cross-stage deduplication excludes all Stage-1 content from Stage-2 selection before sampling.

Mid-Training (Capability-Oriented Continued Pre-Training)

• The authors apply pool-wide deduplication to the candidate pool, yielding 2.86T tokens, from which they construct a 600.51B-token sampled schedule organized into 16K, 64K, and 256K context stages.
• Maximum sequence length increases progressively across stages, following the multi-stage context expansion of Qwen2.5-1M. Each later stage is cumulative: the 64K stage contains 180.89B tokens at up to 16K and 59.11B tokens in the 16K-64K range; the 256K stage combines 127.81B tokens at up to 16K, 21.72B tokens in the 16K-64K range, and 30.98B tokens above 64K.
• The 16K stage trains at standard context length with a balanced mixture of code, mathematics, knowledge, and reasoning. The 64K stage introduces longer documents and reasoning sequences. The 256K stage adds ultra-long documents, cross-document information, and long-horizon agentic trajectories, while replaying data from shorter length ranges.
• Code and mathematics each remain close to 20% throughout the curriculum. As context length grows, knowledge data increases from 10.50% to 14.23%, and agentic data increases from 1.50% to 3.30%. Pre-training replay decreases from 13.00% to 9.00%. Web, QA, reasoning, and instruction data remain comparatively stable.
• Reasoning data: Sources are normalized to a common schema and routed into accepted, rewrite, pending, holdout, or rejected views based on self-containment, answer evidence, and reasoning value. Valuable questions with unsuitable original traces are extracted as standalone problems, reconstructed or improved by teacher models, and admitted only after checks for parsing validity, completeness, leakage risks, duplication, and trainability.
• Instruction data: The corpus includes Tulu- and FLAN-derived data converted into pre-training-compatible raw context. The pipeline preserves legitimate short-label tasks, routes translation-risk subsets through model review, and rewrites selected single-turn examples as multi-turn discussions.
• Agentic data: Covers general interaction traces and software-engineering trajectories. General traces are reformulated as Markov decision process-style state-conditioned next-action prediction examples. Software-engineering trajectories undergo execution-aware filtering and preserve interaction context for planning, tool use, and iterative refinement.

Supervised Fine-Tuning (SFT)

• The SFT corpus combines open-source datasets with internally distilled data, containing 4,921,933 examples organized into general and agentic data. All examples are normalized to a common message and tool schema, then filtered for structural validity, response correctness, and tool-turn consistency.
• General data: Covers instruction following, knowledge, mathematics, science, code, dialogue, and reasoning. A multi-stage curation pipeline uses deterministic heuristic rules followed by a model-based evaluator calibrated against human-annotated benchmarks. Examples are graded on educational value, logical soundness, reasoning coherence, factual consistency, and safety compliance, then classified into discrete quality tiers. Aggressive tiered filtering, pruning roughly 50% of raw candidates, improved the six-benchmark mean from 67.78 to 68.83, with gains concentrated in reasoning.
• Decontamination uses an 8-gram matching sliding window; any sample with greater than 50% 8-gram overlap with evaluation prompts or reference responses is purged.
• Mixture ablation sweeps reveal that disproportionately scaling long chain-of-thought (CoT) trajectories introduces verbosity bias. The authors calibrate proportions of long-form reasoning, direct-response QA, and strict formatting directives. The final corpus interweaves think examples (explicit reasoning chains in structured tags) with no-think examples (concise immediate responses).
• Agentic data: Comprises three branches: deep research, software engineering, and terminal interaction. Each task family aligns its training schema with its downstream inference environment, including system instructions, message roles, tool definitions, structured calls, and observation placement. Tool-protocol validation converts source-specific formats into the target structured representation, validates argument structure and call-observation pairing, and discards irrecoverable examples.
• Deep research: 20,217 multi-step research trajectories normalized into a shared interaction environment with search and visit tools, provided in aligned think and no-think forms.
• Software engineering: 30,014 execution-grounded and 30,000 execution-free trajectories, filtered and scored, then projected to the GLM-5.1 structured-tool format.
• Terminal: Trajectories collected from terminal task environments in isolated Docker environments with persistent shells and structured bash tools, deduplicated at the task-environment level. A paired-source ablation view contains 22,309 trajectories and a 15,748-trajectory verifier-successful subset.
• A joint schedule interleaving general and agentic examples outperforms a sequential schedule and is used for the final training run.

Reinforcement Learning (RL) Data

• The RL corpus covers mathematics, code, and general-capability tasks. Prompt difficulty is estimated through rollouts from the initial policy, and problems the model already solves frequently are removed to concentrate training on prompts that provide useful learning signals.

Method

ZGCM-1 follows a decoder-only Transformer architecture with approximately 7.39 billion parameters, utilizing Grouped-Query Attention, RMSNorm, SwiGLU activation, and Rotary Position Embedding. The model features a hybrid causal-attention backbone that interleaves gated sliding-window attention with global attention at a 5:1 local-to-global ratio. Of the 32 Transformer layers, 27 employ gated sliding-window attention with a 128-token window, while five utilize global causal attention placed at specific intervals to allow information flow beyond the local window.

Refer to the framework diagram:

For a normalized hidden state $h$h, the gated sliding-window attention module forms query, key, value, and gate projections in parallel. RMS normalization and Partial RoPE are applied to the query and key branches. The gated attention output is computed as:

where $A _ {\mathrm{SWA}} (q, k, v)$ASWA​(q,k,v) denotes the 128-token sliding-window Grouped-Query Attention output, and the learned sigmoid gate modulates the local-attention output element-wise before the output projection. This hybrid design significantly reduces the key-value cache footprint at inference time, dropping the per-token footprint from 128 KiB to 20 KiB and enabling efficient processing of up to 256K tokens.

The pre-training process consists of two consecutive phases: General Pre-Training and Mid-Training.

As shown in the figure below:

General Pre-Training builds broad language, knowledge, mathematics, and code capabilities across two data stages. Stage 1 utilizes 0.99T tokens with a curriculum pretraining phase where general-language documents are ordered by lexical complexity, while code and mathematics are interleaved independently. Stage 2 expands the corpus to 3.20T tokens, increasing the relative contribution of code and mathematics and introducing specialized reasoning data. Mid-Training then retains the full-sequence causal language-modeling objective while introducing denser reasoning, instruction, and agentic data, progressively extending the context from 16K to 64K and 256K.

The data mixture composition across these stages is carefully balanced to optimize capability coverage and loss-convergence efficiency.

Refer to the data mixture charts:

Web data remains the largest source throughout General Pre-Training. During Mid-Training, code and mathematics each remain close to 20 percent to preserve core programming and reasoning capabilities. As the context length grows to 256K, the density of knowledge and agentic data increases to support long-document and multi-step task examples, while pre-training replay is maintained to preserve broad coverage.

For agentic data construction, the authors leverage an AI-driven self-iterating governance process to develop and validate source-specific processing recipes.

As illustrated in the governance pipeline diagram:

This pipeline routes and samples each source family independently, processing them through source-specific rule filtering, strong-model audits, failure mining, human spot-checks, and iterative script revision before accepted examples enter the validated reasoning-data pool.

The post-training pipeline consists of supervised fine-tuning followed by mixed reinforcement learning. The supervised fine-tuning phase establishes response modes using a unified think and direct-response supervision objective, interleaving explicit reasoning chains with concise responses to equip the model to dynamically toggle between full deliberation and efficient zero-shot responses. The reinforcement learning phase utilizes Group Relative Policy Optimization with domain-specific rewards, combining outcome-based rewards with reference-policy KL regularization and a mild length penalty to stabilize long-horizon optimization.

Experiment

Experiments on attention variants showed that SWA 5:1 matched full-attention loss while achieving the highest throughput, with its speed advantage widening from 1.13x at 4K context to 3.94x at 256K. The hybrid architecture reduced KV cache memory 6.4x over full attention at 256K context, directly improving decode efficiency. Training monitoring revealed heterogeneous capability development across domains and identified a throughput degradation issue that was resolved by periodic CUDA cache clearing. Evaluation of the 7B model across 20 benchmarks and two agentic settings showed competitive performance, particularly in binary function search where it reached 62% accuracy versus 12% or less for similarly sized baselines. Contributor assessments of AI autonomy in the R&D lifecycle found operational tasks reaching L4 autonomy while design tasks remained at L2.

The supervised fine-tuning corpus is predominantly composed of general data, covering areas like instruction following, knowledge, and reasoning, with a small agentic portion for tool and environment interactions. A tiered curation pipeline filters for quality, and decontamination removes samples overlapping with evaluation sets. The data quality emphasis is supported by findings that pruning candidates can improve benchmark performance. General data makes up over 96% of the corpus, with agentic data under 4%. The curation pipeline uses rule-based checks and model-based scoring to assign quality tiers. Decontamination removes any sample with more than 50% 8-gram overlap with evaluation data. Pruning half of the candidate pool improved mean benchmark performance, especially in reasoning tasks.

ZGCM-1-7B shows competitive or superior performance across many reasoning and general benchmarks compared to other 7B-8B scale models, particularly excelling in MATH-500 and HMMT 2025. Its strengths are most pronounced in reasoning-heavy tasks, while some models outperform it on specific AIME subsets. ZGCM-1-7B achieves the best score on MATH-500 among all compared models. ZGCM-1-7B leads on HMMT 2025 and HMMT 2026, with notably higher scores than most peers. On AIME 2024 and AIME 2025, ZGCM-1-7B trails DeepSeek-R1-0528-Qwen3-8B and MiniCPM4.1-8B but still outperforms several other baselines. The largest performance gaps in favor of ZGCM-1-7B appear on HMMT benchmarks, where it exceeds the second-best model by a clear margin.

ZGCM-1-7B achieves strong performance on web-environment deep-research benchmarks, leading on WebWalkerQA and BrowseComp while remaining competitive on GAIA text-only. In binary function search, it reaches 62% accuracy, far surpassing similarly sized baselines and approaching larger frontier models. ZGCM-1-7B achieves the best WebWalkerQA score among all compared systems, including specialized research agents and larger open-weight models. On BrowseComp, ZGCM-1-7B outperforms all open-source baselines and is second only to Kimi-K2. For GAIA text-only, ZGCM-1-7B is competitive with specialized agents and open-weight models, though Kimi-K2 leads. In binary function search, ZGCM-1-7B reaches 62% accuracy, far exceeding similarly sized baselines (12% for Qwen3-8B, 0% for others) and approaching GLM-5.1's 66%.

The table evaluates binary function search accuracy across several models, with top performers achieving 76% accuracy. Smaller models like ZGCM-1-7B show competitive performance, surpassing similarly sized baselines and approaching larger frontier models. DeepSeek-V4 Flash and Qwen3.5-397B-A17B tie for the best accuracy at 76%. GLM-5.1 achieves 66% accuracy, while ZGCM-1-7B and Kimi-K2 both reach 62%. ZGCM-1-7B outperforms similarly sized baselines by a large margin, with accuracy far exceeding Qwen3-8B and other small models. DeepSeek-R1 lags behind with only 36% accuracy despite being a larger model.

The table defines a five-level rubric for AI autonomy in R&D workflows, ranging from basic assistance to full autonomy. Ratings from nine contributors across 11 task categories show that autonomy is uneven and task-dependent, with operational tasks reaching higher levels than design tasks. The assigned levels cluster between L2 and L4, and no task is rated above L4, indicating that full autonomy is not yet demonstrated. Operational tasks like experimentation and deployment engineering reach L4, while design tasks such as model architecture and learning algorithm design remain at L2 and never exceed L3. Only one of the 99 ratings is L5, and no task is assigned above L4, so the workflow's autonomy is better described per task than as a single overall level. The L4/L5 boundary is defined by autonomous objective formation: L4 works within human-defined objectives, while L5 also identifies research objectives and coordinates across stages.

The evaluation setup combines a tiered data curation pipeline with benchmark tests across reasoning, general knowledge, and agentic tasks. Results show that pruning half of the candidate corpus improves mean benchmark performance, especially on reasoning tasks, and that ZGCM-1-7B is competitive or superior to similarly sized models on reasoning-heavy benchmarks like MATH-500 and HMMT, while trailing some peers on specific AIME subsets. In web-environment deep-research tasks, ZGCM-1-7B leads on WebWalkerQA and BrowseComp, and its binary function search accuracy (62%) far exceeds similarly sized baselines, approaching larger frontier models. A separate autonomy rubric assessment indicates that R&D workflow autonomy is task-dependent, with operational tasks reaching L4 but design tasks remaining at L2-L3, and no task exceeding L4, suggesting full autonomy is not yet achieved.