One-sentence Summary

IQuest Research introduces IQuest-Q1, a 320B-parameter sparse Mixture-of-Experts agentic foundation model with 15B activated parameters, trained through Stable SplitNorm, Muon, and μP pre-training with a coding-heavy two-stage curriculum, domain-expanded mid-training with progressive context extension, and multi-scaffold post-training combining SFT, domain-specialized RL, and MOPD with conflict-mitigating model merging, and released open-source for agentic CLI, coding, and general tasks.

Key Contributions

• The paper introduces IQuest-Q1, an open-weight sparse Mixture-of-Experts agentic model with 320B total parameters and 15B activated per token, optimized for CLI interaction and released to support community research. Evaluations on DeepSWE v1.1, CyberGym, NL2Repo, JobBench, Agents’ Last Exam, and Humanity’s Last Exam report capabilities in coding, general agentic tasks, and reasoning.
• The paper presents a CLI-centric multi-stage training pipeline: stable pre-training with Stable SplitNorm, Muon, μP, and a two-stage curriculum; fine-grained mid-training with broadened data and extended context; and multi-scaffold post-training with SFT, domain-specialized reinforcement learning, and multi-teacher on-policy distillation. Model merging further combines experts and checkpoints from different stages while mitigating parameter-update conflicts.
• The paper demonstrates a human-on-the-loop recursive self-improvement process in which IQuest-Q1 participates in an internal model-development infrastructure, evaluates capabilities, and proposes improvements under human supervision, contributing to successor development and shortening the development cycle.

Introduction

LLM-based agents increasingly need to act through command-line interfaces to use tools, access files, and execute code, which makes CLI capability a core requirement for general-purpose and software-engineering agents. Prior work still faces challenges in reliable long-horizon CLI execution, robust tool use across different agent harnesses, and broad coverage from coding to cybersecurity tasks. The authors introduce IQuest-Q1, an open-weight sparse mixture-of-experts model with 320B total parameters and 15B active parameters per token, trained through a CLI-centric multi-stage pipeline that combines curriculum pretraining, long-context mid-training, and CLI-focused post-training with reinforcement learning and multi-teacher distillation.

Dataset

The authors describe dataset construction across pre-training, mid-training, and post-training.

• Pre-training corpus

  • Sources: web pages, public GitHub code, books, academic papers, news, and domain-specific materials.
  • Code data preserves diversity across programming languages, modalities, and task granularities.
  • Processing: unified pipeline with cleaning, quality filtering, exact deduplication, and fuzzy deduplication.
  • Usage:
    • Stage 1 builds foundational language and general knowledge with diverse corpora, maximizing unique token counts across domains.
    • Stage 2 increases code and STEM data, raising the code data percentage from 40% to 70% through a four-phase pipeline.
    • Mid-training combines high-quality natural corpora, QA pairs, and agentic trajectories.
    • Total pre-training and mid-training volume: 29T tokens.

• Agentic trajectory data for mid-training

  • Sources: selected trajectories used as behavioral anchors, plus pre-training documents to broaden technical and task coverage.
  • Construction: executable, containerized instances are built from compatible code repositories.
  • Scale: more than one million instances.
  • Filtering: an agent is rolled out on each instance and only valid, complete trajectories are retained for mid-training.

• Post-training task synthesis

  • General agent:
    • Sources: APIs and MCP servers from RapidAPI and Smithery, public skills from the web, and workspace files.
    • Coverage is guided by the Standard Occupational Classification.
    • Tasks and environments are refined through a solve-then-refine loop.
  • Reasoning agent:
    • Document-grounded pipeline expands a concept inventory into fine-grained scientific topics.
    • Documents are retrieved, curated, and filtered before being converted into tasks with reference answers.
  • Coding agent:
    • Sources: GitHub repositories, industrial data, PRs/issues, repository knowledge graphs, documentation, and cleaned de-identified usage logs.
    • Executable environments are built with coding-agent harnesses and domain-specific runtimes.
    • Verifiers combine agent-written tests, adapted repository tests, and checks from documented behavior.
    • Offline instances replace external dependencies with local fixtures or mocks.
    • An agent-judge loop checks executability, leakage, and consistency.
    • Reference solutions must pass while no-op solutions must fail.
    • Rubric-based trajectory review filters cheating and low-quality execution.
    • Repeated agent runs estimate difficulty; overly easy instances are enriched with additional skills or constraints.
    • Construction modes include building new environments from raw sources and updating verified environments through re-tasking, rewriting, fault injection, evolution, hardening, repair, and selective re-rolling.
    • Resulting coverage includes software engineering, interactive and multimodal tasks, cybersecurity, and skill-grounded professional work.

Method

IQuest-Q1 builds on a Transformer architecture enhanced with sparse Mixture-of-Experts (MoE) layers and a hybrid sliding-window attention mechanism. Refer to the framework diagram for a detailed view of the architecture.

The attention stack interleaves Grouped-Query Attention (GQA) and Sliding Window Attention (SWA). Within each attention module, a query-aware sink token provides an adaptive destination for excess attention mass. Query-key normalization (QKNorm) stabilizes attention by controlling query and key scales, with QKNorm Clip applied to prevent excessively large affine parameters. Partial RoPE applies rotary position embeddings to only a subset of dimensions in each attention head. On the feed-forward side, the first Transformer layer replaces sparse MoE routing with a dense MLP to mitigate expert load imbalance in the earliest layer. To improve scaling effectiveness, the authors adopt Stable SplitNorm, a hybrid normalization design that splits the normalization strategy across Transformer sublayers: the residual stream entering the attention sublayer is normalized, whereas the residual stream entering the feed-forward network (FFN) is kept unnormalized. This combines the stability benefits of Pre-Norm with the stronger depth utilization of Post-Norm. For efficient and stable scaling, the model combines the Muon optimizer with $\mu$μP parameterization, supporting efficient optimization and learning-rate transfer across model sizes.

The post-training process builds diverse CLI capabilities through synthesized tasks executed in production environments. As shown in the figure below, the pipeline integrates Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), Multi-Teacher On-Policy Distillation (MOPD), and model merging.

SFT is initialized from the mid-trained checkpoint and proceeds in two stages to prepare the policy for RL. The authors optimize the autoregressive cross-entropy loss on assistant responses, including tool-call tokens, while masking user instructions and tool observations. Following SFT, the model undergoes environment-centered RL targeting verifiable task completion. To consolidate the specialized policies, the authors employ MOPD, initializing the student from the SFT Stage 2 checkpoint and distilling knowledge from four frozen expert models covering agentic user experience, multi-harness operations, long-horizon tasks, and general agentic capabilities. The distillation process collects rollouts and selects a frozen teacher for each task based on its domain, optimizing a length-normalized surrogate loss using clipped distillation advantages. Throughout the post-training stages, the authors interleave training with residual model merging across stages and within expert branches, ultimately incorporating expert and MOPD checkpoints into the final model.

To support this post-training pipeline, the authors organize a task synthesis framework into general, reasoning, and coding settings. For coding agents, they construct executable environments from GitHub repositories and industrial data. As illustrated in the figure below, the resulting library covers diverse domains such as software engineering, terminal and CLI workflows, frontend and multimedia, code comprehension, repository generation, cybersecurity, and skill-grounded professional work, each utilizing specific environments, verifiers, and construction methods.

Experiment

The experiments evaluate IQuest-Q1 from controlled scaling comparisons through agentic benchmarks and applied CLI workflows. Scaling experiments show that the model attains better training efficiency than a Qwen3-MoE-style baseline and retains substantial stability headroom as model size increases. Pre-training is tracked with held-out trajectory likelihood and atomic ability checks, while downstream evaluations cover repository-level coding, general tool use, search, and reasoning; results indicate broad agentic strength but leave expert-level reasoning as a remaining gap. Additional studies validate interactive frontend generation, human-supervised iterative model development, and defensive cybersecurity support, including expert-validated vulnerability reporting in real-world deployments.

IQuest-Q1 uses a deep hybrid attention transformer with sparse mixture-of-experts feed-forward layers. Attention alternates between sliding-window and full attention, with full attention concentrated at the model boundaries and repeated periodically inside. The architecture has a very large total parameter count but activates only a small fraction of parameters per token. A repeating hybrid attention pattern uses three sliding-window attention layers followed by one full attention layer, while the first two and last two layers use full attention. Most feed-forward layers are sparse mixture-of-experts layers with hundreds of routed experts, whereas the first layer remains dense. Total parameters are much larger than activated parameters, reflecting sparse expert routing. Partial RoPE and query-aware sink attention are applied in attention layers, with different rotary bases for sliding-window and full attention.

IQuest-Q1 demonstrates strong repository-level coding abilities, with leading performance on CyberGym and competitive results on multilingual and repository-generation benchmarks, though Claude Opus-5 and GPT-5.6 Sol retain clear leads on several harder coding tasks. In general capabilities, the model shows strong scientific and mathematical reasoning and search or tool-use performance, while expert-level reasoning remains a relative gap, especially without tool support. IQuest-Q1 posts the highest reported CyberGym score among models with available results, slightly above GPT-5.6 Sol. On SWE-bench Multilingual and NL2Repo-Bench, IQuest-Q1 is second only to Claude Opus-5 and outperforms the other compared models. Claude Opus-5 leads SWE-Bench Pro and ProgramBench by a wide margin, while IQuest-Q1 lands in the middle to lower range on those benchmarks. IQuest-Q1 scores above 90 on GPQA Diamond and IMOAnswerBench-Verified, but HLE scores of 39.2 without tools and 55.7 with tools show expert-level reasoning still has room to improve.

IQuest-Q1 is evaluated as a sparse mixture-of-experts hybrid attention transformer that alternates sliding-window and full attention, combining large total capacity with low per-token activation. The evaluations cover repository-level coding, multilingual and repository-generation benchmarks, and general reasoning, science, mathematics, search, and tool use. The model shows leading repository-level coding performance on CyberGym and competitive results on multilingual and repository-generation tasks, while general capabilities are strong in scientific reasoning and tool use. Expert-level reasoning remains a relative weakness, especially without tool support.