One-sentence Summary

Qwen Team and Huazhong University of Science and Technology present Qwen-Drive-1.0, a vision-language foundation model for autonomous driving that integrates 3D perception, visual question answering, and motion planning into a unified framework by extending a pretrained VLM with a bird's-eye-view head for joint 3D object detection, semantic occupancy prediction, and map segmentation, a Planning Expert for trajectory generation, and a staged training recipe that combines driving supervision with general vision-language data, thereby achieving strong 3D perception and competitive motion planning across open-loop, pseudo-closed-loop, and closed-loop evaluations while largely preserving general vision-language capabilities.

Key Contributions

• Qwen-Drive-1.0 is a vision-language foundation model for autonomous driving that integrates 3D perception, driving visual question answering, and motion planning without altering the pretrained VLM architecture.
• An external BEV perception head jointly learns 3D object detection, semantic occupancy prediction, and BEV map segmentation, serving as an explicit and inspectable 3D probe that adds perception outputs to the shared VLM while preserving its general vision-language capability.
• A staged training recipe with a unified data pipeline unifies cross-dataset labels, rewrites VQA responses for consistency, filters samples, and combines driving supervision with general vision-language data, enabling domain adaptation and mitigating catastrophic forgetting.

Introduction

The authors address the challenge of building unified vision-language-action models for autonomous driving, where pretrained vision-language models (VLMs) can leverage broad world knowledge for reasoning in rare or out-of-distribution scenarios. Prior approaches typically adapt VLMs through driving-specific visual question answering, but this text-only supervision fails to enforce explicit 3D spatial understanding and often causes catastrophic forgetting of the general pretrained knowledge that is critical for both open-world reasoning and cockpit-driving integration on a single compute platform. To overcome these limitations, the authors introduce Qwen-Drive-1.0, a foundation model that, for the first time, integrates explicit 3D perception, driving scene question answering, and motion planning without modifying the native VLM architecture. By attaching a bird’s-eye-view perception head as a 3D probe and a dedicated Planning Expert, and by training with a staged recipe that blends driving-specific and general vision-language data, the model acquires robust scene understanding and trajectory generation while preserving its original general-purpose capabilities.

Dataset

The authors structure their training data into three groups: perception, vision-language, and planning. Each group unifies heterogeneous task definitions before mixing sources, and together they drive a multi-stage training recipe.

Perception data

• Sources and scales nuScenes (official split): 28,000 annotated keyframes for training, 6,000 for validation; six surround cameras; semantic occupancy labels from nuScenes-OccNet. OpenScene: 607,000 training frames and 9,000 validation frames; eight cameras; split created by holding out 16 logs from the trainval pool to balance city and time of day.
• Label unification
  • 3D detection adopts the seven OpenScene classes (vehicle, bicycle, generic_object, pedestrian, traffic_cone, barrier, czone_sign). nuScenes categories are merged: car/truck/trailer/bus/construction vehicle → vehicle; bicycle/motorcycle → bicycle; debris, pushable/pullable objects, bicycle racks, animals → generic_object; pedestrian, traffic_cone, barrier map directly; czone_sign is supervised only from OpenScene.
  • Semantic occupancy uses a shared ten-class space (seven detection classes plus driveable, background, empty). nuScenes maps driveable surface to driveable, free space to empty, and other surfaces/man-made/vegetation to background. OpenScene maps foreground classes directly, background surfaces and reserved labels to background, unknown/free to empty.
  • Offline completion augments missing categories: OpenScene lacks driveable occupancy; the authors rasterize the nuPlan vector map and relabel ground voxels inside driveable regions that were previously background. nuScenes lacks generic_object occupancy; pseudo-labels are generated from 3D boxes of bicycle racks, debris, and pushable/pullable objects, only updating voxels that already held semantic labels.
  • BEV map labels are rasterized online with a six-class schema (driveable surface, road line, road edge, crosswalk, walkway, background) rather than remapping existing raster annotations.
• Spatial unification Both datasets store 200x200x16 voxel grids but with different physical extents, voxel sizes, and LiDAR-to-ego transforms (nuScenes: ±40 m horizontal, z∈[-1.0,5.4] m, 0.4 m voxels, ~1.84 m vertical offset; OpenScene: ±50 m, z∈[-4.0,4.0] m, 0.5 m voxels, identity transform). To avoid misalignment and categorical resampling, native occupancy grids are preserved. A differentiable trilinear sampling operation maps a lifted feature volume onto each dataset’s grid during occupancy prediction. For nuScenes, target voxel centers are defined in the ego frame and transformed back to the LiDAR frame; an expanded vertical range [-5.0,5.4] m covers the offset. For OpenScene, the identity transform needs no frame conversion. BEV map supervision uses an ego-centered local patch rasterized at 0.15 m resolution (400x200 target).

Vision-language data

• Open-source driving data 24 public datasets are aggregated (including CODA-LM, DRAMA, DriveLM, NuInstruct, OmniDrive, and others). Only training splits are used. Preprocessing: Qwen3.5-Plus rewrites prompts and responses into a common conversational schema; bounding boxes are normalized to [0,1000) coordinates; view and frame tags are inserted; textual view references are revised; most multiple-choice questions become open-ended. Consistency filter: Qwen3.5-Flash checks semantic consistency with the source annotation. This reduces the set from 5.53M to 3.09M samples (55.9% retention). The retained data span multi-view (61.6%), single-view (20.1%), single-view temporal (10.1%), multi-view temporal (4.4%), and video (3.7%) samples, covering captioning, QA, grounding, spatial reasoning, and planning reasoning.
• Self-constructed driving data
  • Planning reasoning: Chain-of-Causation traces are built on NAVSIM, Waymo, and PAI-AV. A rule-based classifier produces a motion prior from the future ego trajectory; Qwen3.7-Plus generates reasoning traces conditioned on multi-view images, historical trajectory, motion prior, and navigation instruction. A multistage audit (judge classification + programmatic aggregation) filters traces, checking maneuver accuracy, causal role attribution, and future information leakage. Rare scenes receive higher sampling priority. Two response formats are saved: trace only, and trace + JSON future trajectory.
  • Camera ordering: Surround-view images are shuffled and view tags removed; the model must identify the front view and recover the clockwise order.
  • In-house perception QA: 30K examples from Chinese road scenes provide traffic-light grounding and 3D object detection supervision (3D boxes in global frame, camera pose in text).
• Stage 2 mixture The filtered public driving data exceeds the budget, so approximately 20% is sampled from each public source, stratified by task and question type. This subset is mixed with the self-constructed driving data and general-purpose vision-language data, yielding 1.54M examples before repetition. The shares are 9.7% 3D perception supervision, 26.0% general-purpose vision-language, and 64.3% driving vision-language. Group-specific repetition factors give perception examples a larger factor (to update the BEV head), while vision-language examples are repeated for two to three epochs. The effective mixture after repetition is 12.7% perception, 31.0% general-purpose VL, and 56.3% driving VL.

Planning data

• Sources and scale (Stage 3) NAVSIM, OpenScene (both nuPlan-derived but with different ego-motion distributions), WOD-E2E, and PAI-AV together form a training set of roughly 2.83M samples.
  • NAVSIM + OpenScene: 890K samples from 2.5K clips.
  • WOD-E2E: 557K samples from 2K clips.
  • PAI-AV: 1.38M samples from 156K clips (only a small number of evenly spaced frames are retained per clip). Of these samples, 685K (24.2%) carry an accepted planning-reasoning trace as a condition; the remaining 75.8% omit it.
• Preprocessing Every future trajectory is expressed in the current ego frame and converted to a 50-waypoint representation (positions and headings at 10 Hz). For NAVSIM/OpenScene, data is read directly from the nuPlan database. For PAI-AV, future egomotion is recovered from per-clip annotations and resampled. For WOD-E2E (which provides 4 Hz positions), a natural cubic spline is fit to the position sequence and evaluated at 10 Hz; velocities and accelerations are derived, and the heading rate is computed as (v_x a_y - v_y a_x)/||v||^2 for speeds ≥0.3 m/s, else zero, and integrated to obtain future headings. Samples are retained only if historical and future acceleration magnitudes stay within 9.8 m/s^2, and heading rates pass derivative-based and adjacent-step checks with a 1.2 rad/s threshold. Input composition: Each example includes front, front-left, and front-right images at four timesteps (current and three historical at 0.5 s intervals). Historical images are resized to 320p and current images to 720p, providing two visual token budgets while preserving spatial detail in the current observation. Together with the historical ego trajectory, current ego state, navigation instruction, and the optional planning-reasoning trace, these form the conditions for planning prediction.

Method

The authors leverage a unified architecture for Qwen-Drive-1.0 that integrates a shared vision encoder and a Vision-Language Model (VLM) to process diverse driving inputs.

The vision encoder converts images into visual tokens, which the VLM encodes alongside textual prompts to generate responses autoregressively. To handle multi-view and multi-frame inputs without explicit structural identifiers, the authors introduce view and frame tags. View tags denote eight canonical directions, while frame tags associate images with specific timesteps. Input serialization varies by task: question-answering examples use frame-major order, whereas planning examples use view-major order to expose temporal variation within each view. The VLM is trained using a standard next-token prediction objective.

Two external modules utilize features from this shared pathway to perform geometric and trajectory predictions without altering the core VLM architecture.

The BEV perception head constructs a shared ego-frame Bird's-Eye-View (BEV) representation for 3D object detection, semantic occupancy prediction, and BEV map segmentation. It fuses two complementary feature streams: low-level appearance features from the vision encoder and broader scene context features from the VLM output. A depth-based view transform lifts the single-scale vision encoder features into a 3D volume using a lightweight depth network that predicts per-pixel categorical depth distributions. A query-based BEV transformer then aggregates multi-scale VLM features onto the BEV plane, initialized with the height-collapsed 3D volume to provide geometric priors. Task-specific decoders, including a DETR-style decoder for detection and UNet-style heads for occupancy and map segmentation, operate on this integrated BEV feature.

The Planning Expert predicts future ego motion formulated as conditional generation. It employs a 32-layer diffusion transformer that conditions noisy trajectory tokens on cached keys and values from the VLM's grouped-query softmax attention layers. Shared adaptive layer normalization injects flow time, navigation instructions, and current ego state. The authors train this module using flow matching with an x-prediction parameterization, directly estimating the clean trajectory endpoint to reduce sensitivity to sensor noise. The objective combines flow matching loss with temporal regularizers to suppress waypoint jitter and abrupt acceleration changes.

The authors implement a four-stage training recipe to progressively adapt the model components.

In Stage 1, the authors pretrain the BEV perception head while keeping the vision encoder and VLM fixed. This initializes the view transform, BEV transformer, and task decoders to construct ego-frame representations.

Stage 2 involves joint training of the perception head, vision encoder, and VLM. Since head-only training yields limited performance, this stage optimizes the entire shared pathway for explicit 3D prediction using a mixture of perception and vision-language samples. The BEV perception head uses a learning rate 20 times higher than the VLM to adapt rapidly.

In Stage 3, the authors pretrain the Planning Expert while freezing the vision encoder and VLM. The training mixture includes samples with and without textual planning reasoning, supervising only the future trajectory through the flow matching objective. This separates trajectory learning from vision-language representation changes.

Stage 4 refines the Planning Expert through reinforcement learning. Since trajectory regression does not explicitly capture collision avoidance or human preference, the authors optimize the module using task-level rewards. To enable policy gradient optimization over the deterministic flow, they introduce stochastic transitions over the final three integration steps within a smooth low-frequency temporal subspace. This approach generates coherent trajectory variations while limiting deviation from the pretrained flow. The authors compute group-relative advantages across multiple rollouts and optimize the Planning Expert with a discounted policy gradient objective.

Experiment

Qwen-Drive-1.0 integrates 3D perception, driving visual understanding, and motion planning while preserving broad general vision-language capabilities. Joint adaptation with a BEV perception head yields leading detection and map segmentation results and maintains reasoning, while chain-of-causation supervision dramatically improves causal scene understanding. Reinforcement learning in motion planning enhances preference alignment and reduces closed-loop off-road events, though with a modest trade-off in forward progress, and planning ability continues to improve with more training data. The perception pathway also demonstrates plausible qualitative transfer to previously unseen camera rigs, showing operational 3D perception without dataset-specific adaptation.

Cross-dataset label unification aligns diverse perception annotations into shared task-specific taxonomies. Merging and completion strategies reduce class counts to 7 for 3D detection, 10 for semantic occupancy, and 6 for BEV map segmentation. Five vehicle classes from the source datasets are merged into a single class for 3D detection. Bicycle and motorcycle are combined into one unified class in the 3D detection taxonomy. The OpenScene-only czone_sign class is retained in the final 3D detection label set. Semantic occupancy processing uses dataset-specific lookup tables and completes generic_object from 3D boxes and driveable from map data. BEV map segmentation employs online rasterization of vector maps from both datasets under a shared 6-class schema.

On the unified nuScenes benchmark, Qwen-Drive-1.0-SFT achieves the strongest detection and map segmentation results among compared methods. Replacing ResNet-50 with a vision-language pretrained ViT backbone consistently improves detection, but a perception head trained on those same features still falls short of a multi-task ResNet-50 baseline in 3D detection and occupancy, indicating that visual-text alignment does not directly provide the 3D structure needed for driving perception. Vision-language pretrained backbones (SigLIP-Qwen) substantially boost detection: BEVFormerV2's mAP on nuScenes rises from 33.04 with ResNet-50 to 40.78 with the ViT. Qwen-Drive-1.0-SFT surpasses the multi-task BEVFormerV2* by 2.01 mAP and PETRv2 by 3.37 map mIoU on nuScenes, and also improves over the head-only joint model on OpenScene detection and map segmentation.

Qwen3.5-4B achieves the highest average across the six higher-is-better driving VQA metrics, outperforming larger models. Gemma4-12B leads on Waymo safety questions and the in-house driving decision benchmark, while Cosmos-Reason1-7B, despite the lowest overall VQA average, attains the best key-object accuracy in causal chain-of-causation reasoning, revealing that standard benchmarks do not fully capture causal driving competence. Qwen3.5-4B attains the top composite average on public driving VQA and spatial understanding benchmarks, with the best scores on LingoQA, Ego3D accuracy, VLAD, and SURDS. Cosmos-Reason1-7B obtains the highest key-object accuracy on PAI-AV-Coc causal reasoning even though it ranks last on the overall VQA average, while Gemma4-12B excels on Waymo safety accuracy and the in-house decision benchmark.

A qualitative example of temporal understanding shows that the model's prediction for the number of parked vehicles in a video sequence is internally inconsistent, starting with a claim of zero, then two, and finally three parked vehicles. This self-contradiction highlights difficulties in maintaining a coherent agent-state estimate across frames. The model initially states there are no parked vehicles, then later describes two parked vehicles on opposite sides of the road, and finally counts three on the right side in the last frame. The shifting count for the same scene demonstrates a failure in stable temporal state tracking, which is critical for reliable scene understanding over time.

After driving adaptation, Qwen-Drive-1.0-SFT retains general knowledge and reasoning abilities within one point of Qwen3.5-4B on group (a) benchmarks, while surpassing it on spatial understanding and grounding in group (b). It achieves top scores on MMStar, RealWorldQA, ERQA, and ODinW13, and outperforms Cosmos-Reason2-32B by over five points on average across all settings, demonstrating preserved broad competence with strengthened spatial reasoning. On knowledge, reasoning, and recognition benchmarks, Qwen-Drive-1.0-SFT averages 66.41, staying within one point of Qwen3.5-4B's 67.40. On spatial understanding and grounding benchmarks, it averages 53.96 and exceeds Qwen3.5-4B's 52.99. It ranks first or second on 6 of the 10 group (a) settings, including the best scores on MMStar and RealWorldQA. It achieves the best scores on ERQA and ODinW13, covering embodied spatial reasoning and open-set object grounding. It matches or exceeds Cosmos-Reason2-32B on 10 of the 15 settings, with a 5.48-point higher overall average.

The evaluation unifies labels across multiple perception datasets into shared taxonomies for 3D detection, semantic occupancy, and BEV map segmentation, then assesses models on the resulting benchmark. Vision-language pretrained backbones substantially improve detection, but multi-task perception heads trained on these features still lag in three-dimensional understanding, indicating that visual-text alignment alone does not provide the required 3D structure. On driving visual question answering, standard aggregate scores fail to reflect causal reasoning competence, and qualitative tests expose inconsistent temporal state tracking across frames. After driving adaptation, the model preserves general knowledge and reasoning while surpassing both its base version on spatial grounding and larger models on overall average performance.