DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving

End-to-end autonomous driving (E2E-AD) has emerged as a trend in the field ofautonomous driving, promising a data-driven, scalable approach to systemdesign. However, existing E2E-AD methods usually adopt the sequential paradigmof perception-prediction-planning, which leads to cumulative errors andtraining instability. The manual ordering of tasks also limits the system`sability to leverage synergies between tasks (for example, planning-awareperception and game-theoretic interactive prediction and planning). Moreover,the dense BEV representation adopted by existing methods brings computationalchallenges for long-range perception and long-term temporal fusion. To addressthese challenges, we present DriveTransformer, a simplified E2E-AD frameworkfor the ease of scaling up, characterized by three key features: TaskParallelism (All agent, map, and planning queries direct interact with eachother at each block), Sparse Representation (Task queries direct interact withraw sensor features), and Streaming Processing (Task queries are stored andpassed as history information). As a result, the new framework is composed ofthree unified operations: task self-attention, sensor cross-attention, temporalcross-attention, which significantly reduces the complexity of system and leadsto better training stability. DriveTransformer achieves state-of-the-artperformance in both simulated closed-loop benchmark Bench2Drive and real worldopen-loop benchmark nuScenes with high FPS.