Deep Neural Networks Outperform Traditional Models in E-Commerce Ad Ranking: A DoorDash Case Study

Online advertising platforms have seen significant transformations over the past decade, evolving from basic impression-based strategies to complex, conversion-optimized systems. Platforms like DoorDash and Airbnb prioritize conversions, making the Cost Per Acquisition (CPA) model highly effective. This focus on CVR (Conversion Rate) prediction is crucial for ranking high-quality ads, optimizing auction efficiency, and maintaining a fair marketplace. However, traditional models, particularly tree-based ones like Gradient-Boosted Decision Trees (GBDT), have reached their limits in capturing the intricacies of user interactions, leading companies to embrace Deep Neural Networks (DNNs). Why the Shift to DNNs? Tree-based models, while efficient, struggle to handle the complexity and diversity of consumer interactions. They plateau in performance, limiting further advancements. In contrast, DNNs offer advanced representational learning capabilities, excelling in processing large-scale, heterogeneous data, including temporal trends, contextual signals, and multimodal inputs like text, images, and graphs. DNNs also enable cross-domain knowledge sharing through transfer learning and support holistic user behavior modeling via multi-task learning (MTL). Transition to DNNs: The DoorDash Case Study Step 1: Establishing a Baseline The foundation of any online ad platform lies in two critical components: the Model Training Service and the Ad Exchange Service. The Model Training Service uses historical prediction and engagement logs to train model artifacts offline, while the Ad Exchange Service leverages request metadata and these artifacts to rank ads in real-time auctions. Robust logging mechanisms capture high-quality data for continuous model training. Before migrating to DNNs, DoorDash established a strong baseline to ensure a data-driven transition, minimizing risks and maximizing business value. Step 2: Model Training and Evaluation To optimize training throughput, DoorDash implemented normalization layers as a pre-processing step, offloading tasks to CPU pools and reserving GPUs for computationally intensive operations. This strategic workload distribution minimized bottlenecks, maximized hardware efficiency, and accelerated model training. Key performance metrics, such as Area Under the Curve (AUC) and Normalized Binary Cross Entropy (BCE), were used to evaluate the models. Step 3: Model Evolution Stages 3.1: Adopting Deep Learning Recommendation Models Initial A/B experiments comparing GBDT and DNN models showed clear advantages of DNNs. DoorDash then focused on refining DNN-specific architecture designs and feature engineering. While adding sparse features initially provided marginal gains, scaling up the feature set led to severe overfitting and a wider gap between offline and online performance. Systematic debugging and in-depth analysis were required to overcome these challenges. 3.2: Deep Personalization Two key user behavior patterns were identified: a high tendency for repeat purchases and a reluctance to explore new options. Time of order was also a critical factor. DoorDash engineered new features to capture signals such as daypart (time of day), user preferences for specific stores and dishes, and overall price sensitivity. Pre-trained embeddings were introduced to mitigate cold-start and data sparsity issues. These enhancements resulted in a 2.8% improvement in CVR. 3.3: Bridging the AUC Gap Despite improvements, a 4.3% AUC gap remained between offline model evaluations and live online performance. Initial hypotheses suspected the age of training data, but deeper analysis revealed that discrepancies in feature logging were the primary culprits. DoorDash implemented feature-specific join windows to handle delays and enabled online logging for features prone to inconsistencies. Although this led to a 10% increase in load on the prediction service, the business gains—improved ad ranking quality and reduced latency—justified the additional costs. Conclusion The transition to DNNs in e-commerce ad ranking demonstrates significant improvements in model accuracy and CVR, highlighting their capability to handle complex, diverse data and enable deeper personalization. The DoorDash case study illustrates that with the right infrastructure and optimizations, DNNs can deliver tangible business value. As such, DNNs are poised to become a robust foundation for the next generation of ad ranking systems, offering a superior alternative to traditional tree-based models. Industry Insights and Company Profiles Industry insiders agree that the shift to DNNs is a necessary evolution for e-commerce platforms aiming to stay competitive. According to leading analysts, DNNs' ability to process multimodal data and capture nuanced user behaviors is a game-changer in ad ranking. DoorDash, a prominent player in the food delivery market, has showcased its commitment to innovation and technology by successfully integrating DNNs into its ad ecosystem, setting a benchmark for others to follow. This transition underscores the company's dedication to enhancing user experience and maximizing business outcomes, solidifying its position as a leader in the sector.