One-sentence Summary

Researchers at Google Cloud AI propose TabNet, a high-performance, interpretable deep tabular learning architecture that uses sequential attention to choose which features to reason from at each decision step, achieving interpretable feature attributions and insights into its global behavior while outperforming other variants on a wide range of non-performance-saturated tabular datasets and enabling self-supervised learning that significantly improves performance when unlabeled data is abundant.

Key Contributions

• TabNet is a deep tabular learning architecture that uses sequential attention to select the most salient features at each decision step, enabling efficient capacity utilization and interpretability through instance-wise feature selection masks.
• Experiments show that TabNet outperforms prior models on a wide range of non-performance-saturated tabular datasets and yields interpretable feature attributions that reveal global decision-making behavior.
• Self-supervised pre-training on unlabeled tabular data significantly boosts TabNet performance and allows fast adaptation, especially when unlabeled data is abundant.

Introduction

Deep neural networks have achieved remarkable success in domains like images and text, but tabular data remains a challenging frontier where gradient-boosted trees often still reign. Prior deep learning approaches for tabular data typically process all features at once, making it hard to isolate the most salient inputs and limiting model interpretability. The authors introduce TabNet, a novel architecture that uses a sequential attention mechanism to perform instance-wise feature selection: at each decision step, the model picks a subset of semantically meaningful features to attend to, allocating capacity only to the most relevant information. This design improves efficiency and enables direct visualization of the feature selection masks, yielding more interpretable decisions while outperforming earlier methods across diverse tabular benchmarks.

Experiment

The table illustrates a real-world tabular dataset where feature columns are interdependent, so missing values in one column can often be inferred from others. This property motivates masked self-supervised learning, which exploits these correlations to produce a stronger encoder for downstream supervised tasks. Missing education is accompanied by occupational roles like exec-managerial or farming-fishing that suggest likely education levels. Missing gender coincides with relationship labels such as wife or husband, making gender easy to infer. Masked self-supervised learning leverages these interdependencies to improve the encoder for supervised fine-tuning.

INVASE consistently yielded the highest test AUC across all six synthetic datasets, often by a large margin. Global feature selection was the second strongest performer, closely tracking INVASE and tying on Syn5. Tree ensemble methods achieved competitive scores on some datasets but were less stable, while Lasso-regularized and L2X models frequently underperformed even the baseline that uses all features without selection. INVASE achieved the best AUC on every dataset, with particularly large gains over other methods on Syn1, Syn4, and Syn6. Global feature selection performed nearly on par with INVASE, tying on Syn5 and trailing by only a few points on the remaining datasets. Lasso-regularized and L2X models often scored below the no-selection baseline, especially on Syn1 and Syn2, indicating they discarded useful features.

On the Forest Cover Type dataset, gradient boosting methods XGBoost and LightGBM achieved similar accuracy, while CatBoost scored several points lower. AutoML Tables and TabNet substantially outperformed the boosting models, with TabNet reaching the highest accuracy. XGBoost and LightGBM performed nearly identically, with only a marginal difference in test accuracy. CatBoost lagged behind other boosting methods, falling short by several percentage points. AutoML Tables improved over traditional boosting, and TabNet delivered the best overall accuracy, surpassing all other models.

On the Poker Hand induction dataset, TabNet achieves near-perfect accuracy at 99.2%, vastly outperforming all other models. Tree-based ensemble methods (XGBoost, LightGBM, CatBoost) reach moderate accuracy around 70%, while a deep neural decision tree offers a modest improvement over simple decision trees and MLPs, which remain at the 50% baseline. TabNet reaches 99.2% test accuracy, leaving all other models far behind. XGBoost, LightGBM, and CatBoost cluster in the 70% range, with XGBoost slightly ahead at 71.1%. A deep neural decision tree attains 65.1%, outperforming standard decision trees and MLPs that stay at the 50% random baseline.

On the Sarcos dataset, the TabNet-M model achieves a test MSE of 0.28, dramatically outperforming all other methods, while the compact TabNet-S reaches a test MSE of 1.25 with only 6.3K parameters. This makes TabNet-S competitive with the adaptive neural tree (MSE 1.23) using 95% fewer parameters. TabNet-M reduces test MSE to 0.28, far below the next best model (adaptive neural tree at 1.23). TabNet-S matches the adaptive neural tree's performance with only 6.3K parameters, a 95% reduction in model size while still outperforming gradient boosted trees and random forests.

The experiments evaluate masked self-supervised learning for exploiting feature interdependencies in tabular data, instance-wise feature selection with INVASE on synthetic datasets, and the TabNet model on multiple supervised tasks. The masked learning setup shows that leveraging correlations among missing values improves the encoder for downstream use, while INVASE consistently selects better features than global selection methods. TabNet achieves superior accuracy across diverse benchmarks, substantially outperforming boosting and tree-based models, and its compact variant matches larger models with a fraction of the parameters.