Abstract
While deep learning has enabled tremendous progress on text and image
datasets, its superiority on tabular data is not clear. We contribute extensive
benchmarks of standard and novel deep learning methods as well as tree-based
models such as XGBoost and Random Forests, across a large number of datasets
and hyperparameter combinations. We define a standard set of 45 datasets from
varied domains with clear characteristics of tabular data and a benchmarking
methodology accounting for both fitting models and finding good
hyperparameters. Results show that tree-based models remain state-of-the-art on
medium-sized data ($\sim$10K samples) even without accounting for their
superior speed. To understand this gap, we conduct an empirical investigation
into the differing inductive biases of tree-based models and Neural Networks
(NNs). This leads to a series of challenges which should guide researchers
aiming to build tabular-specific NNs: 1. be robust to uninformative features,
2. preserve the orientation of the data, and 3. be able to easily learn
irregular functions. To stimulate research on tabular architectures, we
contribute a standard benchmark and raw data for baselines: every point of a 20
000 compute hours hyperparameter search for each learner.
Users
Please
log in to take part in the discussion (add own reviews or comments).