Semi-Structured Distributional Regression

Abstract

Combining additive models and neural networks allows to broaden the scope of statistical regression and extend deep learning-based approaches by interpretable structured additive predictors at the same time. Existing attempts uniting the two modeling approaches are, however, limited to very specific combinations and, more importantly, involve an identifiability issue. As a consequence, interpretability and stable estimation are typically lost. We propose a general framework to combine structured regression models and deep neural networks into a unifying network architecture. To overcome the inherent identifiability issues between different model parts, we construct an orthogonalization cell that projects the deep neural network into the orthogonal complement of the statistical model predictor. This enables proper estimation of structured model parts and thereby interpretability. We demonstrate the framework’s efficacy in numerical experiments and illustrate its special merits in benchmarks and real-world applications.

Keywords:

• Generalized additive models for location
• Scale and shape
• Identifiability
• Multimodal learning
• Neural networks
• Orthogonalization

Supplementary Materials

Further Details: The Supplementary Material includes proofs, algorithmic details as well as further specifications and results of numerical experiments.

Reproducibility: All codes used for this work are available at https://github.com/davidruegamer/semi-structured/_distributional/_regression

Acknowledgments

We thank Almond Stöcker and Dominik Thalmeier for their comments and helpful discussions.

Additional information

Funding

David Rügamer has been partly funded by the German Federal Ministry of Education and Research (BMBF) under grant no. 01IS18036A. Nadja Klein acknowledges support by the Deutsche Forschungsgemeinschaft (DFG; German research foundation) through the Emmy Noether grant KL 3037/1-1.