Domain Generalization by Functional Regression

Abstract

The problem of domain generalization is to learn, given data from different source distributions, a model that can be expected to generalize well on new target distributions which are only seen through unlabeled samples. In this paper, we study domain generalization as a problem of functional regression. Our concept leads to a new algorithm for learning a linear operator from marginal distributions of inputs to the corresponding conditional distributions of outputs given inputs. Our algorithm allows a source distribution-dependent construction of reproducing kernel Hilbert spaces for prediction, and, satisfies finite sample error bounds for the idealized risk. Numerical implementations and source code are available¹.

Keywords:

• Domain generalization
• finite sample bounds
• function-to-function regression
• operator learning
• reproducing kernel Hilbert space

Acknowledgments

We acknowledge the ELLIS Unit Linz, the LIT AI Lab, and the Institute for Machine Learning at the University of Linz. In addition, the research reported in this paper has been partly funded by the Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK), the Federal Ministry for Digital and Economic Affairs (BMDW), and the Province of Upper Austria in the frame of the COMET–Competence Centers for Excellent Technologies Programme and the COMET Module S3AI managed by the Austrian Research Promotion Agency FFG.

Notes

1 Source code: https://github.com/wzell/FuncRegr4DomGen

2 If we equip ${\mathcal{M}}_1^{+}(\mathcal{X}\times \mathcal{Y})$ with ${\tau }_w(\mathcal{X}\times \mathcal{Y})$, the weakest topology on ${\mathcal{M}}_1^{+}(\mathcal{X}\times \mathcal{Y})$ such that the mapping $L_h:({\mathcal{M}}_1^{+}(\mathcal{X}\times \mathcal{Y}),{\tau }_w(\mathcal{X}\times \mathcal{Y}))\to \mathbb{R}$ with $L_h(P)={\int }_{\mathcal{X}\times \mathcal{Y}}h(x,y)\mathrm{}\mathrm{d}P(x,y)$ is continuous for all bounded and continuous functions $h:\mathcal{X}\times \mathcal{Y}\to \mathbb{R}$ and denote by $\mathcal{B}({\tau }_w(\mathcal{X}\times \mathcal{Y}))$ the associated Borel σ- algebra, then $({\mathcal{M}}_1^{+}(\mathcal{X}\times \mathcal{Y}),\mathcal{B}({\tau }_w(\mathcal{X}\times \mathcal{Y})))$ becomes a itself measurable space, cf. [5, 10].

3 It holds that $m_{P_{\mathcal{X}}}\in {\mathcal{H}}_k\subseteq L^2(\mathcal{X})$, the space of square-integrable functions on $\mathcal{X}$. The mapping $m:P_{\mathcal{X}}\mapsto m_{P_{\mathcal{X}}}$ is well-defined if the kernel k is bounded and it is injective if k is universal [11, 12].

4 The regression function f_P is well-defined since $\mathcal{X}$ and $\mathcal{Y}$ are Polish spaces (as compact subsets of ${\mathbb{R}}^{d_1},{\mathbb{R}}^{d_2}$) and, therefore, every $P\in {\mathcal{M}}_1^{+}(\mathcal{X}\times \mathcal{Y})$ can be factorized $P(x,y)=P(y|x)P_{\mathcal{X}}(x)$ in a conditional probability measure $P(y|x)$ and a marginal (w.r.t. $\mathcal{X}$) probability measure $P_{\mathcal{X}}(x)$, see [13, Theorem 10.2.1].