Quadratic Neural Networks for Solving Inverse Problems

Abstract

In this paper we investigate the solution of inverse problems with neural network ansatz functions with generalized decision functions. The relevant observation for this work is that such functions can approximate typical test cases, such as the Shepp-Logan phantom, better, than standard neural networks. Moreover, we show that the convergence analysis of numerical methods for solving inverse problems with shallow generalized neural network functions leads to more intuitive convergence conditions, than for deep affine linear neural networks.

KEYWORDS:

• Generalized neural network functions
• inverse problems

MATHEMATICS SUBJECT CLASSIFICATION:

• 65J22
• 45Q05
• 42C40
• 65F20

1 Introduction

We consider an inverse problem, which consists in solving an operator equation (1.1) $$F(\mathbf{x})={\mathbf{y}}_0,$$(1.1) where $F:\mathbf{X}\to \mathbf{Y}$ denotes an operator, mapping between function spaces $(\mathbf{X},\Vert ·\Vert )$ and $(\mathbf{Y},\Vert ·\Vert )$. For the numerical solution of equation (1.1)(1.1) $$F(\mathbf{x})={\mathbf{y}}_0,$$(1.1) an appropriate set of ansatz-functions $\mathbf{P}$ has to be selected, on which an approximate solution of equation (1.1)(1.1) $$F(\mathbf{x})={\mathbf{y}}_0,$$(1.1) is calculated, for instance a function $\mathbf{p}\in \mathbf{P}\subseteq \mathbf{X}$, which solves (1.2) $$F(\mathbf{p})=\mathbf{y},$$(1.2) where $\mathbf{y}$ is an approximation of ${\mathbf{y}}_0$. The history on the topic of approximating the solution of equation (1.1)(1.1) $$F(\mathbf{x})={\mathbf{y}}_0,$$(1.1) by discretization and regularization is long: For linear inverse problems this has been investigated for instance in [1]. Much later on neural network ansatz functions have been used for solving equation (1.1)(1.1) $$F(\mathbf{x})={\mathbf{y}}_0,$$(1.1) ; see for instance [2]. In this paper we investigate solving equation (1.2)(1.2) $$F(\mathbf{p})=\mathbf{y},$$(1.2) when $\mathbf{P}$ is a set of neural network functions with generalized decision functions. In order to spot the difference to classical neural network functions we recall the definition of them first:

Definition 1.1

(Affine linear neural network functions). Let $m\ge n\in \mathbb{N}$. Line vectors in ${\mathbb{R}}^m$ and ${\mathbb{R}}^n$ are denoted by $$\mathbf{w}={(w^{(1)},w^{(2)},\dots ,w^{(m)})}^T\text{and}\overrightarrow{x}={(x_1,x_2,\dots ,x_n)}^T,\text{respectively}.$$

• A shallow affine linear neural networks (ALNN) is a function (here m = n) (1.3) $$\overrightarrow{x}\in {\mathbb{R}}^n\to \mathbf{p}(\overrightarrow{x}):=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x}):=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\theta }_j\right),$$(1.3) with ${\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n$; σ is an activation function, such as the sigmoid, ReLU ${}^k$ or Softmax function. Moreover, (1.4) $$\overrightarrow{p}=[{\alpha }_1,\dots ,{\alpha }_N;{\mathbf{w}}_1,\dots ,{\mathbf{w}}_N;{\theta }_1,\dots ,{\theta }_N]\in {\mathbb{R}}^{n_{*}}\text{with}{n}_{*}=(n+2)N$$(1.4) denotes the according parameterization of $\mathbf{p}$. In this context the set of ALNNs is given by $$\mathbf{P}:=\left\{\mathbf{p}\text{of the form equation}(1.3):\overrightarrow{p}\in {\mathbb{R}}^{n_{*}}\right\}.$$

• More recently deep affine linear neural network functions (DNNs) have become popular: An $(L+2)$-layer network looks as follows: (1.5) $$\begin{array}{cc}\overrightarrow{x}\in {\mathbb{R}}^n\to \hfill & \mathbf{p}(\overrightarrow{x}):=\hfill \\ \hfill & \Psi \left[\overrightarrow{p}\right](\overrightarrow{x}):={\alpha }^T{\sigma }_L\left(p_L\left(\dots \left({\sigma }_1\left(p_1(\overrightarrow{x})\right)\right)\right)\right).\hfill \end{array}$$(1.5) where $$p_l(\overrightarrow{x})={({\mathbf{w}}_{1,l},{\mathbf{w}}_{2,l},\dots ,{\mathbf{w}}_{N_l,l})}^T\overrightarrow{x}+{({\theta }_{1,l},{\theta }_{2,l},\dots ,{\theta }_{N_l,l})}^T$$with $\alpha \in {\mathbb{R}}^{N_L},{\theta }_{j,l}\in \mathbb{R}$ and ${\mathbf{w}}_{j,l}\in {\mathbb{R}}^{N_{l-1}}$ for all $l=1,\dots ,L$ and all $j=1,\dots ,N_l$. Here N_l denotes the number of neurons in l-th internal layer and $N_0=n,{\sigma }_k(s),k=1,\dots ,L$ denote activation functions of each layer; they are used as maps ${\mathbb{R}}^{N_l}\to {\mathbb{R}}^{N_l}$ by acting component-wise on vectors. Moreover, we denote the parametrizing vector by (1.6) $$\overrightarrow{p}=[\alpha ;{\mathbf{w}}_{1,1},\dots ,{\mathbf{w}}_{N_L,L};{\theta }_{1,1},\dots ,{\theta }_{N_L,L}]\in {\mathbb{R}}^{n_{*}}.$$(1.6) A shallow linear neural network can also be called a 3-layer network. The notation 3-layer network is consistent with the literature because input-and output are counted on top. Therefore, a general $(L+2)$-layer network has only L internal layers.

We demonstrate exemplary below (see Section 3.2) that the convergence analysis of an iterative Gauss-Newton method for solving a linear inverse problem restricted to a set of deep neural network ansatz functions is involved (see Example 3.4), while the analysis with shallow neural networks ansatz functions leads to intuitive conditions (see also [3]). Moreover, it is well-known in regularization theory that the quality of approximation of the solution of equation (1.2)(1.2) $$F(\mathbf{p})=\mathbf{y},$$(1.2) is depending on the approximation qualities of the set $\mathbf{P}$: Approximation of functions with shallow affine linear neural networks is a classical topic of machine learning and in approximation theory, see for instance [4–11]. The central result concerns the universal approximation property. Later on the universal approximation property has been established for different classes of neural networks: Examples are dropout neural networks (see [12, 13]), convolutional neural networks (CNN) (see for example [14, 15]), recurrent neural networks (RNN) (see [16, 17]), networks with random nodes (see [18]), with random weights and biases (see [19, 20]) and with fixed neural network topology (see [21]), to name but a few.

In this paper we follow up on this topic and analyze neural network functions with decision functions, which are higher order polynomials, as specified now:

Definition 1.2

(Shallow generalized neural network function). Let $n,m,N\in \mathbb{N}$. Moreover, let $${\mathbf{f}}_j:{\mathbb{R}}^n\to {\mathbb{R}}^m,\hspace{1em}j=1,\dots ,N$$be vector-valued functions. Neural network functions associated to $\left\{{\mathbf{f}}_j:j=1,\dots ,N\right\}$ are defined by (1.7) $$\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x}):=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T{\mathbf{f}}_j(\overrightarrow{x})+{\theta }_j\right)\text{with}{\alpha }_j,{\theta }_j\in \mathbb{R}\text{and}{\mathbf{w}}_j\in {\mathbb{R}}^m.$$(1.7)

Again we denote the parameterization vector, containing all coefficients, with $\overrightarrow{p}$ and the set of all such functions with $\mathbf{P}$. We call (1.8) $$\mathcal{D}:=\left\{\overrightarrow{x}\to {\mathbf{w}}^T{\mathbf{f}}_j(\overrightarrow{x})+\theta :\mathbf{w}\in {\mathbb{R}}^m,\theta \in \mathbb{R},j=1,\dots ,N\right\}$$(1.8) the set of decision functions associated to equation (1.7)(1.7) $$\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x}):=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T{\mathbf{f}}_j(\overrightarrow{x})+{\theta }_j\right)\text{with}{\alpha }_j,{\theta }_j\in \mathbb{R}\text{and}{\mathbf{w}}_j\in {\mathbb{R}}^m.$$(1.7) . The composition of σ with a decision function is called neuron.

We discuss approximation properties of generalized neural networks (see Theorem 2.5), in particular neural network functions with decision functions, which are at most quadratic polynomials in $\overrightarrow{x}$. This idea is not completely new: In [22] neural networks with parabolic decision functions have been applied in a number of applications. In [23], the authors proposed neural networks with radial decision functions, and proved an universal approximation result for ReLU-activated deep quadratic networks.

Outline of this paper

The motivation for this paper is to solve equation (1.2)(1.2) $$F(\mathbf{p})=\mathbf{y},$$(1.2) on a set of neural network functions $\mathbf{P}$ with iterative algorithms, such as for instance a Gauss-Newton method. Good approximation properties of functions of interest (such as the Shepp-Logan phantom) is provided with quadratic decision functions, which have been suggested already in [9]. We concentrate on shallow neural networks because the resulting analysis of iterative methods for solving inverse problems is intuitive (see Section 3.2). We apply different approximation theorems (such as for instance the universal approximation theorem of [6]) to cover all situations from Definition 2.1, below. Moreover, by constructing wavelet frames based on quadratic decision functions, we give an explicit convergence rate for shallow radial network approximations; see Section 4, with main result Theorem 4.6. In Appendix A the important conditions of the approximation to the identity (AtI) from [24] used for proving the convergence rates in Section 4 are verified.

2 Examples of networks with generalized decision functions

The versatility of the network defined in equation (1.7)(1.7) $$\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x}):=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T{\mathbf{f}}_j(\overrightarrow{x})+{\theta }_j\right)\text{with}{\alpha }_j,{\theta }_j\in \mathbb{R}\text{and}{\mathbf{w}}_j\in {\mathbb{R}}^m.$$(1.7) is due to the flexibility of the functions ${\mathbf{f}}_j,j=1,\dots ,N$ to chose.

We give a few examples.

Definition 2.1

(Neural networks with generalized decision functions). We split them into three categories:

General quadratic neural networks (GQNN): Let $$m=n+1,\hspace{1em}{\mathbf{f}}_j(\overrightarrow{x})=\left(\begin{array}{c}{f}_j^{(1)}(\overrightarrow{x})\\ ⋮\\ f_j^{(n)}(\overrightarrow{x})\\ f_j^{(n+1)}(\overrightarrow{x})\end{array}\right)=\left(\begin{array}{c}{x}_1\\ ⋮\\ x_n\\ {\overrightarrow{x}}^T{A}_j\overrightarrow{x}\end{array}\right)\text{for all}j=1,\dots ,N.$$

That is, ${\mathbf{f}}_j$ is a graph of a quadratic function. Then equation (1.7)(1.7) $$\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x}):=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T{\mathbf{f}}_j(\overrightarrow{x})+{\theta }_j\right)\text{with}{\alpha }_j,{\theta }_j\in \mathbb{R}\text{and}{\mathbf{w}}_j\in {\mathbb{R}}^m.$$(1.7) reads as follows: (2.1) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\overrightarrow{x}}^T{A}_j\overrightarrow{x}+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,A_j\in {\mathbb{R}}^{n\times n}.\hfill \end{array}$$(2.1) $\overrightarrow{p}$ denotes the parameterization of $\mathbf{p}$: (2.2) $$\begin{array}{cc}\overrightarrow{p}\hfill & =[{\alpha }_1\dots ,{\alpha }_N;{\mathbf{w}}_1,\dots ,{\mathbf{w}}_N;A_1,\dots ,A_N;{\theta }_1,\dots ,{\theta }_N]\in {\mathbb{R}}^{n_{*}}\text{with}\hfill \\ n_{*}\hfill & =(n^2+n+2)N.\hfill \end{array}$$(2.2)

Note that in equation (2.1)(2.1) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\overrightarrow{x}}^T{A}_j\overrightarrow{x}+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,A_j\in {\mathbb{R}}^{n\times n}.\hfill \end{array}$$(2.1) all parameters of the entries of the matrices A_j are parameters.

Constrained quadratic neural networks (CQNN): These are networks, where the entries of the matrices A_j, $j=1,\dots ,N$ are constrained:

1. Let $$\mathbf{f}(\overrightarrow{x})={\mathbf{f}}_j(\overrightarrow{x})=\left(\begin{array}{c}{f}_j^{(1)}(\overrightarrow{x})\\ ⋮\\ f_j^{(n)}(\overrightarrow{x})\\ f_j^{(n+1)}(\overrightarrow{x})\end{array}\right)=\left(\begin{array}{c}{x}_1\\ ⋮\\ x_n\\ 0\end{array}\right)\text{for all}j=1,\dots ,N.$$That is A_j = 0 for all $j=1,\dots ,N$. This set of CQNNs corresponds with the ALNNs defined in equation (1.3)(1.3) $$\overrightarrow{x}\in {\mathbb{R}}^n\to \mathbf{p}(\overrightarrow{x}):=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x}):=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\theta }_j\right),$$(1.3) .

2. Let $A_j\in {\mathbb{R}}^{n\times n},j=1,\dots ,N$ be chosen and fixed. We denote by MCNN the family of functions (2.3) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\overrightarrow{x}}^T{A}_j\overrightarrow{x}+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j,{\xi }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,A_j\in {\mathbb{R}}^{n\times n},\hfill \end{array}$$(2.3) which is parameterized by the vector (2.4) $$\begin{array}{cc}\overrightarrow{p}\hfill & =[{\alpha }_1\dots ,{\alpha }_N;{\mathbf{w}}_1,\dots ,{\mathbf{w}}_N;{\xi }_1,\dots ,{\xi }_N;{\theta }_1,\dots ,{\theta }_N]\in {\mathbb{R}}^{n_{*}}\text{with}\hfill \\ n_{*}\hfill & =(n+3)N.\hfill \end{array}$$(2.4)

In particular, choosing $A_j=\mathcal{I}$ we get: (2.5) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,{\xi }_j\in \mathbb{R}.\hfill \end{array}$$(2.5)

Radial quadratic neural networks (RQNN)s: For ${\xi }_j\ne 0$ the argument in equation (2.5)(2.5) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,{\xi }_j\in \mathbb{R}.\hfill \end{array}$$(2.5) rewrites to (2.6) $$\begin{array}{c}{\nu }_j(\overrightarrow{x}):={\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\widehat{\mathbf{w}}}_j^T\overrightarrow{x}+{\theta }_j={\xi }_j{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^2+{\kappa }_j\hfill \end{array}$$(2.6) with (2.7) $${\overrightarrow{y}}_j=-\frac{1}{2{\xi }_j}{\widehat{\mathbf{w}}}_j^T\text{and}{\kappa }_j={\theta }_j-\frac{{\Vert {\widehat{\mathbf{w}}}_j\Vert }^2}{4{\xi }_j}.$$(2.7)

We call the set of functions from equation (2.5)(2.5) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,{\xi }_j\in \mathbb{R}.\hfill \end{array}$$(2.5) , which satisfy (2.8) $${\xi }_j\ge 0\text{and}{\kappa }_j\le 0\text{for all}j=1,\dots ,N,$$(2.8)

RQNNs, radial neural network functions, because the level sets of ν_j are circles. These are radial basis functions (see for instance [25] for a general overview).

Sign based quadratic (SBQNN) and cubic neural networks (CUNN): Let m = n.

1. (SBQNN): Let (2.9) $$f^{(i)}(\overrightarrow{x})=\text{sgn}(x_i)x_i^2\text{for}i=1,\dots ,n,$$(2.9) or alternatively $$f^{(i)}(\overrightarrow{x})={\Vert \overrightarrow{x}\Vert }^2\text{sgn}(x_i)\text{for}i=1,\dots ,n.$$In the first case, and in the second case similarly, we obtain the family of functions (2.10) $$\begin{array}{cc}\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left(\sum _{i=1}^n{w}_j^{(i)}\text{sgn}(x_i)x_i^2+{\theta }_j\right)\text{with}\hfill \\ {\alpha }_j,{\theta }_j\hfill & \in \mathbb{R}\text{and}{\mathbf{w}}_j\in {\mathbb{R}}^n.\hfill \end{array}$$(2.10) We call these functions signed squared neural networks. Note, here $\overrightarrow{p}\in {\mathbb{R}}^{n_{*}}$ with $n_{*}=(n+2)N$.

2. (CUNN): Let (2.11) $$f^{(i)}(\overrightarrow{x})=x_i^3\text{for}i=1,\dots ,n.$$(2.11) We obtain the family of functions (2.12) $$\Psi \left[\overrightarrow{p}\right](\overrightarrow{x}):=\sum _{j=1}^N{\alpha }_j\sigma \left(\sum _{i=1}^n{w}_j^{(i)}{x}_i^3+{\theta }_j\right)\text{with}{\alpha }_j,{\theta }_j\in \mathbb{R}\text{and}{\mathbf{w}}_j\in {\mathbb{R}}^n.$$(2.12) We call these functions cubic neural networks. Again $\overrightarrow{p}\in {\mathbb{R}}^{n_{*}}$ with $n_{*}=(n+2)N$.

Remark 1.

1. It is obvious that generalized quadratic neural network functions and matrix constrained neural networks are more versatile than affine linear works, and thus they satisfy the universal approximation property (see [6]).

2. The constraint equation (2.8)(2.8) $${\xi }_j\ge 0\text{and}{\kappa }_j\le 0\text{for all}j=1,\dots ,N,$$(2.8) does not allow to use the universal approximation theorem in a straight forward manner. In fact we prove an approximation result indirectly via a convergence rates result (see Section 4).

3. Sign based quadratic and cubic neural networks satisfy the universal approximation property, which is proven below in Corollary 2.6 by reducing it to the classical result from [6].

In order to prove the universal approximation property of SBQNNs and CUNNs we review the universal approximation theorem as formulated by [6] first. It is based discriminatory properties of the functions σ.

Definition 2.2

(Discriminatory function). A function $\sigma :\mathbb{R}\to \mathbb{R}$ is called discriminatory (see [6]) if every measure μ on ${[0,1]}^n$, which satisfies $${\int }_{{[0,1]}^n}\sigma ({\mathbf{w}}^T\overrightarrow{x}+\theta )d\mu (\overrightarrow{x})=0\hspace{1em}\text{for all}\mathbf{w}\in {\mathbb{R}}^n\text{and}\theta \in \mathbb{R}$$implies that $\mu \equiv 0$.

Example 2.3.

Note that every non-polynomial function is discriminatory (this follows from the results in [8]). Therefore the choices of activation function in Definition 1.1 are discriminatory for the Lebesgue-measure.

With these basic concepts we are able to recall Cybenko’s universal approximation result.

Theorem 2.4.

[6] Let $\sigma :\mathbb{R}\to \mathbb{R}$ be a continuous discriminatory function. Then, for every function $g\in C({[0,1]}^n)$ and every $ϵ>0$, there exists a function (2.13) $$G_{ϵ}(\overrightarrow{x})=\sum _{j=1}^N{\alpha }_j\sigma ({\mathbf{w}}_j^T\overrightarrow{x}+{\theta }_j)\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}\text{with}N\in \mathbb{N},{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,$$(2.13) satisfying $$|G_{ϵ}(\overrightarrow{x})-g(\overrightarrow{x})|<ϵ\text{for all}\overrightarrow{x}\in {[0,1]}^n.$$

In the following we formulate and prove a modification of Cybenko’s universal approximation result [6] for shallow generalized neural networks as introduced in Definition 1.2.

Theorem 2.5

(Generalized universal approximation theorem). Let $\sigma :\mathbb{R}\to \mathbb{R}$ be a continuous discriminatory function and assume that ${\mathbf{f}}_j:{[0,1]}^n\to {\mathbb{R}}^m,j=1,\dots ,N$ are injective (this in particular means that $n\le m$) and continuous, respectively. We denote $\mathbf{f}=({\mathbf{f}}_1,\dots ,{\mathbf{f}}_N)$.

Then for every $g\in C({[0,1]}^n)$ and every $ϵ>0$ there exists some function $${\Psi }^{\mathbf{f}}(\overrightarrow{x}):=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T{\mathbf{f}}_j(\overrightarrow{x})+{\theta }_j\right)\text{with}{\alpha }_j,{\theta }_j\in \mathbb{R}\text{and}{\mathbf{w}}_j\in {\mathbb{R}}^m$$satisfying $$\left|{\Psi }^{\mathbf{f}}(\overrightarrow{x})-g(\overrightarrow{x})\right|<ϵ\text{for all}\overrightarrow{x}\in {[0,1]}^n.$$

Proof.

We begin the proof by noting that since $\overrightarrow{x}\to {\mathbf{f}}_j(\overrightarrow{x})$ is injective, the inverse function on the range of ${\mathbf{f}}_j$ is well-defined, and we write ${\mathbf{f}}_j{}^{-1}:{\mathbf{f}}_j({[0,1]}^n)\subseteq {\mathbb{R}}^m\to {[0,1]}^n\subseteq {\mathbb{R}}^n$. The proof that ${\mathbf{f}}_j{}^{-1}$ is continuous relies on the fact that the domain ${[0,1]}^n$ of ${\mathbf{f}}_j$ is compact, see for instance [26, Chapter XI, Theorem 2.1]. Then applying the Tietze-Urysohn-Brouwer extension theorem (see [27]) to the continuous function $g°{\mathbf{f}}_j{}^{-1}:{\mathbf{f}}_j({[0,1]}^n)\to \mathbb{R}$, we see that this function can be extended continuously to ${\mathbb{R}}^m$. This extension will be denoted by $g^{*}:{\mathbb{R}}^m\to \mathbb{R}$.

We apply Theorem 2.4 to conclude that there exist ${\alpha }_j,{\theta }_j\in \mathbb{R}$ and ${\mathbf{w}}_j\in {\mathbb{R}}^m,j=1,\dots ,N$ such that $${\Psi }^{*}(\mathbf{z}):=\sum _{j=1}^N{\alpha }_j\sigma ({\mathbf{w}}_j^T\mathbf{z}+{\theta }_j)\text{for all}\mathbf{z}\in {\mathbb{R}}^m,$$which satisfies (2.14) $$\left|{\Psi }^{*}(\mathbf{z})-g^{*}(\mathbf{z})\right|<ϵ\text{for all}\mathbf{z}\in {[0,1]}^m.$$(2.14)

Then, because ${\mathbf{f}}_j$ maps into ${\mathbb{R}}^m$ we conclude, in particular, that $$\begin{array}{cc}{\Psi }^{*}({\mathbf{f}}_j(\overrightarrow{x}))\hfill & =\sum _{j=1}^N{\alpha }_j\sigma ({\mathbf{w}}_j^T{\mathbf{f}}_j(\overrightarrow{x})+{\theta }_j)\hfill \end{array}$$and $$\begin{array}{c}\left|{\Psi }^{*}({\mathbf{f}}_j(\overrightarrow{x}))-g(\overrightarrow{x})\right|=\left|{\Psi }^{*}({\mathbf{f}}_j(\overrightarrow{x}))-g^{*}({\mathbf{f}}_j(\overrightarrow{x}))\right|<ϵ.\hfill \end{array}$$

Therefore ${\Psi }^{\mathbf{f}}(·):={\Psi }^{*}({\mathbf{f}}_j(·))$ satisfy the claimed assertions. □

It is obvious that the full variability in $\mathbf{w}$ is the key to bring our proof and the universal approximation theorem in context. That is, if w_j, θ_j, $j=1,\dots ,N$ are allowed to vary over ${\mathbb{R}}^n,\mathbb{R}$, respectively. RQNNs are constrained to ${\theta }_j<{\Vert {\overrightarrow{w}}_j\Vert }^2/(4{\xi }_j)$ in equation (2.7)(2.7) $${\overrightarrow{y}}_j=-\frac{1}{2{\xi }_j}{\widehat{\mathbf{w}}}_j^T\text{and}{\kappa }_j={\theta }_j-\frac{{\Vert {\widehat{\mathbf{w}}}_j\Vert }^2}{4{\xi }_j}.$$(2.7) and thus Theorem 2.5 does not apply. Interestingly Theorem 4.6 applies and allows to approximate functions in ${\mathcal{L}}^1({\mathbb{R}}^n)$ (even with rates).

Corollary 2.6

(Universal approximation properties of SBQNNs and CUNNs). Let the discriminatory function $\sigma :\mathbb{R}\to \mathbb{R}$ be Lipschitz continuous. All families of neural network functions from Definition 2.1 satisfy the universal approximation property on ${[0,1]}^n$.

Proof.

The proof follows from Theorem 2.5 and noting that all our functions ${\mathbf{f}}_j,j=1,\dots ,N$ defined in Definition 2.1 are injective. □

3 Motivation

3.1 Motivation 1: the Shepp-Logan phantom

Almost all tomographic algorithms are tested with the Shepp-Logan phantom (see [28]). It can be exactly represented with 10 characteristic functions on ellipses, and therefore it is much better approximated by a GQNN than with an ALNN or a DNN, respectively. This observation extends immediately to all function that contains “localized” features, because of the compactness of the characteristic sets. In this sense sparse approximations with ALNNs and DNNs would be more costly in practice.

3.2 Motivation 2: the Gauss-Newton iteration

Let us assume that $F:\mathbf{X}\to \mathbf{Y}$ is a linear operator. We consider the solution $\mathbf{p}$ of equation (1.2)(1.2) $$F(\mathbf{p})=\mathbf{y},$$(1.2) to be an element of an ALNN or DNN, as defined in Definition 1.1. Therefore $\mathbf{p}$ can be parameterized by a vector $\overrightarrow{p}$ as in equation (1.4)(1.4) $$\overrightarrow{p}=[{\alpha }_1,\dots ,{\alpha }_N;{\mathbf{w}}_1,\dots ,{\mathbf{w}}_N;{\theta }_1,\dots ,{\theta }_N]\in {\mathbb{R}}^{n_{*}}\text{with}{n}_{*}=(n+2)N$$(1.4) (for shallow linear neural networks) or equation (1.6)(1.6) $$\overrightarrow{p}=[\alpha ;{\mathbf{w}}_{1,1},\dots ,{\mathbf{w}}_{N_L,L};{\theta }_{1,1},\dots ,{\theta }_{N_L,L}]\in {\mathbb{R}}^{n_{*}}.$$(1.6) (for deep neural networks). For GQNNs the adequate parameterization can be read out from Definition 2.1. In other words, we can write every searched for function $\mathbf{p}$ via an operator $\Psi$ which maps a parameter $\overrightarrow{p}\in {\mathbb{R}}^{n_{*}}$ to a function in $\mathbf{X}$, i.e. $$\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x}).$$

Therefore, we aim for solving the nonlinear operator equation (3.1) $$\mathcal{N}(\overrightarrow{p}):=F°\Psi \left[\overrightarrow{p}\right]=\mathbf{y}.$$(3.1)

For ALNNs we have proven in [3] that the Gauss-Newton method (see equation (3.7)(3.7) $$\begin{array}{c}{\overrightarrow{p}}^{k+1}={\overrightarrow{p}}^k-\mathcal{N}\prime {({\overrightarrow{p}}^k)}^{†}(\mathcal{N}({\overrightarrow{p}}^k)-\mathbf{y})\hspace{1em}k\in {\mathbb{N}}_0,\hfill \end{array}$$(3.7) ) is locally, quadratically convergent under conditions, which guarantee that during the iteration the gradients of $\Psi$ does not degenerate, i.e. that $\mathbf{P}$ is a finite-dimensional manifold. Let us now calculate derivatives of the radial neural network operators $\Psi$ (see equation (2.5)(2.5) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,{\xi }_j\in \mathbb{R}.\hfill \end{array}$$(2.5) ), which are the basis to prove that also the Gauss-Newton with RQNNs is convergent:

Example 3.1.

Let $\sigma :\mathbb{R}\to \mathbb{R}$ be continuous activation function, where all derivatives up to order 2 are uniformly bounded, i.e. in formulas (3.2) $$\sigma \in C^2(\mathbb{R};\mathbb{R})\cap {\mathcal{B}}^2(\mathbb{R};\mathbb{R}),$$(3.2) such as the sigmoid function. Then, the derivatives of a radial neural network (RQNN) $\Psi$ as in equation (2.5)(2.5) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,{\xi }_j\in \mathbb{R}.\hfill \end{array}$$(2.5) with respect to the coefficients of $\overrightarrow{p}$ in equation (2.2)(2.2) $$\begin{array}{cc}\overrightarrow{p}\hfill & =[{\alpha }_1\dots ,{\alpha }_N;{\mathbf{w}}_1,\dots ,{\mathbf{w}}_N;A_1,\dots ,A_N;{\theta }_1,\dots ,{\theta }_N]\in {\mathbb{R}}^{n_{*}}\text{with}\hfill \\ n_{*}\hfill & =(n^2+n+2)N.\hfill \end{array}$$(2.2) are given by the following formulas, where ν_s is defined in equation (2.6)(2.6) $$\begin{array}{c}{\nu }_j(\overrightarrow{x}):={\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\widehat{\mathbf{w}}}_j^T\overrightarrow{x}+{\theta }_j={\xi }_j{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^2+{\kappa }_j\hfill \end{array}$$(2.6) :

• Derivative with respect to α_s, $s=1,\dots ,N$: (3.3) $$\begin{array}{cc}\frac{\partial \Psi }{\partial {\alpha }_s}\left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & =\sigma ({\nu }_s)\text{for}s=1,\dots ,N.\hfill \end{array}$$(3.3)

• Derivative with respect to $w_s^{(t)}$ where $s=1,\dots ,N,t=1,\dots ,n$: (3.4) $$\begin{array}{cc}\frac{\partial \Psi }{\partial w_s^{(t)}}\left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & =\sum _{j=1}^N{\alpha }_j\sigma \prime ({\nu }_j){\delta }_{s=j}{x}_t={\alpha }_s\sigma \prime ({\nu }_s)x_t.\hfill \end{array}$$(3.4)

• Derivative with respect to θ_s where $s=1,\dots ,N$: (3.5) $$\begin{array}{cc}\frac{\partial \Psi }{\partial {\theta }_s}\left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & =\sum _{j=1}^N{\alpha }_j\sigma \prime ({\nu }_j){\delta }_{s=j}={\alpha }_s\sigma \prime ({\nu }_s).\hfill \end{array}$$(3.5)

• Derivative with respect ${\xi }_s$: (3.6) $$\frac{\partial \Psi }{\partial {\xi }_s}\left[\overrightarrow{p}\right](\overrightarrow{x})={\alpha }_s\sigma \prime ({\nu }_s){\Vert \overrightarrow{x}\Vert }^2.$$(3.6) Note, that all the derivatives above are functions in $\mathbf{X}=L^2({[0,1]}^n)$.

For formulating a convergence result for the Gauss-Newton method (see equation (3.7)(3.7) $$\begin{array}{c}{\overrightarrow{p}}^{k+1}={\overrightarrow{p}}^k-\mathcal{N}\prime {({\overrightarrow{p}}^k)}^{†}(\mathcal{N}({\overrightarrow{p}}^k)-\mathbf{y})\hspace{1em}k\in {\mathbb{N}}_0,\hfill \end{array}$$(3.7) ) we need to specify the following assumptions, which were postulated first in [3] to prove local convergence of a Gauss-Newton method on a set of shallow affine linear network functions. Here we verify convergence of the Gauss-Newton method on a set of RQNNs, as defined in equation (2.1)(2.1) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\overrightarrow{x}}^T{A}_j\overrightarrow{x}+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,A_j\in {\mathbb{R}}^{n\times n}.\hfill \end{array}$$(2.1) . The proof is completely analogous as in [3].

Assumption 3.2.

• $F:\mathbf{X}=L^2({[0,1]}^n)\to \mathbf{Y}$ be a linear, bounded operator with trivial nullspace and dense range.

• $\Psi :\mathcal{D}(\Psi )\subseteq {\mathbb{R}}^{(n+2)N}\to \mathbf{X}$ is a shallow RQNN generated by a strictly monotonic activation function σ which satisfies equation (3.2)(3.2) $$\sigma \in C^2(\mathbb{R};\mathbb{R})\cap {\mathcal{B}}^2(\mathbb{R};\mathbb{R}),$$(3.2) , like a sigmoid function.

• All derivatives in equations (3.3)–(3.6)(3.6) $$\frac{\partial \Psi }{\partial {\xi }_s}\left[\overrightarrow{p}\right](\overrightarrow{x})={\alpha }_s\sigma \prime ({\nu }_s){\Vert \overrightarrow{x}\Vert }^2.$$(3.6) are locally linearly independent functions.

Now, we recall a local convergence result:

Theorem 3.3

(Local convergence of Gauss-Newton method with RQNNs). Let $\mathcal{N}$ be as in equation (3.1)(3.1) $$\mathcal{N}(\overrightarrow{p}):=F°\Psi \left[\overrightarrow{p}\right]=\mathbf{y}.$$(3.1) be the composition of a linear operator F and the RQNN network operator as defined in equation (2.5)(2.5) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,{\xi }_j\in \mathbb{R}.\hfill \end{array}$$(2.5) , which satisfy Assumption 3.2. Moreover, let ${\overrightarrow{p}}^0\in \mathcal{D}(\Psi )$, which is open, be the starting point of the Gauss-Newton iteration (3.7) $$\begin{array}{c}{\overrightarrow{p}}^{k+1}={\overrightarrow{p}}^k-\mathcal{N}\prime {({\overrightarrow{p}}^k)}^{†}(\mathcal{N}({\overrightarrow{p}}^k)-\mathbf{y})\hspace{1em}k\in {\mathbb{N}}_0,\hfill \end{array}$$(3.7) where $\mathcal{N}\prime {({\overrightarrow{p}}^k)}^{†}$ denotes the Moore-Penrose inverse of $\mathcal{N}\prime ({\overrightarrow{p}}^k)$.¹

Moreover, let ${\overrightarrow{p}}^{†}\in \mathcal{D}(\Psi )$ be a solution of equation (3.1)(3.1) $$\mathcal{N}(\overrightarrow{p}):=F°\Psi \left[\overrightarrow{p}\right]=\mathbf{y}.$$(3.1) , i.e. (3.8) $$\mathcal{N}({\overrightarrow{p}}^{†})=\mathbf{y}.$$(3.8)

Then the Gauss-Newton iterations are locally, that is if ${\overrightarrow{p}}^0$ is sufficiently close to ${\overrightarrow{p}}^{†}$, and quadratically converging.

Remark 2.

The essential condition in Theorem 3.3 is that all derivatives of $\Psi$ with respect to $\overrightarrow{p}$ are linearly independent, which is a nontrivial research question. In [3] we studied convergence of the Gauss-Newton method on a set of shallow affine linear neural networks, where the convergence result required linear independence of the derivatives specified in equations (3.3)–(3.5)(3.5) $$\begin{array}{cc}\frac{\partial \Psi }{\partial {\theta }_s}\left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & =\sum _{j=1}^N{\alpha }_j\sigma \prime ({\nu }_j){\delta }_{s=j}={\alpha }_s\sigma \prime ({\nu }_s).\hfill \end{array}$$(3.5) . Here, for RQNNs, on top, linear independence of the 2nd order moment function equation (3.6)(3.6) $$\frac{\partial \Psi }{\partial {\xi }_s}\left[\overrightarrow{p}\right](\overrightarrow{x})={\alpha }_s\sigma \prime ({\nu }_s){\Vert \overrightarrow{x}\Vert }^2.$$(3.6) needs to hold. Even linear independence of neural network functions (with derivatives) is still a challenging research topic (see [30]).

Under our assumptions the ill–posedness of F does not affect convergence of the Gauss-Newton method. The generalized inverse completely annihilates F in the iteration. The catch however is, that the data has to be attained in a finite dimensional space spanned by neurons (that is equation (3.8)(3.8) $$\mathcal{N}({\overrightarrow{p}}^{†})=\mathbf{y}.$$(3.8) holds). This assumption is in fact restrictive: Numerical tests show that convergence of Gauss-Newton methods becomes arbitrarily slow if it is violated. Here the ill-posedness of F enters. The proof of Theorem 3.3 is based on the affine covariant condition (see for instance [31]).

In the following we show the complexity of the convergence condition of a Gauss-Newton method in the case when affine linear deep neural network functions are used for coding.

Example 3.4

(Convergence conditions of deep neural networks). We restrict our attention to the simple case n = 1. We consider a 4-layer DNN (consisting of two internal layers) with ${\sigma }_1={\sigma }_2=\sigma$, which reads as follows: (3.9) $$x\to \Psi \left[\overrightarrow{p}\right](x):=\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}\sigma \left(w_{j_2,2}\left(\sum _{j_1=1}^{N_1}{\alpha }_{j_1,1}\sigma \left(w_{j_1,1}x+{\theta }_{j_1,1}\right)\right)+{\theta }_{j_2,2}\right).$$(3.9)

Now, we calculate by chain rule $\frac{\partial \Psi }{\partial w_{1,1}}[\overrightarrow{p}](x)$. For this purpose we define $$w_{1,1}\to \rho (w_{1,1}):=\sum _{j_1=1}^{N_1}{\alpha }_{j_1,1}\sigma \left(w_{j_1,1}x+{\theta }_{j_1,1}\right).$$

With this definition we rewrite equation (3.9)(3.9) $$x\to \Psi \left[\overrightarrow{p}\right](x):=\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}\sigma \left(w_{j_2,2}\left(\sum _{j_1=1}^{N_1}{\alpha }_{j_1,1}\sigma \left(w_{j_1,1}x+{\theta }_{j_1,1}\right)\right)+{\theta }_{j_2,2}\right).$$(3.9) to $$x\to \Psi \left[\overrightarrow{p}\right](x):=\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}\sigma \left(w_{j_2,2}\rho (w_{1,1})+{\theta }_{j_2,2}\right),$$

Since $$\rho \prime (w_{1,1}):={\alpha }_{j_1,1}\sigma \prime \left(w_{1,1}x+{\theta }_{1,1}\right)x,$$

we therefore get (3.10) $$\begin{array}{cc}\frac{\partial \Psi }{\partial w_{1,1}}\hfill & =\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}\sigma \prime \left(w_{j_2,2}\rho (w_{1,1})+{\theta }_{j_2,2}\right)w_{j_2,2}\rho \prime (w_{1,1})\hfill \\ \hfill & ={\alpha }_{j_1,1}x\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}{w}_{j_2,2}\sigma \prime \left(w_{j_2,2}\rho (w_{1,1})+{\theta }_{j_2,2}\right)\sigma \prime \left(w_{1,1}x+{\theta }_{1,1}\right).\hfill \end{array}$$(3.10)

Note that $\Psi [\overrightarrow{p}]$ is a function of x depending on the parameter $\overrightarrow{p}$, which contains $w_{1,1}$ as one component.

The complicating issue in a potential convergence analysis concerns the last identity, equation (3.10)(3.10) $$\begin{array}{cc}\frac{\partial \Psi }{\partial w_{1,1}}\hfill & =\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}\sigma \prime \left(w_{j_2,2}\rho (w_{1,1})+{\theta }_{j_2,2}\right)w_{j_2,2}\rho \prime (w_{1,1})\hfill \\ \hfill & ={\alpha }_{j_1,1}x\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}{w}_{j_2,2}\sigma \prime \left(w_{j_2,2}\rho (w_{1,1})+{\theta }_{j_2,2}\right)\sigma \prime \left(w_{1,1}x+{\theta }_{1,1}\right).\hfill \end{array}$$(3.10) , where evaluations of $\sigma \prime$ at different arguments appear simultaneously, which makes it complicated to analyze and interpret linear independence of derivatives of $\Psi$ with respect to the single elements of $\overrightarrow{p}$. Note that for proving convergence of the Gauss-Newton method, according to equation (3.10)(3.10) $$\begin{array}{cc}\frac{\partial \Psi }{\partial w_{1,1}}\hfill & =\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}\sigma \prime \left(w_{j_2,2}\rho (w_{1,1})+{\theta }_{j_2,2}\right)w_{j_2,2}\rho \prime (w_{1,1})\hfill \\ \hfill & ={\alpha }_{j_1,1}x\sum _{j_2=1}^{N_2}{\alpha }_{j_2,2}{w}_{j_2,2}\sigma \prime \left(w_{j_2,2}\rho (w_{1,1})+{\theta }_{j_2,2}\right)\sigma \prime \left(w_{1,1}x+{\theta }_{1,1}\right).\hfill \end{array}$$(3.10) , we need to verify linear independence of then functions $$\begin{array}{cc}x\to \hfill & \sigma \prime \left(w_{j_2,2}\rho (w_{1,1})+{\theta }_{j_2,2}\right)\sigma \prime \left(w_{1,1}x+{\theta }_{1,1}\right)\hfill \\ \hfill & =\sigma \prime \left(w_{j_2,2}\left(\sum _{j_1=1}^{N_1}{\alpha }_{j_1,1}\sigma \left(w_{j_1,1}x+{\theta }_{j_1,1}+{\theta }_{j_2,2}\right)\right)\right)\sigma \prime \left(w_{1,1}x+{\theta }_{1,1}\right).\hfill \end{array}$$

In other word, the potential manifold of quadratic neural networks is extremely complex.

The conclusion of this section is that the analysis of iterative regularization methods, like a Gauss-Newton method, gets significantly more complicated if deep neural networks are used as ansatz functions. In contrast using neural network with higher order nodes (such as radial) results in transparent moment conditions. So the research question discussed in the following is whether shallow higher order neural networks have similar approximation properties than deep neural network functions, which would reveal clear benefits of such for analyzing iterative methods for solving ill–posed problems.

4 Convergence rates for universal approximation of RQNNs

In the following we prove convergence rates of RQNNs (as defined in equation (2.5)(2.5) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,{\xi }_j\in \mathbb{R}.\hfill \end{array}$$(2.5) ) in the ${\mathcal{L}}^1$-norm. To be precise we specify a subclass on RQNNs for which we prove convergence rates results. This is a much finer result than the standard universal approximation result, Theorem 2.5, since it operates on a subclass and provides convergence rates. However, it is the only approximation result so far, which we can provide. We recall that the constraints, ${\xi }_j\ne 0$ and ${\kappa }_j/{\xi }_j<0$ (see equation (2.7)(2.7) $${\overrightarrow{y}}_j=-\frac{1}{2{\xi }_j}{\widehat{\mathbf{w}}}_j^T\text{and}{\kappa }_j={\theta }_j-\frac{{\Vert {\widehat{\mathbf{w}}}_j\Vert }^2}{4{\xi }_j}.$$(2.7) ), induce that the level-sets of the neurons are compact (circles) and not unbounded, as they are for shallow affine linear neurons. Therefore we require a different analysis (and different spaces) than in the most advanced and optimal convergence rates results for affine linear neural networks as for instance in [10, 11]. In fact the analysis is closer to an analysis of compact wavelets.

The convergence rate will be expressed in the following norm:

Definition 4.1.

${\mathcal{L}}^1$ denotes the space of square integrable functions on ${[0,1]}^n$, which satisfy [32, (1.10)] $$\Vert f{\Vert }_{{\mathcal{L}}^1}:=\mathrm{\text{inf}}\left\{\sum _{g\in D}\left|c_g\right|:f=\sum _{g\in D}{c}_{g}g\right\}<\infty .$$

Here c_g are the coefficients of an wavelet expansion and D is a countable, general set of wavelet functions. A discrete wavelet basis (see, for example, [33–36]) is a (not necessarily orthonormal) basis of the space of square integrable functions, where the basis elements are given by dilating and translating a mother wavelet function.

Remark 3.

Notice that the notation ${\mathcal{L}}^1$ does not refer to the common L¹-function space of absolutely integrable functions and depends on the choice of the wavelet system. For more properties and details on this space see [37, Remark 3.11].

Definition 4.2

(Circular scaling function and wavelets). Let r > 0 be a fixed constant and σ be a discriminatory function as defined in Definition 2.2 such that $\left|{\int }_{{\mathbb{R}}^n}\sigma (r^2-\Vert \overrightarrow{x}{\Vert }^2)d\overrightarrow{x}\right|<\infty$. Then let (4.1) $$\overrightarrow{x}\in {\mathbb{R}}^n\to \phi (\overrightarrow{x}):=C_n\sigma (r^2-{\Vert \overrightarrow{x}\Vert }^2),$$(4.1) where C_n is a normalizing constant such that ${\int }_{{\mathbb{R}}^n}\phi (\overrightarrow{x})d\overrightarrow{x}=1$.

Then we define for $k\in \mathbb{Z}$ the radial scaling functions and wavelets (4.2) $$\begin{array}{cc}(\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to S_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y}))\text{and}\hfill \\ (\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to {\psi }_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^{-k/2}(S_k^C(\overrightarrow{x},\overrightarrow{y})-S_{k-1}^C(\overrightarrow{x},\overrightarrow{y})).\hfill \end{array}$$(4.2)

Often $\overrightarrow{y}\in {\mathbb{R}}^n$ is considered a parameter and we write synonymously (4.3) $$\begin{array}{c}\overrightarrow{x}\to S_{k,\overrightarrow{y}}^C(\overrightarrow{x})=S_k^C(\overrightarrow{x},\overrightarrow{y})\text{and}\overrightarrow{x}\to {\psi }_{k,\overrightarrow{y}}^C(\overrightarrow{x})={\psi }_k^C(\overrightarrow{x},\overrightarrow{y}).\hfill \end{array}$$(4.3)

In particular, this notation means that $S_{k,\overrightarrow{y}}^C$ and ${\psi }_{k,\overrightarrow{y}}^C$ are considered solely functions of the variable $\overrightarrow{x}$.

We consider the following subset of RQNNs (4.4) $${\mathcal{S}}_d^C:=\left\{S_{k,\overrightarrow{y}}^C:k\in \mathbb{Z}\text{and}\overrightarrow{y}\in 2^{-k/n}{\mathbb{Z}}^n\right\}.$$(4.4)

This is different to standard universal approximation theorems, where an uncountable number of displacements are considered. Moreover, we define the discrete Wavelet space (4.5) $${\mathcal{W}}_d^C:=\left\{{\psi }_{k,\overrightarrow{y}}^C:=2^{-k/2}\left(S_{k,\overrightarrow{y}}^C-S_{k-1,\overrightarrow{y}}^C\right):k\in \mathbb{Z}\text{and}\overrightarrow{y}\in 2^{-k/n}{\mathbb{Z}}^n\right\}.$$(4.5)

According to Theorem A.3, in order to prove the following approximation result, we only require to verify that ${\mathcal{S}}_d^C$ satisfies the conditions for symmetric AtI and the double Lipschitz condition (see Definition A.1), which originate from [24, Def. 3.4].

Corollary 4.3.

${\mathcal{W}}_d^C$ is a frame and for every function $f\in {\mathcal{L}}^1({\mathbb{R}}^n)$ there exists a linear combination of N elements of ${\mathcal{W}}_d^C$, denoted by f_N, satisfying (4.6) $${\Vert f-f_N\Vert }_{L^2}\le {\Vert f\Vert }_{{\mathcal{L}}^1}{(N+1)}^{-1/2}.$$(4.6)

For proving the approximation to identity, AtI, property of the functions $S_k^C,k\in \mathbb{Z}$, we use the following basic inequality.

Lemma 4.4.

Let $h:{\mathbb{R}}^n\to \mathbb{R}$ be a twice differentiable function, which can be expressed in the following way: $$h(\overrightarrow{x})=h_s({\Vert \overrightarrow{x}\Vert }^2)\text{for all}\overrightarrow{x}\in {\mathbb{R}}^n.$$

Then the spectral norm of the Hessian of h can be estimated as follows:² (4.7) $$\Vert {\nabla }^{2}h(\overrightarrow{x})\Vert \le \mathrm{\text{max}}\left\{\left|4{\Vert \overrightarrow{x}\Vert }^2{h}_s^{\prime }\prime ({\Vert \overrightarrow{x}\Vert }^2)+2h_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2)\right|,\left|2h_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2)\right|\right\}.$$(4.7)

Proof.

Since ${\nabla }^{2}h(\overrightarrow{x})$ is a symmetric matrix, its operator norm is equal to its spectral radius, namely the largest absolute value of an eigenvalue. By routine calculation we can see that $${\nabla }_{x_i{x}_j}h(\overrightarrow{x})=4x_i{x}_j{h}_s^{\prime }\prime ({\Vert \overrightarrow{x}\Vert }^2)+2{\delta }_{\mathit{\text{ij}}}{h}_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2).$$ $$4h_s^{\prime }\prime ({\Vert \overrightarrow{x}\Vert }^2)C\overrightarrow{z}=(-2h_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2)+\lambda )\overrightarrow{z}.$$

Or in other words $\frac{-2h_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2)+\lambda }{4h_s^{\prime }\prime ({\Vert x\Vert }^2)}$ is an eigenvalue of C. Moreover, $C=\overrightarrow{x}{\overrightarrow{x}}^T$ is a rank one matrix and thus the spectral values are 0 with multiplicity $(n-1)$ and ${\Vert \overrightarrow{x}\Vert }^2$. This in turn shows that the eigenvalues of the Hessian are $+2h_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2)$ (with multiplicity n – 1) and $4{\Vert \overrightarrow{x}\Vert }^2{h}_s^{\prime }\prime ({\Vert \overrightarrow{x}\Vert }^2)+2h_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2)$, which proves equation (4.7)(4.7) $$\Vert {\nabla }^{2}h(\overrightarrow{x})\Vert \le \mathrm{\text{max}}\left\{\left|4{\Vert \overrightarrow{x}\Vert }^2{h}_s^{\prime }\prime ({\Vert \overrightarrow{x}\Vert }^2)+2h_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2)\right|,\left|2h_s^{\prime }({\Vert \overrightarrow{x}\Vert }^2)\right|\right\}.$$(4.7) . □

In the following lemma, we will prove that the kernels ${(S_k^C)}_{k\in \mathbb{Z}}$ are an AtI.

Lemma 4.5.

Let r > 0 be fixed. Suppose that the activation function $\sigma :\mathbb{R}\to \mathbb{R}$ is monotonically increasing and satisfies for the i-th derivative (i = 0, 1, 2) (4.8) $$\left|{\sigma }^i(r^2-t^2)\right|\le C_{\sigma }{(1+{\left|t\right|}^n)}^{-1-(2i+1)/n}\text{for all}t\in \mathbb{R}.$$(4.8)

Then the kernels ${(S_k^C)}_{k\in \mathbb{Z}}$ as defined in equation (4.2)(4.2) $$\begin{array}{cc}(\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to S_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y}))\text{and}\hfill \\ (\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to {\psi }_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^{-k/2}(S_k^C(\overrightarrow{x},\overrightarrow{y})-S_{k-1}^C(\overrightarrow{x},\overrightarrow{y})).\hfill \end{array}$$(4.2) form an AtI as defined in Definition A.1 that also satisfy the double Lipschitz condition equation (A.4)(A.4) $$\begin{array}{cc}\hfill & \left|S_k(\overrightarrow{x},\overrightarrow{y})-S_k(\overrightarrow{x}\prime ,\overrightarrow{y})-S_k(\overrightarrow{x},\overrightarrow{y}\prime )+S_k(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )\right|\hfill \\ \le \hfill & \tilde{C}{\left(\frac{C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }{\left(\frac{C_{\rho }{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }\frac{2^{-kϵ}}{{(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{1+ϵ}}\hfill \end{array}$$(A.4) .

Proof.

We verify the three conditions from Definition A.1 as well as equation (A.4)(A.4) $$\begin{array}{cc}\hfill & \left|S_k(\overrightarrow{x},\overrightarrow{y})-S_k(\overrightarrow{x}\prime ,\overrightarrow{y})-S_k(\overrightarrow{x},\overrightarrow{y}\prime )+S_k(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )\right|\hfill \\ \le \hfill & \tilde{C}{\left(\frac{C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }{\left(\frac{C_{\rho }{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }\frac{2^{-kϵ}}{{(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{1+ϵ}}\hfill \end{array}$$(A.4) . First of all, we note that (4.9) $$\left|{\sigma }^i(r^2-{\Vert \overrightarrow{x}\Vert }^2)\right|\le C_{\sigma }{(1+{\Vert \overrightarrow{x}\Vert }^n)}^{-1-(2i+1)/n}\text{for all}\overrightarrow{x}\in {\mathbb{R}}^n.$$(4.9)

• Verification of (i) in Definition A.1: equations (4.1)(4.1) $$\overrightarrow{x}\in {\mathbb{R}}^n\to \phi (\overrightarrow{x}):=C_n\sigma (r^2-{\Vert \overrightarrow{x}\Vert }^2),$$(4.1) and 4.8(4.1) $$\overrightarrow{x}\in {\mathbb{R}}^n\to \phi (\overrightarrow{x}):=C_n\sigma (r^2-{\Vert \overrightarrow{x}\Vert }^2),$$(4.1) imply that (4.10) $$\begin{array}{cc}0\le \phi (\overrightarrow{x}-\overrightarrow{y})\hfill & =C_n\sigma (r^2-{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^2)\le C_{\sigma }{C}_n{(1+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{-1-1/n}\hfill \\ \hfill & \mathrm{\hspace{1em}}\mathrm{\hspace{1em}}\text{for all}\overrightarrow{x},\overrightarrow{y}\in {\mathbb{R}}^n.\hfill \end{array}$$(4.10) Therefore $$\begin{array}{cc}{S}_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & =2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y}))\le C_{\sigma }{C}_n{2}^k{(1+2^k{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{-1-1/n}\hfill \\ \hfill & =C_{\sigma }{C}_n{2}^{-k/n}{(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{-1-1/n}.\hfill \end{array}$$Thus (i) in Definition A.1 holds with $ϵ=1/n$ and $C_{\rho }=1$ and $C=C_n{C}_{\sigma }$.

• Verification of (ii) in Definition A.1 with $C_{\rho }=1$ and $C_A=2^{-n}$: Because σ is monotonically increasing it follows from equation (4.1)(4.1) $$\overrightarrow{x}\in {\mathbb{R}}^n\to \phi (\overrightarrow{x}):=C_n\sigma (r^2-{\Vert \overrightarrow{x}\Vert }^2),$$(4.1) and the fact that $S_0(\overrightarrow{x},\overrightarrow{y})=\phi (\overrightarrow{x}-\overrightarrow{y})$ (see equation (4.2)(4.2) $$\begin{array}{cc}(\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to S_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y}))\text{and}\hfill \\ (\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to {\psi }_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^{-k/2}(S_k^C(\overrightarrow{x},\overrightarrow{y})-S_{k-1}^C(\overrightarrow{x},\overrightarrow{y})).\hfill \end{array}$$(4.2) ) and Definition 4.2 that $$F_{\overrightarrow{y}}(\overrightarrow{x}):=\Vert {\nabla }_{\overrightarrow{x}}(S_0(\overrightarrow{x},\overrightarrow{y}))\Vert =2C_n\Vert \overrightarrow{x}-\overrightarrow{y}\Vert \sigma \prime (r^2-{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^2)\text{for all}\overrightarrow{y}\in {\mathbb{R}}^n.$$Then equation (4.9)(4.9) $$\left|{\sigma }^i(r^2-{\Vert \overrightarrow{x}\Vert }^2)\right|\le C_{\sigma }{(1+{\Vert \overrightarrow{x}\Vert }^n)}^{-1-(2i+1)/n}\text{for all}\overrightarrow{x}\in {\mathbb{R}}^n.$$(4.9) implies that $$\begin{array}{cc}{F}_{\overrightarrow{y}}(\overrightarrow{x})\hfill & \le 2C_n{C}_{\sigma }{(1+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{-1-3/n}\Vert \overrightarrow{x}-\overrightarrow{y}\Vert \hfill \\ \hfill & \le 2C_n{C}_{\sigma }{(1+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{-1-3/n}{(1+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{\frac{1}{n}}\hfill \\ \hfill & =2C_n{C}_{\sigma }{(1+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{-1-2/n}.\hfill \end{array}$$From the definition of $S_k^C(\overrightarrow{x},\overrightarrow{y})$, it follows (4.11) $$\begin{array}{cc}\Vert {\nabla }_{\overrightarrow{x}}(S_k^C(\overrightarrow{x},\overrightarrow{y}))\Vert \hfill & =\Vert {\nabla }_{\overrightarrow{x}}(2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y})))\Vert =2^k\Vert {\nabla }_{\overrightarrow{x}}{S}_0(2^{k/n}\overrightarrow{x},2^{k/n}\overrightarrow{y})\Vert \hfill \\ \hfill & =2^{k+k/n}{F}_{2^{k/n}\overrightarrow{y}}(2^{k/n}\overrightarrow{x})\hfill \\ \hfill & \le 2^{-k/n}{C}_n{C}_{\sigma }{(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{-1-2/n}.\hfill \end{array}$$(4.11) From the mean value theorem it therefore follows from equation (4.11)(4.11) $$\begin{array}{cc}\Vert {\nabla }_{\overrightarrow{x}}(S_k^C(\overrightarrow{x},\overrightarrow{y}))\Vert \hfill & =\Vert {\nabla }_{\overrightarrow{x}}(2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y})))\Vert =2^k\Vert {\nabla }_{\overrightarrow{x}}{S}_0(2^{k/n}\overrightarrow{x},2^{k/n}\overrightarrow{y})\Vert \hfill \\ \hfill & =2^{k+k/n}{F}_{2^{k/n}\overrightarrow{y}}(2^{k/n}\overrightarrow{x})\hfill \\ \hfill & \le 2^{-k/n}{C}_n{C}_{\sigma }{(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{-1-2/n}.\hfill \end{array}$$(4.11) and equation (A.6)(A.6) $${\Vert \overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}\Vert }^n\ge 2^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-2^{-n}{2}^{-k}.$$(A.6) that (4.12) $$\begin{array}{cc} \hfill & \frac{\left|S_k^C(\overrightarrow{x},\overrightarrow{y})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y})\right|}{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }\hfill \\ \le \hfill & {\underset{\left\{\overrightarrow{z}=t\overrightarrow{x}\prime +(1-t)\overrightarrow{x}:t\in [0,1]\right\}}{\mathrm{\text{max}}}}_{}\Vert {\nabla }_{\overrightarrow{x}}(S_k^C(\overrightarrow{z},\overrightarrow{y}))\Vert \hfill \\ \le \hfill & 2^{-k/n}{C}_n{C}_{\sigma }{\underset{\left\{\overrightarrow{z}=\overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x}):t\in [0,1]\right\}}{\mathrm{\text{max}}}}_{}{(2^{-k}+{\Vert \overrightarrow{z}-\overrightarrow{y}\Vert }^n)}^{-1-2/n}\hfill \\ =\hfill & 2^{-k/n}{C}_n{C}_{\sigma }{\left(2^{-k}+{\underset{\left\{\overrightarrow{z}=\overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x}):t\in [0,1]\right\}}{\mathrm{\text{min}}}}_{}{\Vert \overrightarrow{z}-\overrightarrow{y}\Vert }^n\right)}^{-1-2/n}.\hfill \end{array}$$(4.12) Then application of equation (A.6)(A.6) $${\Vert \overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}\Vert }^n\ge 2^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-2^{-n}{2}^{-k}.$$(A.6) and noting that $\frac{1-2^{-n}}{2^{-n}}\ge 1$ gives $$\begin{array}{cc} \hfill & \frac{\left|S_k^C(\overrightarrow{x},\overrightarrow{y})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y})\right|}{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }\hfill \\ \le \hfill & 2^{-k/n}{C}_n{C}_{\sigma }{\left((1-2^{-n})2^{-k}+2^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right)}^{-1-2/n}\hfill \\ \le \hfill & 2^{-k/n}{\left(2^{-n}\right)}^{-1-2/n}{C}_n{C}_{\sigma }{\left(\frac{(1-2^{-n})}{2^{-n}}{2}^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right)}^{-1-2/n}\hfill \\ \le \hfill & 2^{-k/n}{2}^{n+2}{C}_n{C}_{\sigma }{\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right)}^{-1-2/n}.\hfill \end{array}$$Therefore (ii) is satisfied with $C_{\rho }=1,\zeta =1/n$, $ϵ=1/n$ and $C=2^{n+2}{C}_n{C}_{\sigma }$.

• Verification of (iii) in Definition A.1: From the definition of $S_k^C$ (see equation (4.2)(4.2) $$\begin{array}{cc}(\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to S_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y}))\text{and}\hfill \\ (\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to {\psi }_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^{-k/2}(S_k^C(\overrightarrow{x},\overrightarrow{y})-S_{k-1}^C(\overrightarrow{x},\overrightarrow{y})).\hfill \end{array}$$(4.2) ) it follows that for every $k\in \mathbb{Z}$ and $\overrightarrow{y}\in {\mathbb{R}}^n$ $$1={\int }_{{\mathbb{R}}^n}{S}_k^C(\overrightarrow{x},\overrightarrow{y})d\overrightarrow{x}={\int }_{{\mathbb{R}}^n}{2}^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y}))d\overrightarrow{x}.$$

• Verification of the double Lipschitz condition equation (A.4)(A.4) $$\begin{array}{cc}\hfill & \left|S_k(\overrightarrow{x},\overrightarrow{y})-S_k(\overrightarrow{x}\prime ,\overrightarrow{y})-S_k(\overrightarrow{x},\overrightarrow{y}\prime )+S_k(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )\right|\hfill \\ \le \hfill & \tilde{C}{\left(\frac{C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }{\left(\frac{C_{\rho }{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }\frac{2^{-kϵ}}{{(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{1+ϵ}}\hfill \end{array}$$(A.4) in Definition A.1: By using the integral version of the mean value theorem, we have $$S_k^C(\overrightarrow{x},\overrightarrow{y})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y})-S_k^C(\overrightarrow{x},\overrightarrow{y}\prime )+S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )$$ $$=S_k^C(\overrightarrow{x},\overrightarrow{y})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y})-(S_k^C(\overrightarrow{x},\overrightarrow{y}\prime )-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y}\prime ))$$ $$={\int }_0^1〈{\nabla }_{\overrightarrow{y}}{S}_k^C(\overrightarrow{x},\overrightarrow{y}\prime +t(\overrightarrow{y}-\overrightarrow{y}\prime )),\overrightarrow{y}-\overrightarrow{y}\prime 〉\mathit{\text{dt}}$$ $$\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}-{\int }_0^1〈{\nabla }_{\overrightarrow{y}}{S}_k^C(\overrightarrow{x}\prime ,\overrightarrow{y}\prime +t(\overrightarrow{y}-\overrightarrow{y}\prime )),\overrightarrow{y}-\overrightarrow{y}\prime 〉\mathit{\text{dt}}$$ $$={\int }_0^1{\int }_0^1〈{\nabla }_{\overrightarrow{x},\overrightarrow{y}}{S}_k^C(\overrightarrow{x}\prime +s(\overrightarrow{x}-\overrightarrow{x}\prime ),\overrightarrow{y}\prime +t(\overrightarrow{y}-\overrightarrow{y}\prime ))(\overrightarrow{x}-\overrightarrow{x}\prime ),\overrightarrow{y}-\overrightarrow{y}\prime 〉\mathit{\text{dtds}}.$$Following this identity, we get (4.13) $$\begin{array}{cc} \hfill & \frac{\left|S_k^C(\overrightarrow{x},\overrightarrow{y})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y})+S_k^C(\overrightarrow{x},\overrightarrow{y}\prime )-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )\right|}{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert \Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }\hfill \\ \le \hfill & {\mathrm{\text{max}}}_{\mathtt{I}}\Vert {\nabla }_{\overrightarrow{y}}\left(\frac{S_k^C(\overrightarrow{x},\overrightarrow{z})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{z})}{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }\right)\Vert \le {\mathrm{\text{max}}}_{\mathtt{Q}}\Vert {\nabla }_{\overrightarrow{x}\overrightarrow{y}}^2{S}_k^C(\overrightarrow{z}\prime ,\overrightarrow{z})\Vert ,\hfill \end{array}$$(4.13) where $$\begin{array}{cc}\mathtt{I}\hfill & :=\left\{\overrightarrow{z}=t\overrightarrow{y}+(1-t)\overrightarrow{y}\prime :t\in [0,1]\right\},\hfill \\ \mathtt{Q}\hfill & :=\left\{(\overrightarrow{z}=t_{\overrightarrow{y}}\overrightarrow{y}+(1-t_{\overrightarrow{y}})\overrightarrow{y}\prime ,\overrightarrow{z}\prime =t_{\overrightarrow{x}}\overrightarrow{x}+(1-t_{\overrightarrow{x}})\overrightarrow{x}\prime ):t_{\overrightarrow{y}}\in [0,1],t_{\overrightarrow{x}}\in [0,1]\right\},\hfill \end{array}$$and $\Vert {\nabla }_{\overrightarrow{x}\overrightarrow{y}}^2{S}_k^C(\overrightarrow{z}\prime ,\overrightarrow{z})\Vert$ denotes again the spectral norm of ${\nabla }_{\overrightarrow{x}\overrightarrow{y}}^2{S}_k^C(\overrightarrow{z}\prime ,\overrightarrow{z})$.

Now, we estimate the right hand side of equation (4.13)(4.13) $$\begin{array}{cc} \hfill & \frac{\left|S_k^C(\overrightarrow{x},\overrightarrow{y})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y})+S_k^C(\overrightarrow{x},\overrightarrow{y}\prime )-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )\right|}{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert \Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }\hfill \\ \le \hfill & {\mathrm{\text{max}}}_{\mathtt{I}}\Vert {\nabla }_{\overrightarrow{y}}\left(\frac{S_k^C(\overrightarrow{x},\overrightarrow{z})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{z})}{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }\right)\Vert \le {\mathrm{\text{max}}}_{\mathtt{Q}}\Vert {\nabla }_{\overrightarrow{x}\overrightarrow{y}}^2{S}_k^C(\overrightarrow{z}\prime ,\overrightarrow{z})\Vert ,\hfill \end{array}$$(4.13) : From the definition of $S_k^C$, equation (4.2)(4.2) $$\begin{array}{cc}(\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to S_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y}))\text{and}\hfill \\ (\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to {\psi }_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^{-k/2}(S_k^C(\overrightarrow{x},\overrightarrow{y})-S_{k-1}^C(\overrightarrow{x},\overrightarrow{y})).\hfill \end{array}$$(4.2) , and the definition of $\phi$, equation (4.1)(4.1) $$\overrightarrow{x}\in {\mathbb{R}}^n\to \phi (\overrightarrow{x}):=C_n\sigma (r^2-{\Vert \overrightarrow{x}\Vert }^2),$$(4.1) , it follows with the abbreviation $\overrightarrow{\omega }=2^{k/n}(\overrightarrow{z}\prime -\overrightarrow{z})$: $$\begin{array}{cc}\Vert {\nabla }_{\overrightarrow{x}\overrightarrow{y}}^2{S}_k^C(\overrightarrow{z}\prime ,\overrightarrow{z})\Vert \hfill & =2^k\Vert {\nabla }_{\overrightarrow{x}\overrightarrow{y}}^2(\phi °(2^{k/n}·))(\overrightarrow{z}\prime -\overrightarrow{z}))\Vert =2^{k(1+2/n)}\Vert {\nabla }^2\phi (\overrightarrow{\omega }))\Vert .\hfill \end{array}$$

Applications of Lemma 4.4 with $\overrightarrow{x}\to h(\overrightarrow{x})=\phi (\overrightarrow{x})$ and $$t\to h_s(t)=C_n\sigma (r^2-t)$$shows that (note that $h_s^{\prime }(t)=-C_n\sigma \prime (r^2-t)$) (4.14) $$\begin{array}{cc}\hfill & \Vert {\nabla }^2\phi (\overrightarrow{\omega })\Vert \hfill \\ \le \hfill & C_n\mathrm{\text{max}}\left\{\left|4{\Vert \overrightarrow{\omega }\Vert }^2\sigma ″(r^2-{\Vert \overrightarrow{\omega }\Vert }^2)-2\sigma \prime (r^2-{\Vert \overrightarrow{\omega }\Vert }^2)\right|,\left|2\sigma \prime (r^2-{\Vert \overrightarrow{\omega }\Vert }^2)\right|\right\}\hfill \\ \le \hfill & 2^2{C}_n\mathrm{\text{max}}\left\{2{\Vert \overrightarrow{\omega }\Vert }^2\left|\sigma ″(r^2-{\Vert \overrightarrow{\omega }\Vert }^2)\right|,\left|\sigma \prime (r^2-{\Vert \overrightarrow{\omega }\Vert }^2)\right|\right\}.\hfill \end{array}$$(4.14)

Thus from equation (4.9)(4.9) $$\left|{\sigma }^i(r^2-{\Vert \overrightarrow{x}\Vert }^2)\right|\le C_{\sigma }{(1+{\Vert \overrightarrow{x}\Vert }^n)}^{-1-(2i+1)/n}\text{for all}\overrightarrow{x}\in {\mathbb{R}}^n.$$(4.9) it follows that $$\begin{array}{cc}\hfill & \Vert {\nabla }_{\overrightarrow{x}\overrightarrow{y}}^2{S}_k^C(\overrightarrow{z}\prime ,\overrightarrow{z})\Vert \hfill \\ \hfill & \hspace{1em}\le 2^2{2}^{k(1+2/n)}{C}_n{C}_{\sigma }\mathrm{\text{max}}\left\{2{\Vert \overrightarrow{\omega }\Vert }^2{(1+{\Vert \overrightarrow{\omega }\Vert }^n)}^{-1-5/n},{(1+{\Vert \overrightarrow{\omega }\Vert }^n)}^{-1-3/n}\right\}\hfill \\ \hfill & \hspace{1em}\le 2^3{2}^{k(1+2/n)}{C}_n{C}_{\sigma }{(1+{\Vert \overrightarrow{\omega }\Vert }^n)}^{-1-3/n}\hfill \\ \hfill & \hspace{1em}\le 2^{-k/n}{2}^3{C}_n{C}_{\sigma }{(2^{-k}+{\Vert \overrightarrow{z}\prime -\overrightarrow{z}\Vert }^n)}^{-1-3/n}.\hfill \end{array}$$

In the next step we note that from equation (A.7)(A.7) $${\Vert \overrightarrow{x}+t_{\overrightarrow{x}}(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}-t_{\overrightarrow{y}}(\overrightarrow{y}\prime -\overrightarrow{y})\Vert }^n\ge 3^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-3^{-n}{2}^{1-k}$$(A.7) it follows that (4.15) $${\Vert \overrightarrow{z}\prime -\overrightarrow{z}\Vert }^n\ge 3^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-3^{-n}{2}^{1-k}.$$(4.15)

Thus we get because $\frac{1-3^{-n}2}{3^{-n}}\ge 1$ $$\begin{array}{cc} \hfill & \frac{\left|S_k^C(\overrightarrow{x},\overrightarrow{y})-S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y})-S_k^C(\overrightarrow{x},\overrightarrow{y}\prime )+S_k^C(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )\right|}{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert \Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }\hfill \\ \le \hfill & 2^{-k/n}{2}^3{C}_n{C}_{\sigma }{\left((1-3^{-n}2)2^{-k}+3^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right)}^{-1-3/n}\hfill \\ \le \hfill & 2^{-k/n}{2}^3{(3^{-n})}^{-1-3/n}{C}_n{C}_{\sigma }{\left(\frac{1-3^{-n}2}{3^{-n}}{2}^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right)}^{-1-3/n}\hfill \\ \le \hfill & 2^{-k/n}{2}^3{3}^{n+3}{C}_n{C}_{\sigma }{\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right)}^{-1-3/n}.\hfill \end{array}$$

Therefore (iii) is satisfied with $C_{\rho }=1,\tilde{C}=2^3{3}^{n+3}{C}_n{C}_{\sigma },\zeta =1/n$, and $ϵ=1/n$.□

We have shown with Lemma 4.5 that the functions $\left\{S_k^C:k\in \mathbb{Z}\right\}$ form an AtI, which satisfies the double Lipschitz condition (see Definition A.1). Moreover, for activations functions, like the sigmoid function, equation (4.8)(4.8) $$\left|{\sigma }^i(r^2-t^2)\right|\le C_{\sigma }{(1+{\left|t\right|}^n)}^{-1-(2i+1)/n}\text{for all}t\in \mathbb{R}.$$(4.8) it follows that (4.16) $$\underset{t\to \pm \infty }{\mathrm{\text{lim}}}{(1+{\left|t\right|}^n)}^{1+(2i+1)/n}\left|{\sigma }^i(r^2-t^2)\right|=0,$$(4.16) which shows equation (3.2)(3.2) $$\sigma \in C^2(\mathbb{R};\mathbb{R})\cap {\mathcal{B}}^2(\mathbb{R};\mathbb{R}),$$(3.2) . The limit result is an easy consequence of the fact that $t\to {\mathrm{e}}^{-t^2}$ and its derivative are faster decaying to zero than the functions $t\to {(1+{\left|t\right|}^n)}^{-1-(2i+1)/n}$. Therefore, it follows from Theorem A.3 that the set ${\mathcal{W}}_d^C$ is a wavelet frame and satisfies the estimate equation (A.14)(A.14) $${\Vert f-f_N\Vert }_{L^2}\le {\Vert f\Vert }_{{\mathcal{L}}^1}{(N+1)}^{-1/2}.$$(A.14) . We summarize this important result:

Theorem 4.6

(${\mathcal{L}}^1$-convergence of radial wavelets). Let σ be an activation function that satisfies the conditions in Lemma 4.5. Then, for every function $f\in {\mathcal{L}}^1({\mathbb{R}}^n)$ (see Definition 4.1 for a definition of the space) and every $N\in \mathbb{N}$, there exists a function $$f_N\in {\text{span}}_N({\mathcal{W}}_d^C)\subseteq {\mathcal{L}}^1({\mathbb{R}}^n),$$

and ${\text{span}}_N$ denotes linear combinations of at most N terms in the set, such that (4.17) $${\Vert f-f_N\Vert }_{L^2}\le {\Vert f\Vert }_{{\mathcal{L}}^1}{(N+1)}^{-1/2}.$$(4.17)

Using Theorem 4.6 we are able to formulate the main result of this paper:

Corollary 4.7

(${\mathcal{L}}^1$-convergence of RQNNs). Let the assumptions of Theorem 4.6 be satisfied. Then, for every function $f\in {\mathcal{L}}^1({\mathbb{R}}^n)$ (see Definition 4.1) and every $N\in \mathbb{N}$, there exists a parametrization vector $$\overrightarrow{p}=[{\alpha }_1,\dots ,{\alpha }_{2N};{\mathbf{w}}_1,\dots ,{\mathbf{w}}_{2N};{\xi }_1,\dots ,{\xi }_{2N};{\theta }_1,\dots ,{\theta }_{2N}].$$such that (4.18) $${\Vert f-\Psi \left[\overrightarrow{p}\right]\Vert }_{L^2}\le {\Vert f\Vert }_{{\mathcal{L}}^1}{(N+1)}^{-1/2},$$(4.18) where $\Psi [\overrightarrow{p}]$ is a RQNN from equation (2.5)(2.5) $$\begin{array}{cc}\overrightarrow{x}\to \mathbf{p}(\overrightarrow{x})=\Psi \left[\overrightarrow{p}\right](\overrightarrow{x})\hfill & :=\sum _{j=1}^N{\alpha }_j\sigma \left({\mathbf{w}}_j^T\overrightarrow{x}+{\xi }_j{\Vert \overrightarrow{x}\Vert }^2+{\theta }_j\right)\hfill \\ \hfill & \text{with}{\alpha }_j,{\theta }_j\in \mathbb{R},{\mathbf{w}}_j\in {\mathbb{R}}^n,{\xi }_j\in \mathbb{R}.\hfill \end{array}$$(2.5) .

Proof.

From Theorem 4.6 it follows that there exists $f_N\in {\text{span}}_N({\mathcal{W}}_d^C)$, which satisfies equation (4.17)(4.17) $${\Vert f-f_N\Vert }_{L^2}\le {\Vert f\Vert }_{{\mathcal{L}}^1}{(N+1)}^{-1/2}.$$(4.17) . By definition, $$f_N(\overrightarrow{x})=\sum _{k\in \mathbb{Z}}\sum _{\overrightarrow{j}\in {\mathbb{Z}}^n}{\chi }_{f,N}(k,\overrightarrow{j}){\beta }_{k,\overrightarrow{j}}{\psi }_{k,2^{-k/n}\overrightarrow{j}}^C(\overrightarrow{x}),$$where ${\chi }_{f,N}$ denotes the characteristic function of a set of indices of size N. Using the definition of ${\psi }_k^C$ we get from equation (4.2)(4.2) $$\begin{array}{cc}(\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to S_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^k\phi (2^{k/n}(\overrightarrow{x}-\overrightarrow{y}))\text{and}\hfill \\ (\overrightarrow{y},\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\to {\psi }_k^C(\overrightarrow{x},\overrightarrow{y})\hfill & :=2^{-k/2}(S_k^C(\overrightarrow{x},\overrightarrow{y})-S_{k-1}^C(\overrightarrow{x},\overrightarrow{y})).\hfill \end{array}$$(4.2) and equation (4.1)(4.1) $$\overrightarrow{x}\in {\mathbb{R}}^n\to \phi (\overrightarrow{x}):=C_n\sigma (r^2-{\Vert \overrightarrow{x}\Vert }^2),$$(4.1) $$\begin{array}{cc}{f}_N(\overrightarrow{x})\hfill & =\sum _{k\in \mathbb{Z}}\sum _{\overrightarrow{j}\in {\mathbb{Z}}^n}{\chi }_{f,N}(k,\overrightarrow{j}){\beta }_{k,\overrightarrow{j}}{2}^{-k/2}\left(S_{k,2^{-k/n}\overrightarrow{j}}^C(\overrightarrow{x})-S_{k-1,2^{-k/n}\overrightarrow{j}}^C(\overrightarrow{x})\right)\hfill \\ \hfill & =\sum _{k\in \mathbb{Z}}\sum _{\overrightarrow{j}\in {\mathbb{Z}}^n}{\chi }_{f,N}(k,\overrightarrow{j}){\beta }_{k,\overrightarrow{j}}{2}^{-k/2}\hfill \\ \hfill & \times \left(2^k\phi (2^{k/n}(\overrightarrow{x}-2^{-k/n}\overrightarrow{j}))-2^{k-1}\phi (2^{(k-1)/n}(\overrightarrow{x}-2^{-k/n}\overrightarrow{j}))\right)\hfill \\ \hfill & =\sum _{k\in \mathbb{Z}}\sum _{\overrightarrow{j}\in {\mathbb{Z}}^n}{\chi }_{f,N}(k,\overrightarrow{j})C_n{\beta }_{k,\overrightarrow{j}}{2}^{-k/2}(2^k\sigma \left(r^2-{\Vert 2^{k/n}\left(\overrightarrow{x}-2^{-k/n}\overrightarrow{j}\right)\Vert }^2\right)\hfill \\ \hfill & -2^{k-1}\sigma \left(r^2-{\Vert 2^{(k-1)/n}\left(\overrightarrow{x}-2^{-k/n}\overrightarrow{j}\right)\Vert }^2\right)).\hfill \end{array}$$

The arguments where σ is evaluated are in general different, and thus we get $$\begin{array}{cc}{f}_N(\overrightarrow{x})\hfill & =\sum _{k\in \mathbb{Z}}\sum _{\overrightarrow{j}\in {\mathbb{Z}}^n}\underset{={\alpha }_j}{\underbrace{{\chi }_{f,N}(k,\overrightarrow{j})C_n{\beta }_{k,\overrightarrow{j}}{2}^{k/2}}}\sigma \left(\underset{={\theta }_j}{\underbrace{r^2}}\underset{={\xi }_j}{\underbrace{-4^{k/n}}}\Vert \overrightarrow{x}-\underset{={\overrightarrow{y}}_j}{\underbrace{2^{-k/n}\overrightarrow{j}}}{\Vert }^2\right)\hfill \\ \hfill & \hspace{1em}+\sum _{k\in \mathbb{Z}}\sum _{\overrightarrow{j}\in {\mathbb{Z}}^n}\underset{{\alpha }_j}{\underbrace{{\chi }_{f,N}(k,\overrightarrow{j})C_n{\beta }_{k,\overrightarrow{j}}{2}^{k/2-1}}}\sigma \left(\underset{{\theta }_j}{\underbrace{r^2}}\underset{={\xi }_j}{\underbrace{-4^{(k-1)/n}}}\Vert \overrightarrow{x}-\underset{{\overrightarrow{y}}_j}{\underbrace{2^{-k/n}\overrightarrow{j}}}{\Vert }^2\right)\hfill \end{array}$$from which the coefficients can be read out. Note that for the sake of simplicity we have not explicitly stated the exact index in the subscript of the coefficients $(\overrightarrow{\alpha },\overrightarrow{\theta },\overrightarrow{\xi },\mathbf{y})$, or in other words we use the formal relation $j=(k,\overrightarrow{j})$ to indicate the dependencies. This shows the claim if we identify ${\theta }_j=r^2$ and $${\mathbf{w}}^T=2^{1+2k/n}\overrightarrow{y}.$$

□

Remark 4.

Our results in Section 4 are proved using “similar” methods as in [37], but can not be used directly. Because the main ingredient of the proof is the construction of localizing functions as wavelets. This task requires two layers of affine linear decision functions to accomplish, since the linear combinations of functions of the form $\overrightarrow{x}\to \phi (\overrightarrow{x}):=C_n\sigma ({\overrightarrow{w}}^T\overrightarrow{x}+\theta )$ are not localized; they cannot satisfy (iii) in Definition A.1, as they are not integrable on ${\mathbb{R}}^n$. In comparison, some quadratic decision functions—such as those used in Section 4—naturally gives localized functions, and therefore can work as wavelets on their own.

We mention that there exists a zoo of approximation rates results for neural network functions (see for instance [10, 11], where best-approximation results in dependence of the space-dimension n have been proven). We have based our analysis on convergence rates results of [37] and [24]. This choice is motivated by the fact that the AtI property from [24] is universal, and allows a natural extension of their results to quadratic neural network functions. The motivation here is not to give the optimal convergence rates result for quadratic neural networks, but to verify that convergence rates results are possible. We believe that AtI is a very flexible and elegant tool.

5 Conclusion

In this paper we studied generalized neural networks functions for solving inverse problems. We proved a universal approximation theorem and we have proven convergence rates of radial neural networks, which are the same order as the classical affine linear neural network functions, but with vastly reduced numbers of elements (in particular it spares one layer). The paper presents a proof of concept and thus we restrict attention only to radial neural networks although generalizations to higher order neural networks (such as cubic) is quite straightforward.

We also make the remark about the convergence of the actual training process of a neural network, i.e. optimization of the parameters of each decision function. This is usually done by gradient descent or Gauss-Newton-type methods. From the analysis in [3] the convergence conditions of Newton’s method become very complicated for DNNs, but are transparent for quadratic neural networks, which is fact a matter of the complicated chain rule, when differentiating DNNs with respect to the parameters. For such an analysis there is a clear preference to go higher with the degree of polynomials than increasing the number of layers.

Acknowledgments

The authors would like to thank some referees for their valuable suggestions and their patience.

Additional information

Funding

This research was funded in whole, or in part, by the Austrian Science Fund (FWF) 10.55776/P34981 (OS & LF) – New Inverse Problems of Super-Resolved Microscopy (NIPSUM), SFB 10.55776/F68 (OS) “Tomography Across the Scales,” project F6807-N36 (Tomography with Uncertainties), and 10.55776/T1160 (CS) “Photoacoustic Tomography: Analysis and Numerics.” For open access purposes, the author has applied a CC BY public copyright license to any author-accepted manuscript version arising from this submission. The financial support by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development and the Christian Doppler Research Association is gratefully acknowledged.

Notes

1 For a detailed exposition on generalized inverses see [29].

2 In the following $\nabla$ and ${\nabla }^2$ (without subscripts) always denote derivatives with respect to an n-dimensional variable such as $\overrightarrow{x}$. $\prime$ and $″$ denotes derivatives of a one-dimensional function.

A

Approximation to the identity (AtI)

Definition A.

1 (Approximation to the identity [24]). A sequence of symmetric kernel functions ${(S_k:{\mathbb{R}}^n\times {\mathbb{R}}^n\to \mathbb{R})}_{k\in \mathbb{Z}}$ is said to be an approximation to the identity (AtI) if there exist a quintuple $(ϵ,\zeta ,C,C_{\rho },C_A)$ of positive numbers satisfying the additional constraints (A.1) $$0<ϵ\le \frac{1}{n},0<\zeta \le \frac{1}{n}\text{and}{C}_A<1$$(A.1) the following three conditions are satisfied for all $k\in \mathbb{Z}$:

1. $\left|S_k(\overrightarrow{x},\overrightarrow{y})\right|\le C\frac{2^{-kϵ}}{{(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{1+ϵ}}$ for all $\overrightarrow{x},\overrightarrow{y}\in {\mathbb{R}}^n$;

2. $\left|S_k(\overrightarrow{x},\overrightarrow{y})-S_k(\overrightarrow{x}\prime ,\overrightarrow{y})\right|\le C{\left(\frac{C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }\frac{2^{-kϵ}}{{(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{1+ϵ}}$for all triples $(\overrightarrow{x},\overrightarrow{x}\prime ,\overrightarrow{y})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\times {\mathbb{R}}^n$ which satisfy (A.2) $$C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n\le C_A\left(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right);$$(A.2)

3. ${\int }_{{\mathbb{R}}^n}{S}_k(\overrightarrow{x},\overrightarrow{y})d\overrightarrow{y}=1$ for all $\overrightarrow{x}\in {\mathbb{R}}^n$.

Moreover, we say that the AtI satisfies the double Lipschitz condition if there exist a triple $(\tilde{C},{\tilde{C}}_A,\zeta )$ of positive constants satisfying (A.3) $${\tilde{C}}_A<\frac{1}{2},$$(A.3) such that for all $k\in \mathbb{Z}$ (A.4) $$\begin{array}{cc}\hfill & \left|S_k(\overrightarrow{x},\overrightarrow{y})-S_k(\overrightarrow{x}\prime ,\overrightarrow{y})-S_k(\overrightarrow{x},\overrightarrow{y}\prime )+S_k(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )\right|\hfill \\ \le \hfill & \tilde{C}{\left(\frac{C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }{\left(\frac{C_{\rho }{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }\frac{2^{-kϵ}}{{(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{1+ϵ}}\hfill \end{array}$$(A.4) for all quadruples $(\overrightarrow{x},\overrightarrow{x}\prime ,\overrightarrow{y},\overrightarrow{y}\prime )\in {\mathbb{R}}^n\times {\mathbb{R}}^n\times {\mathbb{R}}^n\times {\mathbb{R}}^n$ which satisfy (A.5) $$C_{\rho }\mathrm{\text{max}}\left\{{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n,{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n\right\}\le {\tilde{C}}_A\left(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.5)

The conditions (ii) and equation (A.5)(A.5) $$C_{\rho }\mathrm{\text{max}}\left\{{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n,{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n\right\}\le {\tilde{C}}_A\left(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.5) are essential for our analysis. We characterize now geometric properties of these constrained sets:

Lemma A.2.

Let $C_{\rho }=1,C_A=2^{-n}$ and ${\tilde{C}}_A=3^{-n}$.

Then set of triples $(\overrightarrow{x},\overrightarrow{x}\prime ,\overrightarrow{y})$ which satisfy equation (A.2)(A.2) $$C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n\le C_A\left(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right);$$(A.2) and for which ${\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\ge 2^{-k}$ and all $t\in [0,1]$ satisfy (A.6) $${\Vert \overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}\Vert }^n\ge 2^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-2^{-n}{2}^{-k}.$$(A.6)

• The set of quadrupels $(\overrightarrow{x},\overrightarrow{x}\prime ,\overrightarrow{y},\overrightarrow{y}\prime )$ which satisfy equation (A.5)(A.5) $$C_{\rho }\mathrm{\text{max}}\left\{{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n,{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n\right\}\le {\tilde{C}}_A\left(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.5) and for which ${\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\ge 2^{-k}$ satisfy (A.7) $${\Vert \overrightarrow{x}+t_{\overrightarrow{x}}(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}-t_{\overrightarrow{y}}(\overrightarrow{y}\prime -\overrightarrow{y})\Vert }^n\ge 3^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-3^{-n}{2}^{1-k}$$(A.7)

for all $t_{\overrightarrow{x}},t_{\overrightarrow{y}}\in [0,1]$.

Proof.

• With the concrete choice of parameters C_A, $C_{\rho }$ equation (A.2)(A.2) $$C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n\le C_A\left(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right);$$(A.2) reads as follows (A.8) $${\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n\le 2^{-n}\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.8) Since we assume that ${\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\ge 2^{-k}$ it follows from equation (A.8)(A.8) $${\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n\le 2^{-n}\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.8) that $$\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert \le 2^{-1}\Vert \overrightarrow{x}-\overrightarrow{y}\Vert .$$In particular $\Vert \overrightarrow{x}-\overrightarrow{y}\Vert -\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert \ge 0$.We apply Jensen’s inequality, which states that for $a,b\ge 0$ (A.9) $$a^n+b^n\ge 2^{1-n}{(a+b)}^n.$$(A.9) We use $a=\Vert \overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}\Vert$ and $b=\Vert t(\overrightarrow{x}\prime -\overrightarrow{x})\Vert$, which then (along with the triangle inequality) gives $${\Vert \overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}\Vert }^n+{\Vert t(\overrightarrow{x}\prime -\overrightarrow{x})\Vert }^n\ge 2^{1-n}{\left(\Vert \overrightarrow{x}-\overrightarrow{y}\Vert \right)}^n.$$In other words, it follows from equation (A.8)(A.8) $${\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n\le 2^{-n}\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.8) that $$\begin{array}{cc}{\Vert \overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}\Vert }^n\hfill & \ge 2^{1-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-t^n{\Vert \overrightarrow{x}\prime -\overrightarrow{x}\Vert }^n\hfill \\ \hfill & \ge 2^{1-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-{\Vert \overrightarrow{x}\prime -\overrightarrow{x}\Vert }^n\hfill \\ \hfill & \ge 2^{1-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-2^{-n}\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right)\hfill \\ \hfill & =2^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-2^{-n}{2}^{-k}.\hfill \end{array}$$

• With the concrete choice of parameters ${\tilde{C}}_A,C_{\rho }$ equation (A.5)(A.5) $$C_{\rho }\mathrm{\text{max}}\left\{{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n,{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n\right\}\le {\tilde{C}}_A\left(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.5) reads as follows (A.10) $$\mathrm{\text{max}}\left\{{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n,{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n\right\}\le 3^{-n}\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.10) Since we assume that ${\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\ge 2^{-k}$ it follows from equation (A.10)(A.10) $$\mathrm{\text{max}}\left\{{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n,{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n\right\}\le 3^{-n}\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.10) that $$\mathrm{\text{max}}\left\{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert ,\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert \right\}\le 3^{-1}\Vert \overrightarrow{x}-\overrightarrow{y}\Vert .$$This in particular shows that $$\Vert \overrightarrow{x}-\overrightarrow{y}\Vert -\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert -\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert \ge 0.$$We apply Jensen’s inequality, which states that for $a,b,c\ge 0$ (A.11) $$a^n+b^n+c^n\ge 3^{1-n}{(a+b+c)}^n.$$(A.11) We use $a=\Vert \overrightarrow{x}+t_{\overrightarrow{x}}(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}-t_{\overrightarrow{y}}(\overrightarrow{y}\prime -\overrightarrow{y})\Vert ,b=\Vert t_{\overrightarrow{x}}(\overrightarrow{x}\prime -\overrightarrow{x})\Vert$ and $c=\Vert t_{\overrightarrow{y}}(\overrightarrow{y}\prime -\overrightarrow{y})\Vert$, which then (along with the triangle inequality) gives $$\begin{array}{cc}\hfill & {\Vert \overrightarrow{x}+t_{\overrightarrow{x}}(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}-t_{\overrightarrow{y}}(\overrightarrow{y}\prime -\overrightarrow{y})\Vert }^n+{\Vert t_{\overrightarrow{x}}(\overrightarrow{x}\prime -\overrightarrow{x})\Vert }^n\hfill \\ \hfill & \hspace{1em}+{\Vert t_{\overrightarrow{y}}(\overrightarrow{y}\prime -\overrightarrow{y})\Vert }^n\ge 3^{1-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n.\hfill \end{array}$$In other words, it follows from equation (A.10)(A.10) $$\mathrm{\text{max}}\left\{{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n,{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n\right\}\le 3^{-n}\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right).$$(A.10) that $$\begin{array}{cc}\hfill & {\Vert \overrightarrow{x}+t_{\overrightarrow{x}}(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}-t_{\overrightarrow{y}}(\overrightarrow{y}\prime -\overrightarrow{y})\Vert }^n\hfill \\ \hfill & \ge 3^{1-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-t_{\overrightarrow{x}}^n{\Vert \overrightarrow{x}\prime -\overrightarrow{x}\Vert }^n-t_{\overrightarrow{y}}^n{\Vert \overrightarrow{y}\prime -\overrightarrow{y}\Vert }^n\hfill \\ \hfill & \ge 3^{1-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-{\Vert \overrightarrow{x}\prime -\overrightarrow{x}\Vert }^n-{\Vert \overrightarrow{y}\prime -\overrightarrow{y}\Vert }^n\hfill \\ \hfill & \ge 3^{1-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-3^{-n}2\left(2^{-k}+{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n\right)\hfill \\ \hfill & =3^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-3^{-n}{2}^{1-k}.\hfill \end{array}$$

□

The approximation to the identity in Definition A.1 can be used to construct wavelet frames that can approximate arbitrary functions in ${\mathcal{L}}^1({\mathbb{R}}^n)$, as shown in the theorem below.

Remark 5.

When ${\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n<2^{-k}$, equation (A.6)(A.6) $${\Vert \overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}\Vert }^n\ge 2^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-2^{-n}{2}^{-k}.$$(A.6) also holds since the right hand side of equation (A.6)(A.6) $${\Vert \overrightarrow{x}+t(\overrightarrow{x}\prime -\overrightarrow{x})-\overrightarrow{y}\Vert }^n\ge 2^{-n}{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n-2^{-n}{2}^{-k}.$$(A.6) is negative, so this inequality is trivial.

Theorem A.

3 ([37]). Let ${(S_k:{\mathbb{R}}^n\times {\mathbb{R}}^n\to \mathbb{R})}_{k\in \mathbb{Z}}$ be a symmetric AtI which satisfies the double Lipschitz condition (see equation (A.4)(A.4) $$\begin{array}{cc}\hfill & \left|S_k(\overrightarrow{x},\overrightarrow{y})-S_k(\overrightarrow{x}\prime ,\overrightarrow{y})-S_k(\overrightarrow{x},\overrightarrow{y}\prime )+S_k(\overrightarrow{x}\prime ,\overrightarrow{y}\prime )\right|\hfill \\ \le \hfill & \tilde{C}{\left(\frac{C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{x}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }{\left(\frac{C_{\rho }{\Vert \overrightarrow{y}-\overrightarrow{y}\prime \Vert }^n}{2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n}\right)}^{\zeta }\frac{2^{-kϵ}}{{(2^{-k}+C_{\rho }{\Vert \overrightarrow{x}-\overrightarrow{y}\Vert }^n)}^{1+ϵ}}\hfill \end{array}$$(A.4) ). Let (A.12) $${\psi }_{k,\overrightarrow{y}}(\overrightarrow{x}):=2^{-k/2}\left(S_k(\overrightarrow{x},\overrightarrow{y})-S_{k-1}(\overrightarrow{x},\overrightarrow{y})\right)\text{for all}\overrightarrow{x},\overrightarrow{y}\in {\mathbb{R}}^n\text{and}k\in \mathbb{Z}.$$(A.12)

Then the set of functions (A.13) $$\mathcal{W}:=\left\{\overrightarrow{x}\to {\psi }_{k,\overrightarrow{b}}(\overrightarrow{x}):k\in \mathbb{Z},\overrightarrow{b}\in 2^{-k/n}{\mathbb{Z}}^n\right\},$$(A.13) is a frame and for every function $f\in {\mathcal{L}}^1({\mathbb{R}}^n)$ there exists a linear combination of N elements of $\mathcal{W}$, denoted by f_N, satisfying (A.14) $${\Vert f-f_N\Vert }_{L^2}\le {\Vert f\Vert }_{{\mathcal{L}}^1}{(N+1)}^{-1/2}.$$(A.14)