Deep impulse control: application to interest rate intervention

Abstract

We propose a deep learning framework for impulse control problems involving multivariate stochastic processes, which can be controllable or uncontrollable. We use this framework to estimate central bank interventions on the (controllable) interest rate to stabilize the (uncontrollable) inflation rate, where the two rates are correlated and cointegrated. This method is useful for small banks or insurance companies with high exposure to Treasury securities to predict and stress-test their potential losses from central bank interventions. We also study the mathematical properties of the proposed framework.

Keywords:

• Central bank intervention
• Deep learning
• Inflation
• Impulse control

1. Introduction

Monetary policy is one of the most important ways in which the government can affect the speed and direction of economic growth through central banking (Friedman 2000). For example, the United States Congress directs the Federal Reserve (Fed) to pursue several economic goals, including maximum employment, stable prices, and moderate long-term interest. The Fed uses monetary policy consisting of actions and communications to achieve these goals.¹ Central banks have multiple ways of controlling economic activity, and recent research shows that one of the direct instruments that the central bank can use to do so is to choose between a short-term interest rate and a reserve quantity (Friedman 2000). In this paper, we focus on interest rate interventions, as they are closely linked to the 2023 small bank failures in the United States, such as Silicon Valley Bank (SVB). In an interview with Douglas Diamond² , winner of the 2022 Nobel Prize for his research on bank runs, the reporter writes,

Douglas Diamond argues that the Fed's choice to signal long-term low interest rates, and then suddenly reverse course by raising interest rates in response to inflation, is a major reason for the collapse at Silicon Valley Bank….

Diamond pointed out that even in the Fed's 2022 stress tests, banks were not tested at treasury yield rates above 2%. Although SVB was not subject to a stress test, it likely would have passed under those parameters.

However, due to rapid inflation beginning in the middle of 2021, the Fed began raising interest rates quickly in 2022. Today, the effective Fed funds rate is 4.57%. This sudden reversal, which according to Diamond was not well-telegraphed by the Fed, is the reason that the market value of SVB's securities began to plummet.

This motivates us to use contemporary technology to predict interest rate interventions to control inflation and set a realistic stress test target for financial institutions.

There is a rich body of literature on interest rate dynamics in an open market, such as instantaneous interest rate models (Vasicek 1977, Hull and White 1990, Black and Karasinski 1991, Cox et al. 2005) and corresponding empirical studies (Chan et al. 1992, Chapman and Pearson 2000). In Guttentag (1969), the importance of central bank intervention is highlighted in addition to the effect of market forces on interest rates. In 2022, the Fed launched seven large-scale interventions on the federal funds rate, causing substantial changes in the interest rate market after each intervention Footnote 1. Some studies explore central bank interventions on interest rates, such as interest rate targeting behavior (Rudebusch 1995), the Fed's target rate on interest rate dynamics (Balduzzi et al. 1997), and its affect on the yield curve (Piazzesi 2005). We specifically discuss studies of optimal interest rate interventions (Cadenillas and Zapatero 1999, 2000, Feng and Muthuraman 2010, Mitchell et al. 2014). All of these studies formulate optimal interventions as impulse control problems. The optimal interventions derived are then based on tractable models and interest rate targets. Like these studies, we consider optimal impulse control. However, unlike them, our task is to develop a deep learning approach to estimate optimal interest rate interventions to control inflation under a general class of models. In this way, our consideration proposed framework can accommodate a wider range of stochastic models and a more flexible economic target, such as the inflation rate target.

In this paper, we consider the situation in which the central bank is able to intervene on several interest rates, such as short-term and long-term rates, to affect the targeted economic variable (e.g. the inflation rate), which is correlated and cointegrated with the interest rates (Booth and Ciner 2001). Thus, the cost function of the central bank is related to the targeted value of the economic variable and the cost of intervention. The central bank then aims to find a control policy to minimize the cost of deviation from the target level and the cost of intervention. In reality, the central bank determines the best time to announce the change in fund rates. Decisions about the timing and extent of intervention constitute the impulse control policy (Constantinides and Richard 1978, Sulem 1986). Under a Brownian filtration, there are computational methods for impulse control problems (Harrison et al. 1983, Feng and Muthuraman 2010). Some studies examine impulse control of exchange and interest rates with Ornstein-Uhlenbeck (OU) processes (Cadenillas and Zapatero 1999, 2000, Mitchell et al. 2014). However, solutions for high-dimensional impulse control problems encounter tremendous challenge in terms of analytical tractability. The conventional approaches encounter challenges when it comes to formulating quasi-variational inequalities for high-dimensional impulse control problems, and the inherent uncontrollable nature of the inflation rate further exacerbates the complexity of the problem. As a consequence, prior studies often (if not always) assume a pre-specified target interest rate and derive analytical solutions with one-dimensional impulse control techniques. In reality, the central bank intervenes the interest rate without knowing the target interest rate but the target inflation rate. When the previous impulse control research concentrates on infinite horizon problems for mathematical tractability, the central bank may want to complete the task in a finite horizon. Machine learning techniques, especially deep learning, offer computational feasibility for high-dimensional quantitative finance problems (Tsang and Wong 2020, Lo and Singh 2023, Mikkilä and Kanniainen 2023, Na and Wan 2023, Yin and Wong 2023). Our deep learning framework can be applied to a more general model setup and can accommodate higher dimensional problems, and handle the finite horizon problems. And some of our numerical comparisons are made against the literature to show the accuracy of our computation.

Our major contribution to the literature is the development of a deep learning framework for impulse control problems, which is based on the deep optimal stopping framework of Becker et al. (2019) and Jia et al. (2023) and additional layers for learning intervention actions. When we apply our framework to real data, our approach predicts Fed funds rate hikes consistent with the Fed's real interventions in terms of aggregate increase, although our predictions are more volatile than those of the real interventions. We suggest that such a discrepancy is associated with the Fed's intention to avoid volatile adjustments, but the aggregate increase in fund rates is similar when the inflation target is reached. Therefore, our framework also contributes to the practical stress test target of interest rate products. In addition, the application of impulse control can go beyond central bank intervention problems, such as inventory control (Cadenillas et al. 2010) and reinsurance strategy (Yan et al. 2022b).

The remainder of this paper is organized as follows. Section 2 presents the problem formulation and the cost function. Section 3 introduces our machine learning framework, including the neural network (NN) architecture, the approximation method, and the training procedure. The numerical study in section 4 examines the accuracy of our proposed method and the effectiveness of preventing the class vanishing problem where the neural networks are easily to be trapped into the local optimal in a one-dimensional setup. We then perform a real case study of taming inflation by intervening on interest rates. We also demonstrate the use of our proposed framework when the central bank applies impulse control to two interest rates. Finally, section 6 concludes the paper.

2. Problem formulation

Let $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$ denote a filtered probability space and T>0 a deterministic fixed future time. The $\mathbb{F}=\{{\mathcal{F}}_t{\}}_{0\le t\le T}$ is the usual filtration. The ${\mathbb{Z}}_{+}$ denotes the set of positive integers and ${\mathbb{R}}_{+}$ is the set of positive real numbers.

Following (Mitchell et al. 2014), we postulate that the interest rate follows the Ito process. There are several differences between our model and previous research (Cadenillas et al. 2010, Mitchell et al. 2014). First, $R_t\in {\mathbb{R}}^{d_1}$ is a multi-dimensional rather than one-dimensional stochastic process with a general form on which impulse control can be applied. Second, we add a multi-dimensional stochastic process, $I_t\in {\mathbb{R}}^{d_2}$, which is cointegrated with $R_t$. However, impulse control cannot be applied to $I_t$. For example, when $I_t$ indicates the inflation rate for which the central bank has a target range, the central bank can only intervene on the interest rate to monitor the inflation rate. Third, to mimic real-world settings, we assume that impulse control can only be applied to a set of pre-fixed time points $\{t_1,t_2,\dots ,t_N\}\equiv \mathcal{T}$ with a fixed action set $\mathcal{C}$. This is a reasonable assumption for controlling the interest rate because the central bank only changes its monetary policy at regular meetings, and interest rate changes are usually made based on a set of fixed actions rather than $\mathbb{R}$. Fourth, we consider a fixed termination time T>0 rather than ∞ because the members of the Federal Open Market Committee change every year. Therefore, it is reasonable to assume that the objective function will change accordingly.

Under the physical measure, the uncontrolled stochastic process is defined as (1) $\begin{array}{rl}& \mathrm{d}{I}_t={\mu }_1(R_t,I_t)\mathrm{d}t+{\sigma }_1(R_t,I_t)\mathrm{d}{W}_t^1,\\ & \mathrm{d}{R}_t={\mu }_2(R_t,I_t)\mathrm{d}t+{\sigma }_2(R_t,I_t)\mathrm{d}{W}_t^2.\end{array}$(1) This general formulation can easily incorporate several commonly used stochastic processes, including the Cox–Ingersoll–Ross process (Cox et al. 2005) and the OU process (Vasicek 1977).

The impulse control applied to $R_t$ is a sequence of immediate (upward or downward) changes in $R_t$ by a certain nonzero amount. We define impulse control v as a sequence of pairs of $({\tau }_i,{\xi }_i{)}_{i=1}^z$, where z is the number of impulse controls applied, $0\le {\tau }_1<{\tau }_2<\cdots <{\tau }_z\le T$ are non-decreasing stopping times belonging to $\mathcal{T}$, and ${\xi }_i\in \mathcal{C}$ is the corresponding control amount for ${\tau }_i$. Given a specific impulse control $v=(({\tau }_1,{\xi }_1),({\tau }_2,{\xi }_2),\dots ,({\tau }_z,{\xi }_z))$, the stochastic processes in (1(1) $\begin{array}{rl}& \mathrm{d}{I}_t={\mu }_1(R_t,I_t)\mathrm{d}t+{\sigma }_1(R_t,I_t)\mathrm{d}{W}_t^1,\\ & \mathrm{d}{R}_t={\mu }_2(R_t,I_t)\mathrm{d}t+{\sigma }_2(R_t,I_t)\mathrm{d}{W}_t^2.\end{array}$(1) ) becomes (2) $\begin{array}{rl}{I}_t& =i_0+{\int }_0^t{\mu }_1(R_s,I_S)d_s+{\int }_0^t{\sigma }_1(R_s,I_s)\mathrm{d}{W}_s,\\ R_t& =r_0+{\int }_0^t{\mu }_2(R_s,I_s)d_s+{\int }_0^t{\sigma }_2(R_s,I_s)\mathrm{d}{W}_s\\ & +\sum _{i=1}^z{I}_{\{{\tau }_i<t\}}{\xi }_i.\end{array}$(2) After modeling the stochastic process, we define the cost and objective functions for the impulse control problem. We assume that the central bank can quantify its preferences for $(R_t,I_t)$ and its aversion to intervening. This indicates that the central bank can evaluate its preferences for $(R_t,I_t)$ through a running cost function and its aversion through an action cost function. The running cost and action cost functions measure the central bank's preferences and aversion rather than the real money it pays, which is similar to the settings in Lohmann (1992) and Cadenillas and Zapatero (1999). We adopt the general function $\varphi (t,R_t,I_t)$ to describe the running cost rate at time t with $(R_t,I_t)$. The action cost of intervention is represented by the function $G(\xi )$ for each time with an action of size ξ. In this paper, we consider a finite time impulse control problem with maturity T and a discount factor β for future costs. By combining these two costs, the total cost to the central bank of impulse control v with an initial value $(r_0,i_0)$ is (3) $\begin{array}{rl}{J}_{r_0,i_0}(v)& =\mathbb{E}[\phantom{\sum _{i=1}^z}{\int }_0^T{e}^{-\mathrm{\beta t}}\varphi (t,R_t,I_t)\mathrm{d}t\\ & +\sum _{i=1}^z{e}^{-\beta {\tau }_i}G({\xi }_i)|R_0=r_0,I_0=i_0].\end{array}$(3) The ultimate goal of the central bank is to find an optimal policy to minimize the objective function J. Assuming that $\varphi ,G>0$, we are interested in impulse control v such that (4) $J_{r_0,i_0}(v)<\mathrm{\infty }.$(4) The collection of all impulse controls is defined as the admissible control set $\mathcal{A}$. The value function V is then given by (5) $V(r_0,i_0)=\underset{v\in \mathcal{A}}{min}{J}_{r_0,i_0}(v).$(5) The general forms of ϕ and G are consistent with previous research on the impulse control of interest rates (Mitchell et al. 2014). The key assumption in our method is that the stochastic processes $(R_t,I_t)$ are Markovian. In many applications, this assumption is not restrictive because Markovian processes can be realized by including past information. There are also restrictions on ϕ so that the solutions are non-trivial. Our method does not rely on these restrictions, and interested readers may refer to the following studies (Constantinides and Richard 1978, Feng and Muthuraman 2010, Mitchell et al. 2014).

3. Deep impulse control framework

Studies of impulse control are generally designed for one-dimensional cases, while the cointegrated process $I_t$ has yet to be considered (Cadenillas et al. 2010, Mitchell et al. 2014). Traditional methods encounter difficulties for higher dimensional problems when impulse control can be applied to the multi-dimensional interest rate vector $R_t$, and the incorporation of vector $I_t$ makes the problem even more difficult. For example, the central bank can control both short-term and long-term rates to control inflation. To handle the general impulse control problem defined in the previous section, we develop a novel NN to identify the optimal impulse control policy, which is inspired by Becker et al. (2019).

3.1. Express impulse control using neural networks

For impulse control v, two decisions must be made at each time point $t\in \mathcal{T}$: whether v is needed at time t, and its magnitude if so. We consider a larger set of actions ${\mathcal{C}}^{\ast }\equiv \mathcal{C}\cup \{0\}$ to combine these two decisions. The non-intervention decision is regarded as an impulse control of magnitude 0. In this approach, the number of impulse controls z corresponds to the cardinal number of $\mathcal{T}$ denoted N in the rest of the paper.

For $n=1,2,\dots N$, the optimal impulse control at time $t_n$ should generally be based on the whole process of $(I,R)$ from time 0 to time $t_n$. However, only $(R_{t_n},I_{t_n})$ is sufficient to make the decision at time $t_n$ because of the Markov property of the process. Let us assume that the optimal impulse control decision at time $t_n$ is $f_n(R_{t_n},I_{t_n})$ for the measurable function $f_n:{\mathbb{R}}^{d_1+d_2}\to \{e_1,e_2\dots ,e_M\}$. $e_i$ is the one-hot vector taking a value of 1 in the ith coordinate and 0 otherwise, and M is the cardinal number of ${\mathcal{C}}^{\ast }$. The optimal control policy can be represented by a sequence of $f_n(R_{t_n},I_{t_n})$ for $n=1,2\dots ,N$. The idea is to approximate $f_n$ through an NN $f^{{\theta }_n}$ whose input is the pair $R_{t_n},I_{t_n}$ and the output is a one-hot vector. Then, we can obtain the control policy $v^{\theta }\equiv \{f^{{\theta }_1},f^{{\theta }_2},\dots ,f^{{\theta }_N}\}$.

3.2. Neural network approximation

The numerical method approximates $f_n$ with an NN $f^{{\theta }_n}$ for $n\in \{1,2,\dots ,N\}$ through iterative backward training. Before considering the approximation procedure, we define some notations for better illustration. Consider an auxiliary problem (6) $V_n=\underset{v\in {\mathcal{A}}_n}{min}J(v),$(6) where ${\mathcal{A}}_n$ is the set of all admissible controls in which ${\xi }_i=0$ for $i=1,2\dots ,n-1$. We define the expected future cost condition on $(R_{t_n},I_{t_n})$ as (7) $\begin{array}{rl}& J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v)\\ & =\mathbb{E}[{\int }_{t_n^{-}}^T{e}^{-\mathrm{\beta t}}\varphi (t,R_t,I_t)\mathrm{d}t+\sum _{i=n}^N{e}^{-\beta {\tau }_i}G({\xi }_i)|R_{t_n^{-}},I_{t_n^{-}}].\end{array}$(7) Let the magnitude of the m-th intervention on R be ${\xi }^m$. For a fixed $n\in \{1,2,\dots ,N\}$, a given control in ${\mathcal{A}}_n$ is defined as a sequence of measurable functions $f_n,f_{n+1},\dots ,f_N$. The following proposition supports the iterative backward training method.

Proposition 3.1

For a specific $n\in \{0,1,\dots ,N-1\}$, let $v_{n+1}$ be an impulse control policy generated by measurable functions $f_{n+1},\dots ,f_N$. Then, there exists a measurable function $f_n$ such that the impulse control policy given by $f_n,f_{n+1},\dots ,f_N$ satisfies $\begin{array}{rl}J(v_n)-V(n)& \le \sum _{m=1}^M\mathbb{E}[\phantom{\underset{v\in {\mathcal{A}}_{n+1}}{min}}{J}^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v_{n+1})\\ & -\underset{v\in {\mathcal{A}}_{n+1}}{min}{J}^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)],\end{array}$ where $V_n$ and $V_{n+1}$ are defined as in (6(6) $V_n=\underset{v\in {\mathcal{A}}_n}{min}J(v),$(6) ).

Proof.

Let $\begin{array}{rl}{ϵ}_m& =\mathbb{E}[\phantom{\underset{v\in {\mathcal{A}}_{n+1}}{min}}{J}^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v_{n+1})\\ & -\underset{v\in {\mathcal{A}}_{n+1}}{min}{J}^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)]\end{array}$ with $v_n\in {\mathcal{A}}_n$ as the impulse control policy. Then, we have $\begin{array}{rl}{V}_{n+1}-J(v_{n+1})& =\underset{v\in {\mathcal{A}}_{n+1}}{min}J(v)-J(v_{n+1})\\ & =\underset{v\in {\mathcal{A}}_{n+1}}{min}\mathbb{E}[J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v)]\\ & -\mathbb{E}[J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v_{n+1})]\end{array}$ where the cost from time 0 to time $t_n^{-}$ is the same and is therefore subsumed. Let $\begin{array}{rl}{D}_m& =\{J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v_{n+1})\\ & +e^{-\beta t_n}G({\xi }^m)\text{ be the minimum for }\mathrm{\forall \xi }\in {\mathcal{C}}^{\ast }\}\\ E_m& =\{{\xi }_n={\xi }^m\}.\end{array}$ and $f_n\equiv ({\mathbb{I}}_{D_1},{\mathbb{I}}_{D_2},\dots ,{\mathbb{I}}_{D_M})$, with $\mathbb{I}$ being the indicator function. Then $\begin{array}{rl}& V_n-J(v_n)\\ & =\underset{v\in {\mathcal{A}}_n}{min}\mathbb{E}[J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v)]-\mathbb{E}[J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v_n)]\\ & =\underset{v\in {\mathcal{A}}_n}{min}\mathbb{E}[J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v)]\\ & -\mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v_{n+1})+e^{-\beta t_n}G({\xi }^m)){\mathbb{I}}_{D_m}]\\ & \ge \underset{v\in {\mathcal{A}}_n}{min}\mathbb{E}[J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v)]\\ & -\mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v_{n+1})+e^{-\beta t_n}G({\xi }^m)){\mathbb{I}}_{E_m}]\\ & =\underset{v\in {\mathcal{A}}_n}{min}\mathbb{E}[J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v)]\\ & -\sum _{m=1}^M\mathbb{E}[(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v_{n+1})+e^{-\beta t_n}G({\xi }^m)){\mathbb{I}}_{E_m}]\\ & \ge \underset{v\in {\mathcal{A}}_n}{min}\mathbb{E}[J^{\mathrm{\prime }}(t_n^{-},R_{t_n^{-}},I_{t_n^{-}},v)]\\ & -\sum _{m=1}^M(\mathbb{E}[(\underset{v\in {\mathcal{A}}_{n+1}}{min}(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)\\ & +e^{-\beta t_n}G({\xi }^m)){\mathbb{I}}_{E_m}]+{ϵ}_m).\end{array}$ As $E_m$ is arbitrary, it is also satisfied in the optimal case: $V_n-J(v_n)\ge -\sum _{m=1}^M{ϵ}_m.\text{■}$

Proposition 3.2

Let $n\in \{1,2,\dots ,N\}$ and impulse control $v\in \mathcal{A}$. For any depth of the NN with $D\ge 2$ and $ϵ>0$, there exist positive integers $q_1,q_2,\dots ,q_{D-1}$ such that $\begin{array}{rl}& \underset{\theta \in {\mathbb{R}}^q}{inf}\mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)+e^{-\beta t_n}G({\xi }^m))f_m^{\theta }(R_{t_n^{-}},I_{t_n})]\\ & \le \underset{f\in \mathcal{D}}{inf}\mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)\\ & +e^{-\beta t_n}G({\xi }^m))f_m(R_{t_n^{-}},I_{t_n})\phantom{\sum _{m=1}^M}]+ϵ,\end{array}$ where $\mathcal{D}$ is the set of all measurable one-hot functions.

Proof.

For a fixed $ϵ>0$, the integrability condition ensures that there exists a measurable function $\tilde{f}:{\mathbb{R}}^{d_1+d_2}\to \{e_1,\dots ,e_M\}$ such that $\begin{array}{rl}& \mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)+e^{-\beta t_n}G({\xi }^m)){\tilde{f}}_m(R_{t_n^{-}},I_{t_n})]\\ & \le \underset{f\in \mathcal{D}}{inf}\mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)\\ & +e^{-\beta t_n}G({\xi }^m))f_m(R_{t_n^{-}},I_{t_n})\phantom{\sum _{m=1}^M}]+\frac{ϵ}{3},\end{array}$ and there exists a Borel set $A_m$ such that ${\tilde{f}}_m=1_{A_m}$. As the output of $\tilde{f}$ is a one-hot function, $A_m$ is disjoint and $\bigcup _{m=1}^M{A}_m={\mathbb{R}}^{d_1+d_2}$. Based on the integrability condition and (4(4) $J_{r_0,i_0}(v)<\mathrm{\infty }.$(4) ), $B\to \mathbb{E}[(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)+e^{-\beta t_n}G({\xi }^m))1_{A_m}(R_{t_n^{-}},I_{t_n})],$ which defines a finite Borel measure on ${\mathbb{R}}^{d_1+d_2}$. Let ${\rho }_K:{\mathbb{R}}^{d_1+d_2}\to [0,\mathrm{\infty }]$ such that ${\rho }_K(x)=\underset{y\in K}{inf}\Vert x-y{\Vert }_2$. Then, $k_{a,m}(x)=max\{1-a{\rho }_{K_m}(x),-1\},a\in \mathbb{N}$ defines a sequence of continuous functions $k_{a,m}:{\mathbb{R}}^d\to [-1,1]$ that converge pointwisely to $1_{K_m}-1_{K_m^C}$ for each m. By the dominated convergence theorem, there exists $a_m\in \mathbb{N}$ such that $\begin{array}{rl}& \mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)+e^{-\beta t_n}G({\xi }^m)){\mathbb{I}}_{\{k_{a_m,m}(R_{t_n^{-}},I_{t_n})\ge 0\}}]\\ & \le \mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)\\ & +e^{-\beta t_n}G({\xi }^m)){\mathbb{I}}_{K_m}(R_{t_n^{-}},I_{t_n})\phantom{\sum _{m=1}^M}]+\frac{ϵ}{3}.\end{array}$ We require $a_m$ to satisfy at least one $k_{a_m,m}(R_{t_n^{-}},I_{t_n})\ge 0$ for $\mathrm{\forall }(R_{t_n^{-}},I_{t_n})$. This can be done because the union of $A_m$ is $R^{d_1+d_2}$. By Leshno's theorem, $k_{a,m}$ can be approximated uniformly on compact sets by functions of the form (8) $\sum _{i=1}^r(v_i^{\mathrm{\top }}x+c_i{)}^{+}-\sum _{i=1}^s({\omega }_i^{\mathrm{\top }}x+d_i{)}^{+}.$(8) Hence, there exists a function $h_m:{\mathbb{R}}^{d_1+d_2}\to \mathbb{R}$ in (8(8) $\sum _{i=1}^r(v_i^{\mathrm{\top }}x+c_i{)}^{+}-\sum _{i=1}^s({\omega }_i^{\mathrm{\top }}x+d_i{)}^{+}.$(8) ) such that $\begin{array}{rl}& \mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)+e^{-\beta t_n}G({\xi }^m)){\mathbb{I}}_{\{h_m(R_{t_n^{-}},I_{t_n})\ge 0\}}]\\ & \le \mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v)\\ & +e^{-\beta t_n}G({\xi }^m)){\mathbb{I}}_{\{k_{a_m,m}(R_{t_n^{-}},I_{t_n})\ge 0\}}\phantom{\sum _{m=1}^M}]+\frac{ϵ}{3}.\end{array}$ $h_m$ can be expressed as an NN. We can then combine multiple $h_m$ functions to form a larger prediction network.

Based on the above two propositions, we can construct an NN to approximate the optimal policy. We introduce the NN architecture in the next section.

3.3. Neural network architecture

Based on Proposition 3.1, it is theoretically possible to find a fully connected deep neural network (DNN) $f^{{\theta }_n}$ to approximate $f_n$. However, the output of $f^{{\theta }_n}$ is $\{e_1,e_2\dots ,e_M\}$, to which gradient descent cannot be applied. Therefore, we include an NN $F^{{\theta }_n}$ as a transition step for optimization purposes. $F^{{\theta }_n}$ is continuous and almost everywhere differentiable, and our objective is to minimize the following function: $\mathbb{E}[\sum _{m=1}^M(J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},{\xi }_{n+1})+e^{-\beta t_n}G({\xi }^m))F_m^{\theta }(R_{t_n^{-}},I_{t_n})].$ After determining $F^{{\theta }_n}$, the function $f^{{\theta }_n}$ is defined as $f^{{\theta }_n}=\mathrm{pmax}(F^{{\theta }_n})$ where $\mathrm{pmax}:(0,1{)}^M\to \{e_1,e_2\dots ,e_M\}$ is the function used to find the position of the maximum value of the input. The softmax and pmax functions restrict the output of the NN to $\{e_1,e_2,\dots ,e_M\}$, resulting in a smaller value than the indicator functions of Proposition 3.2. When there is more than one maximum value in $F^{{\theta }_n}$, we assume that pmax finds the position of the first maximum value by convention.

During the training process, we find that the general fully connected NN has a high probability of falling into the local minimums. Furthermore, the iterative backward training method worsens the situation because the accuracy of $F^{{\theta }_n}$ depends on an accurate estimate of $f^{{\theta }_{n+1}},\dots ,f^{{\theta }_N}$.

This phenomenon is due to the complex nature of our task. Our problem is neither a traditional regression problem nor a traditional classification problem. It differs from classical regression problems because the outputs of the NN are one-hot vectors. It also differs from classification problems because the objective function is not related to the misclassification error and there is no ground truth available for the training process. Our task is a classification task with a regression-type objective function, and this type of combination results in a high probability of being trapped in local minimums during gradient descent. To the best of our knowledge, there is no research describing this phenomenon, which we refer to as ‘class vanishing’ in the rest of the paper. When class vanishing occurs, the output of the DNN will be a proper subset of $\{e_1,e_2\dots ,e_M\}$, which means that the DNN solves a simpler problem with a smaller set of actions than the original set. We further illustrate the class vanishing problem in the numerical study section for the one-dimensional case, as a benchmark is available for this case.

To solve the class vanishing problem, we propose a new NN architecture that proves useful in the one-dimensional case. Because of the lack of benchmarks for high-dimensional cases, we leave it to future research to test the effectiveness of this NN architecture in high-dimensional situations.

$F^{{\theta }_n}$ takes the following form: (9) $\begin{array}{rl}{F}^{{\theta }_n}& =\psi (F_1^{{\theta }_n},F_2^{{\theta }_n},\dots ,F_M^{{\theta }_n})\\ F^{{\theta }_n,1}& =a_{I_1,1}^{{\theta }_n}\circ {\phi }_{q_{D_1-1,1}}\circ a_{D_1-1,1}^{{\theta }_n}\circ \cdots \circ {\phi }_{q_{1,1}}\circ a_{1,1}^{{\theta }_n}\\ & \cdots \\ F^{{\theta }_n,M}& =a_{D_M,M}^{{\theta }_n}\circ {\phi }_{q_{D_M-1,M}}\circ a_{D_M-1,M}^{{\theta }_n}\circ \cdots \circ {\phi }_{q_{1,M}}\circ a_{1,M}^{{\theta }_n},\end{array}$(9) where $D_k$ is the depth of the NN $F^{{\theta }_n,k}$, $q_{1,k},q_{2,k},\dots q_{D_k-1,k}$ are positive integers indicating the number of nodes in the hidden layer of $F^{{\theta }_n,k}$. $\psi :{\mathbb{R}}^M\to (0,1{)}^M$ is the standard softmax function defined as $\psi (x{)}_i=\frac{e^{x_i}}{\sum _{i=1}^M{e}^{x_i}}\text{for}i=1,2\dots ,M\text{and}x\in {\mathbb{R}}^M.$ $a_{1,k}^{{\theta }_n}:{\mathbb{R}}^{d_1+d_2}\to {\mathbb{R}}^{q_{1,k}},\dots ,a_{D_k-1,k}^{{\theta }_n}:{\mathbb{R}}^{q_{D_k-2,k}}\to {\mathbb{R}}^{q_{D_k-1,k}}$ and $a_{D_k,k}^{{\theta }_n}:{\mathbb{R}}^{q_{D_k-1,k}}\to \mathbb{R}$ are affine functions. ${\phi }_a:{\mathbb{R}}^a\to {\mathbb{R}}^a$ are $\mathbf{ReLU}$ activation functions where $\phi (x_1,x_2\dots ,x_a)=(x_1^{+},x_2^{+},\dots ,x_a^{+})$. The NN $F^{{\theta }_n}$ consist of M different NNs, and each NN is only responsible for predicting one impulse control choice in ${\mathcal{C}}^{\ast }$. The outputs of small NNs are combined through a softmax function, which is commonly used for multi-class classification. An overview of the NN architecture is shown in figure 1. The idea here is very similar to ensemble learning techniques such as xgboost or random forest. However, our aim is to prevent the occurrence of the class vanishing problem rather than to make the bias–variance tradeoff. We refer to this type of NN as the ensemble NN in the rest of the paper. The ensemble NN can be regarded as a kind of regularization method similar to the DropConnect method proposed in Wan et al. (2013). The difference is that DropConnect randomly sets the weights to 0, whereas we do it deterministically.

Figure 1. Neural network architecture of ensemble network. Each small NN will predict one action and the results are normalized by the softmax function.

The training of $F^{{\theta }_n}$ depends on the sample of $(R_{t_n^{-}},I_{t_n})$, and $F^{{\theta }_n}$ may not minimize ${ϵ}_m$ for $\mathrm{\forall }m$ in Proposition 3.1. We can train different $F_m^{{\theta }_n}$ functions for different distributions of $(R_{t_n^{-}},I_{t_n})$ to minimize ${ϵ}_m$. In our numerical study below, one $F^{{\theta }_n}$ function is enough to obtain a satisfactory control policy.

3.4. Parameter optimization

We determine the depth $D_k$ and the number of nodes in the hidden layer $q_{1,k},\dots q_{D_k-1,k}$ for all k and train the NN described in section 3.3. To obtain the parameter ${\theta }_n$ numerically, we simulate H paths of $(R_t,I_t)$ and minimize the sample average, which can be regarded as an approximation of the expectation of the cost function.

Consider a given $n\in \{1,\dots ,N\}$, we assume that the parameters ${\theta }_{n+1},\dots ,{\theta }_N$ are determined and generate an impulse control policy $v_{n+1}$. We denote the cost calculated by the hth path from time $t_n$ to time T by $l_n^h$, which is defined as $l_n^h={\int }_{t_n}^T{e}^{-\mathrm{\beta t}}\varphi (t,r_t^h,i_t^h)\mathrm{d}t+\sum _{i=n}^N{e}^{-\beta {\tau }_i}G({\xi }_i).$ $l_n^h$ is estimated in the discrete case by $l_n^h=\sum _{i=n}^N[e^{-\beta t_{i+1}}\varphi (t_{i+1},r_{t_{i+1}^{-}}^h,i_{t_{i+1}}^h)(t_{i+1}-t_i)+e^{-\beta t_i}G({\xi }_i)],$ where $t_{N+1}\equiv T$. In this paper, the integral is measured by ϕ at the end time point of each interval. We choose the end time point here because it is the most difficult to predict. The input for $f^{{\theta }_n}$ is $(R_{t_n^{-}},I_{t_n})$ and the resulting process values are $(R_{t_n^{-}}+{\xi }_n,I_{t_n})$, which contain no randomness. It is easier for the NN to find the optimal control policy if we include $(R_{t_n^{-}}+{\xi }_n,I_{t_n})$ in the cost calculation.

Suppose that we can apply impulse control according to $F^{{\theta }_n}$ at time $t_n$ and make a decision according to $f^{{\theta }_{n+1}},\dots ,f^{{\theta }_N}$; then, the cost for the hth simulated path is given by $\begin{array}{rl}{\mathrm{\Gamma }}_n^h({\theta }_n)& =\sum _{m=1}^M[(e^{-\beta t_{n+1}}\varphi (t_{n+1},r_{t_{n+1}^{-}}^{h,m},i_{t_{n+1}}^{h,m})\\ & +e^{-\beta t_n}G({\xi }^m)+l_{n+1}^{h,m})F_m^{{\theta }_n}(r_{t_n^{-}}^h,i_{t_n^{-}}^h)],\end{array}$ where $(r^{h,m},i^{h,m})$ is the simulated path with ${\xi }^m$ applied at time $t_n^{-}$ and $l_{n+1}^{h,m}$ is the corresponding cost from time $t_{n+1}$ to time T.

For a large H, the mean of ${\mathrm{\Gamma }}_n^h$, $\frac{1}{H}\sum _{h=1}^H{\mathrm{\Gamma }}_n^h({\theta }_n),$ approximates the expectation $\begin{array}{rl}& \mathbb{E}[\sum _{m=1}^M((J^{\mathrm{\prime }}(t_n,R_{t_n^{-}}+{\xi }^m,I_{t_n},v_{n+1})+e^{-\beta t_n}G({\xi }^m))\\ & \times F_m^{\theta }(R_{t_n^{-}},I_{t_n})\phantom{\sum _{m=1}^M}],\end{array}$ so that it is used to find ${\theta }_n$ through gradient descent.

We divide the training process into two phases, the pre-training phase and the fine-training phase, to prevent class vanishing. In the pre-training phase, the objective function is $\frac{1}{H}\sum _{h=1}^H[{\mathrm{\Gamma }}_n^h({\theta }_n)-\lambda \sum _{m=1}^M\log (F_m^{{\theta }_n}(r_{t_n^{-}}^h,i_{t_n}^h)+\epsilon )].$ The first part of this equation is the expectation that we want to minimize and the second part is the penalty term for the occurrence of a small probability in $F^{{\theta }_n}$. λ is the pre-defined parameter to balance these two objectives and ε is a small number to prevent log explosion during the training process. By adding the penalty term, we can avoid the class vanishing problem but may introduce an additional bias into our predictions. After the pre-training phase, we remove the penalty term and continue to train the model in the fine-training phase to remove this additional bias.

4. Numerical study

In this section, we examine our method in three scenarios. First, we do not consider the uncontrollable stochastic process I and test our method in the one-dimensional case. Because impulse control in the one-dimensional case is well studied, we can use the results of Cadenillas et al. (2010) as a benchmark to examine our method. Second, we illustrate our method using data on real inflation and effective federal funds rates (EFFR) in the United States. We compare our results with the Fed's decision on the EFFR in 2022. Next, we examine the effect of our model with a multi-dimensional R and a one-dimensional I through a simulation study.

4.1. One-dimensional benchmark

The one-dimensional stochastic impulse control problem is studied in Cadenillas et al. (2010), and we follow these settings for our simulation study. Assume that $R_t$ follows an OU process given by $\mathrm{d}{R}_t=a(b-R_t)d_t+\sigma \mathrm{d}{W}_t$ when there is no impulse control. a = 0.2 is the speed of reversion, b = 2 represents the long-term mean level, $\sigma =1.2$ is the instantaneous volatility of the model, and $W_t$ is the one-dimensional Brownian motion. The discount rate β is set to 0.06.

There are several differences between (Cadenillas et al. 2010) and our study, leading to a deviation between our optimal policy and theirs. In Cadenillas et al. (2010), T is set to ∞ and $\mathcal{T}={\mathcal{R}}_{\ge 0}$ are non-negative real numbers, and $\mathcal{C}=\mathbb{R}$. For our numerical study, T is set to 5, $\mathcal{T}=\{0,1,2,\dots ,4\}$, and ${\mathcal{C}}^{\ast }=\{-6,-3,0,3,6\}$. We use the same running cost function and action cost function defined as $\begin{array}{rl}\varphi (r)& =(r-2{)}^2\\ G(\xi )& =\{\begin{array}{ll}5+2\times \xi & \mathrm{if}\xi >0\\ 5& \mathrm{if}\xi =0\\ 5-2\times \xi & \mathrm{if}\xi <0\end{array}\end{array}$ in Cadenillas et al. (2010), respectively. The cost should be $\{-17,-11,5,11,17\}$ for ${\mathcal{C}}^{\ast }$ based on the two cost functions above. However, 0 in ${\mathcal{C}}^{\ast }$ represents the choice of no impulse control rather than an impulse control of magnitude 0. Therefore, we replace 5 with 0, and the cost is $\{-17,-11,0,11,17\}$ in our paper.

We train backwardly from $f^{{\theta }_4}$ with the NN architecture defined in section 3.3. The depth $D_k=3$, and $q_{1,k}=q_{2,k}=8$ for $\mathrm{\forall }k$. The $R_t$ process in (4.1) can be easily simulated, and the initial value is sampled from a uniform distribution $U(-11,15)$ to allow the NN to learn different scenarios. For each $F^{{\theta }_n}$, we simulate 8,192 sample paths for the training process, and we conduct 1,500 training steps in the pre-training phase with $\lambda =0.3$ and 3,000 training steps in the fine-training phase.

$F^{{\theta }_4}$ and $F^{{\theta }_2}$ are shown in figure 2 with different $r_{t_4^{-}}$ and $r_{t_2^{-}}$ values. $F^{{\theta }_0},\dots ,F^{{\theta }_2}$ are all similar, and we show $F^{{\theta }_2}$ here. We also show $F^{{\theta }_4}$ because $F^{{\theta }_4}$ is different from the other functions. The reason may be that this is the last decision point and the NN is more conservative in impulse control.

Figure 2. $F^{\theta }$ for different values at time $t_2^{-}$ and time $t_4^{-}$. The X axis is the value at the corresponding time points and the Y axis is the probabilities $F^{\theta }$ for different actions. The NN at $t_4^{-}$ is more conservative in applying interventions compared with the representative NN at $t_2^{-}$.

We also show a representative class vanishing problem for $F^{{\theta }_4}$ in figure 3. $F^{{\theta }_4}$ is a fully connected NN with a depth of I = 3 and 40 nodes in each hidden layer.³ The training process is the same with the same parameter settings, and we can see from the figure 3 that the three classes −3, 3 and 6 are missing. We increase the number of nodes in the hidden layers to 50, but the class vanishing problem still occurs.

Figure 3. $F^{\theta }$ for different values at time $t_4^{-}$ for 40 and 50 nodes. The X axis and Y axis are defined in the same way as figure 2. The top is NN with 40 nodes and the bottom is NN with 50 nodes. Both NNs encounter class vanishing problem.

4.2. Fed's interest rate illustration

In this section, we illustrate our method with real data from the United States. We collect EFFR and Consumer Price Index (CPI) data from U. B. of Labor Statistics (2023) and F.R.B. of New York (2023). We define the month-over-month (MoM) inflation rate $I_t$ as $I_t=\frac{\mathrm{CP}{I}_{t+1}-\mathrm{CP}{I}_t}{I_t}.$ Assume that the EFFR $(R_{1t})$ and monthly inflation $(I_t)$ follow stochastic processes, (10) $\begin{array}{rl}\mathrm{d}{R}_{1,t}& =a_1(b_1-R_{1,t})\mathrm{d}t+{\sigma }_1\mathrm{d}{W}_t^1,\\ \mathrm{d}{I}_t& =({\alpha }_0+{\alpha }_1{R}_{1,t})\mathrm{d}t+{\sigma }_2\mathrm{d}{W}_t^2,\end{array}$(10) when no impulse control is applied. $R_{1,t}$ follows an OU process with $a_1$ being the speed of reversion, and $b_1$ the long-term mean level. $I_t$ follows a stochastic process that is cointegrated with $R_{1,t}$. ${\sigma }_1$ and ${\sigma }_2$ are constant instantaneous volatilities, and $W_t^1$ and $W_t^2$ are independent one-dimensional Brownian motions. We estimate the parameters using EFFR and CPI data from August 1, 2019 to November 2.⁴ As the Fed launched several interventions on the EFFR during this period, we remove the corresponding days during the estimation process. The estimated parameters are $a_1=0.2246,b_1=-0.0555,{\alpha }_0=0.5220,{\alpha }_1=-0.8213,{\sigma }_1=0.2826$, and ${\sigma }_2=1.1619$.

After determining the parameters of the stochastic processes, we start training our NN for a one-year interest rate impulse control policy for 2022. The maturity T is set to 1. The Fed holds eight regular meetings each year to determine monetary policy, and the time interval between meetings is usually one and a half months. Therefore, we set $\mathcal{T}$ to $\{0,0.125,\dots ,0.875\}$. The Fed could only intervene on the EFFR, so the action set ${\mathcal{C}}^{\ast }$ is set to $\{-0.25,0,0.25,0.5,0.75,1\}$, which contains all of the actions applied by the Fed to the EFFR in 2022. We also add −0.25 to the action set to test whether the NN chooses the right direction of action. Inflation was extremely high in 2022, so we assume that the main goal of the Fed was to control the inflation rate. Because the Fed is aiming for 2% inflation over the long run, we set our MoM inflation rate target to 0.2%. The running cost function and the action cost function are defined as follows: $\begin{array}{rl}\varphi (r,i)& =|i-0.2\mathrm{\%}|\\ G(\xi )& =\{\begin{array}{ll}0.0025+0.06\ast \xi & \mathrm{if}\xi >0\\ 0& \mathrm{if}\xi =0\\ 0.0025-0.06\ast \xi & \mathrm{if}\xi <0\end{array},\end{array}$ where the discount factor $\beta =0.06$. The depth $D_k$ is set to 3, and the number of hidden nodes $q_{1,k}=q_{2,k}=10$ for $\mathrm{\forall }k$. We simulate 30,000 sample paths to train the NN. The training step in the pre-training phase is set to 2000 with $\lambda =0.0001$. The training step in the fine-training phase is set to 1500. After training the NN, there is one last problem to solve before comparing the output of the NN with the actual interventions of the Fed in 2022. In reality, there is a delay in the publication of the CPI, so we use the predicted inflation rate as the input. We compare our prediction results with the Fed's actions from January to September 2022, and the results are shown in table 1. Here, we consider two methods for determining the magnitude of the Fed's actions. First, we follow the general rule to choose the action with the highest probability (I) estimated by the NN. Second, we construct a method that is similar to the dot plot in the Fed's Summary of Economic Projections. We calculate $\sum _{m=1}^M{\xi }^m{F}_m^{{\theta }_n}$ for the decision at time $t_n$ and choose the action in ${\mathcal{C}}^{\ast }$ that is closest to it (II).

Table 1. The Fed's actions and our NN results.

	Fed's action	NN result I	NN result II
01/25	0	0.5	0.5
03/15	0.25	1	1
05/03	0.5	0.75	0.75
06/14	0.75	1	1
07/26	0.75	0	0
09/20	0.75	0	0
Total	3.00	3.25	3.25

Table 1 shows that our impulse control policy is more volatile than the Fed's policy. We speculate that this may be related to the Fed's intention to smooth its intervention levels and the Fed's aim on the year-on-year (YoY) inflation rate rather than the MoM inflation rate. As the YoY inflation rate is less volatile than its MoM counterpart, the induced intervention magnitude is expected to be less volatile too. Despite the intervention volatility, the overall level of interest rate intervention by the Fed is similar to our NN predictions. The predicted aggregate intervention level is useful to set stress testing target for fixed income securities. For instance, around July 26, 2022, the MoM inflation rate was predicted to be 0.2%, and our NN model suggests stopping the interest rate intervention. The Fed should have increased the EFFR by 3.25%, but it was only increased by 1.5% in reality. Therefore, the Fed still needed an increase of at least 0.75%. A similar phenomenon occurs on September 20. Therefore, our model offers an early warning signal to financial institutions for potential future interventions. This signal sets a more realistic stress test target for interest rate products.

4.3. Two-dimensional illustration

We demonstrate the application of our model to short-term and long-term interest rate interventions to monitor inflation. In other words, the central bank can apply impulse control to the EFFR and long-term interest rates. Similar to the previous section, we define the stochastic processes of the EFFR $R_{1,t}$, the long-term interest rate $R_{2,t}$, and the MoM inflation rate $I_t$ as follows: $\begin{array}{rl}\mathrm{d}{R}_{1,t}& =a_1(b_1-R_{1,t})\mathrm{d}t+{\sigma }_1\mathrm{d}{W}_t^1,\\ \mathrm{d}{R}_{2,t}& =a_2(b_2-R_{2,t})\mathrm{d}t+{\sigma }_3\mathrm{d}{W}_t^1,\\ \mathrm{d}{I}_t& =({\alpha }_0+{\alpha }_1{R}_{1,t}+{\alpha }_2{R}_{2,t})\mathrm{d}t+{\sigma }_2\mathrm{d}{W}_t^2,\end{array}$ without intervention, where $R_{1,t}$ and $I_t$ are the same as in the previous section. We assume that $R_{2,t}$ follows an OU process with a reversion speed $a_2=0.25$, a long-term mean level $b_2=0.2$, ${\sigma }_3=0.35$, and ${\alpha }_2=-0.5$. We define the action set ${\mathcal{C}}^{\ast }$ as $\begin{array}{rl}\{(0.25,-0.4),& (0.25,0),(0.25,0.4),\\ (0,-0.4),& (0,0),(0,0.4),\\ (-0.25,-0.4),& (-0.25,0),(-0.25,0.4)\}.\end{array}$ If the central bank wants to monitor the monthly inflation rate by controlling the EFFR and long-term interest rates, we use the same running cost function as in the previous section, but the action cost function is replaced by $\begin{array}{rl}G(\xi )|& =0.02\times |\xi {|}_1+0.0025\times \mathbb{I}(|\xi {|}_1\ne 0)+0.01\\ & \times |\xi {|}_2+0.005\times \mathbb{I}((|\xi {|}_2\ne 0)).\end{array}$ The discount factor $\beta =0.06$.

The depth $D_k$ of the NN is set to 3 and the number of hidden nodes $q_{1,k}=q_{2,k}=10$ for $\mathrm{\forall }k$. We simulate 30,000 paths to train the NN backwardly. After training the NN, we simulate another set of 30,000 paths to illustrate the performance of our NN. Because of the lack of a benchmark, we calculate the sample mean of the overall cost if no impulse control is applied and compare it with the overall cost if we intervene based on the NN. The cost drops from 1.299 to 0.980, indicating that our NN is useful for minimizing the cost. However, it is technically more difficult for the Fed to intervene simultaneously on short-term and long-term rates in reality. Therefore, we do not report the details here, but we mention this potential alternative use of our framework.⁵

5. Discussion and areas for future study

5.1. Relationship with deep reinforcement learning

The primary goal of Deep Reinforcement Learning (DRL) is to optimize a policy for discrete actions that maximizes the expected cumulative rewards. This objective could be made in line with that in (3(3) $\begin{array}{rl}{J}_{r_0,i_0}(v)& =\mathbb{E}[\phantom{\sum _{i=1}^z}{\int }_0^T{e}^{-\mathrm{\beta t}}\varphi (t,R_t,I_t)\mathrm{d}t\\ & +\sum _{i=1}^z{e}^{-\beta {\tau }_i}G({\xi }_i)|R_0=r_0,I_0=i_0].\end{array}$(3) ).⁶ This offers a possibility to link our problem to the discrete-time and discrete-space reinforcement learning problem. We believe that it is a promising future direction to use DRL techniques such as deep Q-learning (Mnih et al. 2013) to solve the interest rate intervention problem. An advantage of our method is that we train the DNN backward to get the optimal impulse control policy for which we can find an upper bound for the errors. The DRL requires exploration such as the ϵ greedy strategy to learn the environment. Further investigation is needed to derive error bound (or regret bound) to DRL for the same problem and compare the empirical results between the two approaches. This is a potential future direction to incorporate our ensemble networks in DRL and then empirically compare for their empirical performances.

5.2. Delay effect of interest rate intervention

Although the implementation of impulse control results in an immediate intervention of the interest rate, there exists an inherent time delay in the inflation response. In the context of inflation, it is anticipated that any adjustments would manifest as a gradual shift rather than an abrupt leap. Our proposed model (10(10) $\begin{array}{rl}\mathrm{d}{R}_{1,t}& =a_1(b_1-R_{1,t})\mathrm{d}t+{\sigma }_1\mathrm{d}{W}_t^1,\\ \mathrm{d}{I}_t& =({\alpha }_0+{\alpha }_1{R}_{1,t})\mathrm{d}t+{\sigma }_2\mathrm{d}{W}_t^2,\end{array}$(10) ) partially captures the intrinsic characteristics by incorporating the impact of the EFFR on the drift term of the inflation rate. Thus, it takes time for the inflation rate to revert to the intervened equilibrium level through the notion of cointegration dynamics. However, it is still interesting to examine and incorporate alternative delayed mechanism. An examples is the delayed stochastic system in Yan et al. (2022a) and Yan and Wong (2022) as follows. $\begin{array}{rl}\mathrm{d}{R}_{1,t}& =a_1(b_1-R_{1,t})\mathrm{d}t+{\sigma }_1\mathrm{d}{W}_t^1,\\ \mathrm{d}{I}_t& =({\alpha }_0+{\int }_{t-\mathrm{\Delta }}^t\alpha (s)R_{1,s}\mathrm{d}s)\mathrm{d}t+{\sigma }_2\mathrm{d}{W}_t^2,\end{array}$ with a specific Δ time lag. The discrete version becomes: $\begin{array}{rl}\mathrm{d}{R}_{1,t}& =a_1(b_1-R_{1,t})\mathrm{d}t+{\sigma }_1\mathrm{d}{W}_t^1,\\ \mathrm{d}{I}_t& =({\alpha }_0+\sum _{s\in [t-\mathrm{\Delta },t]}\alpha (s)R_{1,s})\mathrm{d}t+{\sigma }_2\mathrm{d}{W}_t^2.\end{array}$ The DNN training is the same except for a different input for the DNN. This delayed mechanism changes the Markovian nature of the original problem and complicates the training of the DNN. We leave it a future research.

6. Conclusion

We generalize the impulse control problem and propose a novel deep-learning framework to solve it. The generalized impulse control problem is formulated with a finite time horizon, finite decision points, and a finite set of actions which is more aligned with the realistic situations compared with previous research. In addition, we consider both controllable and uncontrollable processes to replicate the real-world scenarios encountered by the central bank. Our framework can handle high-dimensional cases with cointegrated processes, which cannot be directly controlled by impulse control. We also propose a new NN architecture to solve the class vanishing problem, which occurs because of the complex nature of our task. The ensemble NN can be easily extended to other areas when the problem is a classification task with a regression-type objective function. The accuracy of our method is examined in one-dimensional cases. Our numerical results show reasonable congruence between the predictions of our NN model and the Fed's interventions on the EFFR in 2022. We suggest that our deep impulse control framework is useful for financial institutions and regulatory agency to develop stress test interest rate scenarios for risk management purposes.

Acknowledgements

We are grateful to comments by participants of 26th International Congress on Insurance: Mathematics and Economics at Heriot-Watt University, Recent Advances on Quantitative Finance 2023 at Hong Kong Polytechnic University, and The 3rd Yushan Conference at National Yang Ming Chiao Tung University.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

H.Y. Wong acknowledges the support by HKRGC under grant no. GRF-14308422.

Notes

1 Source: the monetary policy web page of the Board of Governors of the Federal Reserve System. https://www.federalreserve.gov/monetarypolicy.htm.

2 https://www.promarket.org/2023/03/16/nobel-laureate-douglas-diamond-on-how-the-fed-could-have-prevented-svbs-collapse/?amp.

3 Our NN architecture has three layers at each decision time point so that the whole NN collecting all decision points has a total of three times the number of decision points of layers. When the whole NN is trained at once, it is essentially a DNN training (Becker et al. 2021). In addition, our framework is an extension of the deep optimal stopping (Becker et al. 2019), which also uses a three-layer NN at each time point.

4 This is a distinctive period for us to examine our DNN because the last interest rate hikes to control inflation can be dated back to 1979-1982, when the interest rates were fully controlled to retained in a specified interval by the Fed. In other words, the interest rates were not sufficiently random to empirically estimate the interaction between interest rates and the inflation rate.

5 Interested readers could e-mail [email protected] for detailed result.

6 We thank an anonymous referee for pointing out the connection. As our paper aims to stimulate the use of contemporary machine learning method to help stress testing target, this suggestion would further strengthen our purpose.