Impartial competitive learning in multi-layered neural networks

Abstract

The present paper aims to propose a new learning and interpretation method called “impartial competitive learning”, meaning that all participants in a competition should be winners. Due to its importance, the impartiality is forced to be realised even by increasing the corresponding cost in terms of the strength of weights. For the first approximation, three types of impartial competition can be developed: componential, computational, and collective competition. In the componential competition, every weight should have an equal chance on average to win the competition. In the computational competition, all computational procedures should have an equal chance to be applied sequentially in learning. In collective computing for interpretation, all network configurations, obtained by learning, have an equal chance to participate in a process of interpretation, representing one of the most idealised forms of impartiality. The method was applied to a well-known second-language-learning data set. The intuitive conclusion, stressed in the specific science, could not be extracted by the conventional natural language processing methods, because they can deal only with word frequency. The present method tried to extract a main feature beyond the word frequency by competing connection weights and computational procedures impartially, followed by collective and impartial competition for interpretation.

Keywords:

• Impartial
• competitive learning
• componential competition
• computational competition
• collective competition
• cost
• interpretation

1. Introduction

The introductory section explains the concept of impartial competition with componential and computational procedures, where all components as well as computational procedures have a chance to win the competition equally. In addition, this impartiality is applied to interpretation, where all created representations should have an equal chance to be interpreted. This introduction is accompanied by a literature review of the related studies. For example, this competition can be seen from the viewpoint of conventional competitive learning and interpretation. In the review, it is stressed that those conventional methods have not necessarily paid due attention to the equal chance to win the competition. On the contrary, this paper emphasises the equal chance to win the competition, which should be a primary objective of competitive computation.

1.1. Impartiality and competitive computation

The present paper aims to propose a new type of learning method, called “impartial competitive computation ”. Competitive computing has two fundamental properties: competition among all components and an equal chance to win the competition. Though we need to consider all components and their related procedures for competition, we deal here with three types of competition-componential, computational and collective-for interpretation for the first approximation.

First, competition can be applied to all components in a neural network. For example, any components inside a neural network should compete with each other, which can be called “componential competition ”. In particular, we try to deal with competition not among neurons but among connection weights, because we try to consider competition as independently as possible of inputs. Then, all connection weights are supposed to compete with each other by increasing and decreasing the potentiality of connection weights. The potentiality represents to what degree weights can contribute to the inner activity of a neural network. When the potentiality increases, the connection weights can move more freely. By increasing the potentiality of connection weights, all weights tend to be used  equally.

The second type of competition is applied to computational and optimising procedures. In competitive computing, we need to control the potentiality of connection weights, and the cost is needed to control the potentiality. Then, errors between outputs and targets should be decreased in supervised learning. In the conventional methods, those procedures are simultaneously applied and controlled. Thus, those procedures must be controlled by changing many parameters as well as hyper-parameters. When the number of parameters increases, we naturally have much difficulty in compromising among them. On the contrary, in the computational competition, all computational procedures are applied independently of any others. All computational procedures should be used as equally as possible. As is the case with the componential competition, only one procedure wins the competition and is applied to learning, while all the others cease to be applied. This can make the effect of each computational procedure clearer, leading to the efficient use of all computational procedures. At the present stage of research, the equal chance to win the competition can be algorithmically realised in the computational competition.

In sum, the present method can be considered an extension of conventional competitive learning. In particular, one of the most important characteristics lies in the active realisation of an equal chance for all components and computational procedures to win the competition adaptively or algorithmically. Because of this, we call competitive computation “impartial” to emphasise the equal chance to win the competition. In our competitive computing, there is a supposition that the neural network as a model of living systems should try to use all available resources as efficiently and extensively as possible for coping with new and future situations.

1.2. Impartiality and competitive learning

We need to survey competitive learning in neural networks here, because the present method should be considered an extension of the fundamental property of competitive learning. Competitive learning has received due attention from the early stages of neural network research (Fukushima, 1975; Grossberg, 1976; Rumelhart & Zipser, 1985), because neural networks have tried to imitate living systems with abundant competition. The basic concept in (Rumelhart & Zipser, 1985) is that, for an input, neurons compete with each other, and finally one neuron wins the competition, representing the input. A neuron that is closest to the input in terms of similarity or distance should be a winner, and weights to the neuron are updated. One of the most important applications are the self-organising maps by Kohonen (Kohonen, 1990), and nowadays, they have been used extensively as one of the most effective methods for visualisation and clustering (Bao et al., 2022; Bogdan & Rosenstiel, 2001; Brugger et al., 2008; Fernández Rodríguez et al., 2022; Himberg, 2000; Lagani et al., 2021; Lu et al., 2019; Makhzani & Frey, 2014; Xu et al., 2010; Yin, 2002). The simplicity of competition processes such as the winner-take-all (WTA) has had much influences in attempts to simplify computational procedures of complex neural networks. This WTA has been naturally used in the fields directly related to actual biological systems, like spiking neurons (Peng et al., 2021; Yu et al., 2018). In addition, many computational methods, by taking full advantage of the properties of those biological systems, have also focused on the WTA to improve the performance of neural networks (Krotov & Hopfield, 2019; Shinozaki, 2017; Srivastava et al., 2013).

However, as mentioned above, from the early stage of research, it has been stated that the impartiality should be realised to improve the performance of competitive learning. For example, it has been observed that some neurons can be dead or inactive, preventing all neurons from being equally used. For solving this type of problem, many methods have been developed to eliminate the dead neurons (Banerjee & Ghosh, 2004; Choy & Siu, 1998; DeSieno, 1988; Fritzke, 1993, 1996; Li et al., 2022; Van Hulle, 1999, 2004). In those methods, frequently winning neurons are penalised or the entropy of neurons is forced to be maximised, which is quite similar to the method in this paper. The competitive learning principally aims for neurons to respond to some inputs very specifically, and at the same time, for all neurons to respond equally to those inputs at least on average. Due to the difficulty in dealing with the equal chance, much attention has been paid to the specific responses of neurons. To the best of our knowledge, in the conventional competitive learning, the equal chance to win the competition has been treated secondarily. It should be repeated that the present method is certainly an extension of the conventional method. However, attention is primarily paid to the equal chance for competition, and several computational procedures such as the WTA are secondarily considered. More strongly, the equal chance to win the competition is forced to be realised by using even a higher cost in terms of the strength of connection weights. Because the equal chance is a primary objective of competitive computation, it is absolutely necessary to realise this at any cost and at any time.

1.3. Impartiality and interpretation

The impartial competition can be also applied to the problem of interpretation. Neural networks, from the beginning, have been used to create and interpret internal representations themselves relating to the extraction of hidden and unobservable factors in complex data sets (McClelland & Rumelhart, 1986; Rumelhart & McClelland, 1986). However, the modern interpretation methods in neural networks have been heavily dependent on observable entities, even in interpreting hidden representations. For example, the interpretation problems have received much attention in the active field of convolutional neural networks (CNN), dealing mainly with image data sets, where many different types of interpretation methods have been developed. However, in interpreting weights and neuron activations in hidden layers, they have tended to use information directly observable or intuitively acceptable in image data sets (Arbabzadah et al., 2016; Bach et al., 2015; Bai et al., 2022; Binder et al., 2016; da Cunha et al., 2022; Erhan et al., 2009; Lapuschkin et al., 2016; Mahendran & Vedaldi, 2015; Montavon et al., 2019; Nguyen et al., 2019; Sturm et al., 2016). Thus, possible interpretations, far from our intuition on the image data sets, have been under-estimated due to difficulty in accepting them. This under-estimation has been well recognised in recent and active discussions on adversarial features and attacks, where features contrary to our well-accepted way of thinking have been excluded completely, though those features may play important roles in the inference of neural networks (Chen et al., 2016; Ilyas et al., 2019; Wang et al., 2021, 2019; Xia et al., 2020; Xie et al., 2017).

This means that modern interpretation methods have focussed on a specific and human-oriented interpretation, because it seems to be the most reasonable intuitively. However, one of the main capabilities of neural networks lies in producing a great number of different internal representations, and in some cases, we cannot understand their meaning due to the existence of inferences, completely different from ours. Even with those incomprehensible ways, neural networks can produce the corresponding outputs. In terms of competitive computing, all those representations should have an equal chance to be interpreted. We need to develop a method to take into account all or at least as many different representations as possible to obtain a kind of more universal and stable interpretation. The present paper proposes a method to take into account as many representations as possible for interpretation. Then, an interpretation is realised by unifying all those representations.

1.4. Objectives of the paper

After explaining the importance of impartiality in computing and interpretation, we should state the objectives of the present paper in more concrete ways. This paper aims to stress the importance of the equal chance to win the competition in terms of components, computational procedures, and interpretation, and it tries to show that the equal chance leads us to find core factors hidden in direct and observable relations between inputs and targets by using the data set in a specific science.

First, it is shown that competition with the equal chance to win should be applied to components and computational procedures. In componential competition, competition is realised adaptively. In a case where the impartiality cannot be easily realised, a cost-forced method is applied, where the strength of weights (cost) is forced to increase to reduce differences among weights. In addition, the impartiality is enhanced by repeating those forced processes many times. The forced method is introduced to show how the impartiality is important in optimising neural networks.

Second, the method is applied to the interpretation of a data set on second-language learning (L2), because any decisive results have not been obtained by processing the raw natural-language data sets. The present interpretation method tries to deal with as many internal representations as possible, where all representations should have an equal chance to be interpreted. Then, we try to show that the method can extract an important feature, intuitively discussed in the specific science.

1.5. Paper organisation

In Section 2, we try to show how to define and compute the internal potentiality. Then, we show how to compete five computational procedures: cost min and max, potentiality min and max, and error minimisation, used in the present experimental results. In addition, we briefly describe how to interpret the final results. The interpretation is based on the collective competition for interpretation, in which we try to interpret the final results by considering as many representations as possible. In particular, multi-layered neural networks are compressed into the simplest ones without hidden layers for interpreting the final connection weights easily.

In Section 3 on the experimental results, we deal with a second-language-learning data set. By using the conventional methods, clear results could not be obtained by observing the raw data sets in the corresponding science. Then, we tried to extract the simple linear relations between inputs and outputs as well as non-linear ones, which were supposed to be related to the unobservable factors.

2. Theory and computational methods

In this section, we firstly explain the basic terms, such as cost and potentiality, needed to understand the present method. Using these words, we explain the concept of competitive computing with two types of competition: componential and computational competition. In componential competition, all weights compete with each other to gain the higher strength, and in addition, it is supposed that all weights should have an equal chance or equal probability to win the competition. In computational competition, all computational and optimisation procedures compete with each other, and only one computational procedure wins the competition to be applied at a certain period of time. All procedures should have an equal chance to win the competition, as is the case with the weights. Then, we explain how to implement this computing model in actual learning. Finally, we present how to interpret the final results. The interpretation method is called “collective interpretation ”, because all representations by different initial conditions, different learning steps, and different inputs have an equal chance to be interpreted. For this interpretation, we introduce a compression method, in which multi-layered neural networks are simplified into ones without hidden layers.

2.1. Componential competition

We introduce here the internal potentiality of components in a neural network. This potentiality is supposed to represent the degree of activity inside a neural network. When the internal potentiality increases, the components are activated internally without considering output information. Thus, this potentiality maximisation corresponds to the entropy maximisation in information theory (Cover & Thomas, 1991). However, we can more easily understand the meaning of connection weights by the potentiality.

In the first place, we explain a process of competition from maximum to minimum potentiality conceptually. Figure 1 shows a concept of competition from maximum to minimum potentiality. Competition is realised by moving from a state with maximum potentiality to a state with minimum potentiality in Figure 1(a). In the maximum potentiality state, all connection weights should have equal weight strength in Figure 1(b1), while in the minimum potentiality, only one weight should have some strength, and all the others become zero in Figure 1(b2).

Figure 1. Transition from a state with the maximum potentiality to one with minimum potentiality.

Let us explain more concretely a network architecture and the corresponding potentiality. We define here the potentiality for connection weights, because we try to define the potentiality inside a network, underestimating the effects of inputs and outputs as much as possible. Figure 1(b) shows an example of network architecture in which four hidden layers are used, ranging from the second to the fifth layer. We call the input layer the first layer and the final, sixth layer corresponds to the output layer. Thus, the hidden layers range between the second and the fifth layer. For a simple illustration, we focus on connection weights between the tth and t + 1th hidden layers, denoted by the notation $(t,t+1)$, where the subscript t increases from the second to the fourth layer, namely, hidden layers. For computing the potentiality, we need to compute the absolute strength of weights (1) $u_{jk}^{(t,t+1)}=\left|w_{jk}^{(t,t+1)}\right|,$(1) where the subscript t increases from 2 to 4. Then, we normalise this by its maximum value (2) $g_{jk}^{(t,t+1)}=\frac{u_{jk}^{(t,t+1)}}{{\max }_{j^{\prime }{k}^{\prime }}{u}_{j^{\prime }{k}^{\prime }}^{(t,t+1)}},$(2) where the max operation is over all connection weights between the layers. Then, the internal potentiality can be computed by (3) $G^{(t,t+1)}=\sum _{j=2}^5\sum _{k=2}^5\left[\frac{u_{jk}^{(t,t+1)}}{{\max }_{j^{\prime }{k}^{\prime }}{u}_{j^{\prime }{k}^{\prime }}^{(t,t+1)}}\right].$(3) For simplicity's sake, we suppose that at least one weight should be larger than zero, because the layers are completely separated when all connection weights are zero.

In addition, we define the complementary potentiality by (4) ${\overline{g}}_{jk}^{(t,t+1)}=1-\frac{u_{jk}^{(t,t+1)}}{{\max }_{j^{\prime }{k}^{\prime }}{u}_{j^{\prime }{k}^{\prime }}^{(t,t+1)}}.$(4) This complementary potentiality becomes smaller when the ordinary potentiality becomes larger. Thus, this complementary potentiality can be used to reduce the strength of weights with higher potentiality.

Finally, the cost can be computed simply by the sum of all absolute weights, computed by (5) $C^{(t,t+1)}=\sum _j^{}\sum _k^{}{u}_{jk}^{(t,t+1)}.$(5) This cost is especially used to realise a state in which all weights have the same strength, corresponding to the maximum potentiality state. By changing the cost in addition to the potentiality, we can produce a number of different network configurations.

Here, the problem is how to maximise the potentiality to realise the equal chance to win the competition. When potentiality is maximised by reducing the strength of connection weights, as is done in the conventional regularisation, eventually some connection weights tend to have larger weights with some specific information, due to the necessity of minimising errors between outputs and targets. To make the potentiality as large as possible and to make all weights equal in winning, we introduce a controlled cost approach in which the cost is changed to increase and decrease the potentiality.

2.2. Computational competition

The present study supposes that in principle all components and computational procedures compete with others to make the performance of neural networks better as much as possible. Thus, in addition to the componential competition, described above, we need to explain the computational or procedural competition. In this paper, five computational procedures, namely, potentiality maximisation, potentiality minimisation, cost maximisation, cost minimisation, and error minimisation, are used, as shown in Figure 2(a). In the conventional methods, all those procedures are simultaneously applied with many learning parameters and hyper-parameters needing careful adjustment, as shown in Figure 2(a). We think that those conventional methods cannot solve the contradiction among the five computational procedures. The competitive computation tries to use all computational procedures as equally as possible, as shown in Figure 2(b). At a specific learning step, only one computational procedure wins the competition, which is actually applied in learning. For example, in Figure 2(b), all computational procedures are serially disentangled, where only one procedure wins the competition at a certain learning time. Figure 3 shows actual network configurations, corresponding to the conceptual diagram in Figure 2. When potentiality is maximised, all connection weights have the same absolute strength in Figure 3(a). Then, when the potentiality is minimised, only one weight tends to be larger, while all the others become zero in Figure 3(b). When the cost is larger, the strength of all the connection weights becomes larger in Figure 3(c). When the cost is smaller, the strength of all connection weights is smaller in Figure 3(d). Finally, when the error is minimised in Figure 3(e), some weights tend to be larger in the end.

Figure 2. Conventional computing (a) and competitive computation (b), composed of five computational procedures: potentiality max, potentiality min, cost max, cost min, and error min.

Figure 3. Actual network configurations, corresponding to five computation procedures: potentiality max, potentiality min, cost max, cost min, and error minimisation in Figure 2.

2.3. Cost-forced and repeated computational computing

We explain here the actual learning models used in this paper for easy interpretation of experimental results. The reason why we adopted this model is that generalisation performance could be improved. In addition, because the method called “cost-forced” tends to increase the strength of weights excessively, we need to reduce the cost immediately after the cost augmentation.

Let us explain how to use the cost for increasing the potentiality in the first place. Figure 4 shows an example of cost-forced competitive computing used in the experiments. Learning is composed of three modules: the cost-forced potentiality maximising module in Figure 4(a), the cost minimisation module in Figure 4(b), and the cost-forced potentiality minimisation module in Figure 4(c). Figure 5 shows actual network configurations, corresponding to the conceptual diagram in Figure 4. Figure 5(a) shows an example of a network configuration in the cost-forced potentiality maximisation module. As shown in the figure, connection weights in an initial phase tend to have weights with randomly different strengths. By increasing the cost, the strength of weights becomes stronger, and all connection weights tend to be almost equal. Then, by reducing the errors between outputs and targets, several weights tend to be larger or smaller again. In the cost reduction module in Figure 5(b), we have only two procedures: cost minimisation and error minimisation. These procedures are introduced to reduce the cost as much as possible, because in the first module, the cost tends to be extremely augmented in the actual experiments. As shown in Figure 5(b), strong connection weights obtained in the first cost-forced potentiality augmentation phase are forced to be smaller. In the final module in Figure 5(c), the cost is further reduced, and then the potentiality is also forced to decrease. The potentiality decrease corresponds to specific information augmentation, where only a very few weights tend to be stronger. Finally, by error minimisation, some other weights tend to be again larger for error reduction.

Figure 4. Concept of three computational modules used in this paper: cost-forced potentiality maximisation (cost+potentiality max), cost minimisation, and cost-forced potentiality minimisation (cost+potentiality min).

Figure 5. Actual network configurations of three computational modules, corresponding to the concepts in Figure 4.

In all those processes, we try to increase the potentiality or to decrease information on inputs to obtain the minimum possible information. However, it is not so easy to increase the potentiality monotonically. The potentiality is first maximised with larger cost, and then the potentiality is decreased with smaller cost. As shown in Figures 4 and 5, those processes of potentiality maximisation and minimisation are repeated many times, with the corresponding cost reduction and augmentation. In the end, many different weight configurations can be produced for better generalisation.

2.4. Practical competitive computational procedures

Let us show how to update connection weights in learning. In the first cost-forced potentiality maximisation module in Figures 4 and 5(a), all weights are forced to be equal, even by increasing the cost in terms of abstract weight strength. Increasing the cost is realised by making the parameter θ larger than one. These large parameter values force connection weights to be large and equal in their strength, corresponding to maximum potentiality.

Then, for the practical computation, we modify the original complementary potentiality $\overline{g}$ with some additional parameters to eliminate several extreme values we must face in the middle of cost augmentation. The modified complementary potentiality is given by (6) ${\overline{h}}_{jk}^{(t,t+1)}={[1-\frac{u_{jk}^{(t,t+1)}}{{\max }_{j^{\prime }{k}^{\prime }}{u}_{j^{\prime }{k}^{\prime }}^{(t,t+1)}}+ϵ]}^{\gamma }.$(6) Those parameters are used to stabilise learning by weakening the effects of individual potentiality by setting the parameter ϵ and γ to very small values. By using this modified complementary and individual potentiality, weights are modified as (7) $w_{jk}^{(t,t+1)}(n+1)=\theta {\overline{h}}_{jk}^{(t,t+1)}(n)w_{jk}^{(t,t+1)}(n),$(7) where weights at the $n+1$th step are obtained by multiplying the previous nth weights by the corresponding modified potentiality. These update rules can be interpreted as follows. First, the parameter θ is increased, and then the potentiality is increased, followed by error minimisation, corresponding to Figures 4 and 5(a). All three procedures of cost maximisation, potentiality maximisation, and error minimisationcan be performed independently of each other in principle.

In the second module, cost minimisation, in Figures 4 and 5(b), we try to reduce the strength of weights by setting θ smaller, (8) $w_{jk}^{(t,t+1)}(n+1)=\theta w_{jk}^{(t,t+1)}(n).$(8) Then, this smaller cost value is assimilated in the phase of error minimisation.

In the third module in Figures 4 and 5(c), the weights are modified by individual potentiality with one parameter (9) $q_{jk}^{(t,t+1)}={\left[\frac{u_{jk}^{(t,t+1)}}{{\max }_{j^{\prime }{k}^{\prime }}{u}_{j^{\prime }{k}^{\prime }}^{(t,t+1)}}\right]}^{\gamma }.$(9) Then, we use the same type of potentiality assimilation process (10) $w_{jk}^{(t,t+1)}(n+1)=\theta q_{jk}^{(t,t+1)}(n)w_{jk}^{(t,t+1)}(n),$(10) where the parameter θ should be less than one to reduce the strength of weights. This process can be interpreted as follows. First, the parameter θ should be reduced from one to make connection weights smaller. Then, the potentiality is decreased gradually, followed by error minimisation.

2.5. Impartial competitive interpretation

In the impartial competitive interpretation, we suppose that all representations created by a neural network should have the same potentiality for interpretation, implying that all are winners in interpretation. This is a very extreme case and, more strongly, one of the most ideal cases of impartial competition. As shown in Figure 6, as many different internal representations as possible are produced by different initial conditions, different learning steps, and different inputs. Then, all those representations are compressed and unified into the simplest forms for each learning step in Figure 6(a). Then, all compressed weights are averaged over all learning steps, which is called “syntagmatic compression” in Figure 6(b). Then, all those syntagmatically compressed weights are again compressed or averaged in “paradigmatic” ways into the final collective weights in Figure 6(c). This interpretation aims not to deal with a specific network configuration but to deal with a space of network configurations generated by learning with different initial conditions.

Figure 6. Concept of compression for collective weights by syntagmatic (b) and paradigmatic (c) compression.

We explain here in the first place how to compress weights partially or fully, and then how to unify all those compressed weights to obtain the final collective weights for interpretation (Kamimura, 2019). The model compression methods have been developed to compress complicated multi-layered neural networks into the simpler ones (Bucilu et al., 2006; Cheng et al., 2020; Hinton et al., 2015; Luo et al., 2016). Contrary to those models, where compression is performed without keeping information on the corresponding network configuration in original networks, the present compression tries to keep original information as much as possible.

Let us show a process of compression in Figure 7 for a specific learning step, initial condition, and input. In the first place, we show how to compress networks fully, or full compression, in Figure 7(a). Now, we compress connection weights from the first to the second layer, denoted by (1,2), and from the second to the third layer (2,3) for an initial condition and a subset of a data set. Then, we have the compressed weights between the first and the third layer, denoted by (1,3), (11) $w_{ik}^{(1,3)}=\sum _j^{}{w}_{ij}^{(1,2)}{w}_{jk}^{(2,3)}.$(11) Those compressed weights are further combined with weights from the third to the fourth layer (3,4), and we have the compressed weights between the first and the fourth layer (1,4), (12) $w_{il}^{(1,4)}=\sum _k^{}{w}_{ik}^{(1,3)}{w}_{kl}^{(3,4)}.$(12) By repeating these processes, we have the compressed weights between the first and fifth layer, denoted by $w_{iq}^{(1,5)}$. Using those connection weights, we have the final and fully compressed weights (1,6), (13) $w_{ir}^{(1,6)}=\sum _q^{}{w}_{iq}^{(1,5)}{w}_{qr}^{(5,6)}.$(13) This type of compression is applied to all layers to get full compression. In addition, compressed weights are averaged over all initial conditions and subsets of data sets to get the final compressed weights called “collective weights ”.

Figure 7. Compression from a multi-layered network to the corresponding simplest network in full (a) and partial (b) way.

The partial compression is performed with only one hidden layer in Figure 7(b). For example, the weights from the second to the third layer (2,3) are combined with the weights from the first to the second layer (1,2). Those weights are directly combined with the output weights (5,6), and we have (14) $w_{ir}^{(1,(2,3),6)}=\sum _q^{}{w}_{iq}^{(1,3)}{w}_{qr}^{(5,6)},$(14) where the notation (2,3) means that only connection weights from the second and third layer are combined with the input and output layer. In all cases, it is supposed that the number of neurons in hidden layers is the same, but this method can be applied to networks with a different number of neurons in hidden layers.

Then, we should explain how to obtain actually the final collective weights. In principle, the final collective weights can be obtained by averaging all representations created with different initial conditions, subsets of data sets, and different learning steps, because we suppose that all instances of connection weights should have the same importance and that they should be taken into account for the interpretation. As shown in Figure 6, we have two types of compression for considering all learning steps, initial conditions, and input patterns: syntagmatic and paradigmatic compression. Figure 8 shows two types of compression realised by actual networks. Connection weights are trained with an initial condition and with a fixed number of learning steps and an input pattern. For each learning step, we compress obtained weights into the simplest ones in Figure 8( b1–b3). All those compressed weights are further compressed or averaged in Figure 8(b). This process of compression is called “syntagmatic ”. In addition, all those syntagmatically compressed weights are compressed or averaged in Figure 8(c), which is paradigmatic compression. Finally, the compressed weights, called “collective weights ”, can be obtained in Figure 8(d). We should state again that all obtained compressed weights and collective weights should have the same importance, and in principle, they are equally taken into account for interpretation. This is one of the most extreme cases of impartial competition.

Figure 8. Simple (a), syntagmatic (b), and paradigmatic (c) compression to produce the final collective weights (d).

3. Results and discussion

In this experiment, we tried to distinguish between Japanese learners of English with higher and lower TOEIC scores and tried to examine whether the relative clause could be extracted as an important factor in the distinction, as stated in the specific science.

3.1. Distinction between skilled and unskilled learners

Difficulty with English relative clauses in second-language learning (L2) has received much attention in the studies on many languages (Eckman et al., 1988; Izumi, 2003; O'Grady et al., 2003; Papadopoulou & Clahsen, 2003). Thus, the use of relative clauses in writing should be one of the main factors to differentiate between skilled and unskilled learners. Many reports have been published on the difficulty, mainly based on intuitive speculation. To the best of our knowledge, no experimental results have been reported on the importance of the relative clause to English-language learners by analysing the actual English texts with natural language processing systems. For example, in the conventional method, using the well-known ICANALE corpus, several experimental results were reported (Wakamatsu, 2018), but no explicit results could be obtained, simply a statement that the important variables were the type and token naturally, followed by naming several ordinary pronouns and functional words such “I ”, “the”, and so on, which are actually high-frequency words. These results show that the conventional statistical methods could not produce reasonable final results due to a focus on the observable words themselves. We think that those results with conventional statistical methods were confined to the high-frequency observable entities. The low-frequency words, which may be related to the relative clauses and which are unobservable, cannot be easily identified. The present paper aims to extract an input, related to the relative clause, by impartial competitive computing and interpretation.

3.2. Experimental outline

The data set was taken from the data set of ICANLE: the international corpus network of Asian learners of English as shown in the notes on data availability. The input variables were ordered in terms of word frequency, ranging from the most frequent word “the” to the least frequent word “from ”, selected from among the top 50 frequent words. In addition to these actual words, we added the type and token measure. Because these types and tokens are naturally related to word frequency, these measures should have considerable effects on learning, because word frequency is one of the most important observable factors. Ten hidden-layered neural networks with ten neurons in each hidden layer were used, where the number of inputs was 344, and the number of input variables was 52. This was the very redundant network configuration for this problem, to show that the redundant network is necessary for good performance, in particular, for generalisation. We tried to improve generalisation performance, because it has been considered one of the main objectives of neural networks. Figure 9 shows how to compute final collective weights for interpretation, by the parameter θ, ten different initial conditions, and subsets of input patterns. In the experiments, we changed the parameter θ to control the cost or the strength of weights, and we tried to see how generalisation or correlation coefficients were changed.

Figure 9. Cost-forced potentiality max (a), cost min (b), cost-forced potentiality min (c), and final collective weights with an important input representing “who” (d) for the L2 data set.

Let us show how the experiments were performed, and in particular how to compute collective weights for interpretation. In the first place, we determined the value of the parameter θ, and for an initial condition, learning was performed to produce a set of compressed weights in Figure 9(a) by the cost-forced potentiality maximisation module, followed by the cost minimisation module in Figure 9(b) and cost-forced minimisation module in Figure 9(c). Then, the syntagmatic compression was applied to unify compressed weights for all learning steps, ranging from the first learning step to the final learning step. Those syntagmatically compressed weights for all learning steps were again compressed or averaged to produce the final collective weights in Figure 9(d).

In the first place, we used the complementary potentiality to change the connection weights, and for the weights $(t,t+1)$ from the second to the third layer, we have (15) $\theta {\overline{h}}_{jk}^{(t,t+1)}=\theta {[1-\frac{u_{jk}^{(t,t+1)}}{{\max }_{j^{\prime }{k}^{\prime }}{u}_{j^{\prime }{k}^{\prime }}^{(t,t+1)}}+ϵ]}^{\gamma },$(15) where the parameter θ increased from 1 to 1.5. When the parameter increased from 1, connection weights were forced to be larger in their strength. This cost-forced method was introduced to adjust connection weights, whose strength tended to be greater. The other parameter γ was set to 0.05; because connection weights were changed many times, they tended to have extreme values due to the complementary potentiality. Then, we tried to eliminate too-large effects on the weight change by reducing the parameter. The final parameter ϵ was set to 0.01 to eliminate zero values. Secondly, we tried to reduce the strength of weights by setting a smaller θ (16) $w_{jk}^{(t,t+1)}(n+1)=\theta w_{jk}^{(t,t+1)}(n).$(16) The actual small value was 0.95, and because the number of epochs in the error minimisation phase was set to 50, the effect of this small value could be observed in the process of error minimisation. Thirdly, the weights were modified by individual potentiality with two parameters (17) $\theta q_{jk}^{(t,t+1)}=\theta {\left[\frac{u_{jk}^{(t,t+1)}}{{\max }_{j^{\prime }{k}^{\prime }}{u}_{j^{\prime }{k}^{\prime }}^{(t,t+1)}}\right]}^{\gamma }.$(17) The parameter θ was set to 0.95, and the other parameter γ was set to 0.05. This potentiality had an effect to reduce the strength of connection weights when they were smaller. When the connection weights became smaller, the weights became much smaller in their strength. In the end, only a small number of weights with larger strength remained, producing lower potentiality.

Finally, we should note the other parameter values in neural learning in the scikit-learn package. For those values, to reproduce easily the results in this paper, we tried to keep the original and default parameter values as much as possible, except for the activation function of the tangent hyperbolic function and the number of learning epochs, and without conventional regularisation methods.

3.3. Internal potentiality and cost

The best generalisation performance was obtained when we tried to repeat potentiality min-max and cost min-max many times. This means that good performance could be obtained not by the simple and monotonic increase or decrease of potentiality and cost, but by the repeated application of a decrease or increase of potentiality and cost. In other words, the repeated and active application of competitive computing is necessary for learning. The neural networks should try to seek the optimal network configurations by actively repeating max-min operations in the course of learning.

Figure 10 shows the potentiality (left) and cost (right) as a function of the number of learning steps by four methods. As shown in Figure 10(a), by using the conventional method, the potentiality (left) and its cost (right) remained unchanged throughout all learning steps. When maximum generalisation was attained in Figure 10(b), potentiality maximisation and minimisation as well as cost maximisation and minimisation were repeated 20 times. However, in the end, the learning process showed a process of cost minimisation as well as potentiality maximisation. This method tried to maximise the potentiality and to minimise the corresponding cost by repeating the processes of maximisation and minimisation, seeking for optimal network configurations. We think that the monotonic decrease or increase in the conventional regularisation methods could not produce better results due to the complexity of over-parametrised and redundant networks used in the experiments. When maximum correlation was obtained in Figure 10(c), the potentiality immediately reached almost a maximum point. In addition, the cost was forced to be much larger in the end. This means that learning was not sufficiently performed due to the higher cost. When generalisation was the lowest in our experiment, shown in Figure 10(d), the potentiality did not increase sufficiently, because the cost (right) remained extremely small throughout the entire learning process. This means that the cost minimisation was so effective that the networks could not increase the corresponding potentiality, meaning the occurrence of under-fitting.

Figure 10. Internal potentiality (left) and cost (right) as a function of the number of steps when the conventional method was used (a) and when the parameter θ was 1.3 (b), 1.0 (c), and 1.5 (d) for the L2 data set.

These results show that, in increasing generalisation, the monotonic decrease of cost and monotonic increase or decrease of potentiality were not enough, but the networks should try to repeat increased or decreased operations many times by re-activating the competition processes to reach optimal network configurations. In conventional terms, we need to repeat a process of regularisation and the so-called “de-regularization” many times. One of the major problems of this method is that the cost could be increased extremely in the end, as shown in Figure 10(c), due to the large parameter θ. This means that we should develop a method to restrict the increase in the strength of connection weights for future studies.

3.4. Collective weights

The conventional methods and the majority of our methods tried mainly to detect linear correlations between inputs and targets, where only functional words and summary words such as the type and the token words naturally showed larger importance values, corresponding to the results so far reported by the conventional methods. On the other hand, the neural network with the best generalisation could produce collective weights different from the correlation coefficients, in which many inputs, considered not so important linearly, tended to have larger strength or importance.

Figure 11 shows the correlation coefficients between inputs and targets of the original data set (a) and collective weights by the conventional method (b) and the impartial method (c) –(e) with different values of correlation coefficients and generalisation. Figure 11(a) shows that the correlation coefficients between inputs and targets were larger for “for” (13th input) and “their” (48th input), meaning that the functional words tended to have larger importance, corresponding to the results so far reported. In addition, we added for easy interpretation the type (1st input) and token (2nd input) to the data set. The type and token on the leftmost side of the figure showed larger correlation coefficients. This shows that the correlation coefficients seem to be based on the frequency of words, in particular, functional and summary words. These types of experimental results have been so far observed in the results by analysing the texts by many different types of statistical methods (Ishikawa, 2013; Wakamatsu, 2018).

Figure 11. Correlation coefficients between inputs and targets of the original data set (a) and collective weights by the conventional (b) and the impartial methods (c)–(e) with different performance values for the L2 data set.

By the conventional method in Figure 11(b) and by our method with maximum correlation in Figure 11(d), with minimum generalisation performance in Figure 11(e), the collective weights were quite similar to the correlation coefficients in Figure 11(a), though several different inputs tended to have slightly larger values. Thus, the conventional methods as well as our method tried to extract mainly linear relations between inputs and outputs. However, when maximum generalisation performance was obtained in Figure 11(c), collective weights were clearly different from the correlation coefficients in Figure 11(a), and many inputs tended to have larger values. In particular, input No.1 (type) had the highest score, meaning that the networks tried to classify inputs by the number of types. On the other hand, by the conventional method and our method with maximum correlations, input No.2 (token) was larger. This means that, to improve generalisation, we need to pay more attention not to the total number of words, but to the number of distinct words. Though our method seemed to try to detect different inputs as important, we could not see clear characteristics on those collective weights due to too many larger collective weights.

3.5. Relative collective weights

Then, we tried to see the effect of non-linearity by computing the ratio of absolute collective weights to the absolute original correlation coefficients. The results showed clearly that the present method could show characteristics that could not be clarified by using the correlation coefficients. In particular, a specific relative pronoun “who” (42nd input) to represent the relative clause tended to have larger relative collective weights. This means that this word “who” could be shown by the effect of non-linearity or some combination of inputs.

Figure 12 shows the relative values obtained by dividing the absolute collective weights by the corresponding absolute correlation coefficients. Figure 12(a) shows the relative weights by the conventional method, where the relative weights were small, meaning that the conventional method produced collective weights close to the correlation coefficients, paying little attention to the non-linear relations. Figure 12(b) shows the relative weights when maximum generalisation was obtained. The largest relative weight was observed for input No.11 (“smoking”), which was the topic of the piece of writing. This is natural, because the topic words in writing tended to be used frequently. However, the collective weights as well as correlation coefficients could not capture the word in Figure 11. The second largest one was input No.36, representing the preposition “with ”, which could not easily be explained due to the functional word. In addition, the third largest one was input No.42 (“who”), representing the relative pronoun. This input was not considered important in terms of the linear correlation coefficients in Figure 12(a). In addition, when we see the other results on the relative weights by the present method, input No.42 (“who”) should clearly play an important role. For example, when the maximum correlation was obtained in Figure 12(c), input No.42 (“who”) was by far the largest. Then, when generalisation was the lowest in Figure 12(d), input No.42 had a lower value, and the other larger inputs in Figure 12(b) tended to have smaller values. These results show that input No.42, representing the typical relative pronoun, can play a critical role in discrimination in this data set.

Figure 12. Relative collective weights, computed by dividing the absolute collective weights by the corresponding absolute correlation coefficients by the conventional (a) and three impartial methods (b)–(d) for the L2 data set.

3.6. Partial collective weights

By examining collective weights partially, we could see that the networks with the largest generalisation performance could process information from the first hidden layer, and the information decreased in the final hidden layer. On the other hand, the other methods, including the conventional methods, did not process information in the hidden layers near the input layer; in particular, by the impartial method, only the final hidden layer played an important role in learning. The use of almost all hidden layers by our method is related to the corresponding higher generalisation.

Figure 13 shows partial collective weights obtained by four methods. Figure 13(a) shows the partial collective weights obtained by the conventional method. Weights in the hidden layers had smaller values except for the second to the third hidden layer and in layers higher than the seventh hidden layer. On the contrary, when the best generalisation was obtained in Figure 13(b), all hidden layers except the final hidden layer had relatively larger values for all inputs except the final hidden layer. The final hidden layer, closest to the output layer, had the lowest values. When the largest correlation was observed in Figure 13(c), the partial collective weights were stronger only for the final hidden layer. Finally, when the worst generalisation was obtained in Figure 13(d), the partial collective weights were only stronger for the higher two hidden layers. This means that an almost equal use of all hidden layers was needed to improve generalisation.

Figure 13. Partial collective weights from the first hidden layer to the tenth hidden layer by the conventional (a) and potentiality method (b)–(d) for the L2 data set.

3.7. Correlation and generalisation

The present method could produce the best generalisation at the expense of decreasing the correlation coefficients between inputs and targets. However, the results suggest that, with relatively larger correlation coefficients, we could still obtain better generalisation performance than that obtained by the other conventional methods.

Table 1 shows the summary of experimental results on the generalisation and correlation coefficients between collective weights and the original correlation coefficients between inputs and targets. By the impartial competitive computing method, only three cases were inserted with the best generalisation, the largest correlation coefficient, and the lowest generalisation when the parameter θ increased from 1 to 1.5. The best generalisation (0.917) was obtained by the impartial method with $\theta =1.3$. However, the correlation coefficient was the lowest (0.405). This means naturally that to improve generalisation performance the neural network should capture non-linear relations or moderating relations among several inputs between inputs and outputs. However, when the parameter θ was 1.5, the correlation coefficient was the second largest (0.877), and the generalisation performance was still the second highest (0.907). This suggests that by keeping relatively higher linear relations the generalisation performance could be improved. The largest correlation (0.892) was obtained by the logistic regression analysis, having still good generalisation performance (0.896). The conventional method produced the worst generalisation (0.850), having a lower correlation (0.877). The random forest produced the lowest correlation coefficient (0.657), but it could show relatively higher generalisation (0.901). The results show that the linear relations could capture the main parts of the characteristics of input patterns. However, to further improve generalisation, we need to force networks to consider non-linear relations.

Table 1. Summary of experimental results on average correlation coefficients and generalisation performance for the L2 data set.

Method	θ	Correlation	Accuracy
Impartial	1.0	0.828	0.875
	1.3	0.405	0.917
	1.5	0.877	0.907
Conventional		0.877	0.850
Logistic		0.892	0.896
Random		0.657	0.901

3.8. Discussion

The experimental results discussed so far seem to show the effectivity of impartial competitive learning, focusing on the hypothesis that all components should have an equal chance in competition, and more strongly, all should be winners in competition. We discuss here the implication of this importance of impartiality in other research areas of neural networks. In addition, two limitations should be discussed, concerning the properties of potentiality. To make discussion more concrete, we use the recently discussed lottery ticket hypothesis with regularisation problems. Moreover, we try to add some comments on relations between generalisation and network size in general. This is because those regularisation problems have tried to find specific weights, neurons, and network configurations among many, contrary to our principle of impartiality. The present method tries to show the importance of using all resources inside neural networks as equally as possible, which is contrary to the specific use of resources in neural networks.

Competitive processes have been well recognised not only in competitive learning but also in many conventional learning methods. For example, many regularisation methods such as the weight decay and even the dropout procedures (Srivastava et al., 2014; Wu et al., 2021; Xiao et al., 2016) can be considered ones to realise a process of competition among neurons and connection weights. In the case of weight decay, in a process of reducing the strength of weights, competition among weights becomes higher, and finally only a few weights out of many win the competition. This type of regularisation is a method for connection weights to compete with each other under the condition of error minimisation. In our view, one of the main problems in terms of competitive computation is that all connection weights do not have an equal chance to win the competition at any state of learning. This means that there are almost no ways to restore the learning when it has gone into traps, though some random noises, inherent to some specific learning methods, may be of some use (Gunasekar et al., 2017; Mandt et al., 2017). The regularisation may be applied with difficulty in learning due to this inequality in winning the competition.

In connection with the regularisation, a new concept has been stated, closely related to the present hypothesis of competitive computing in this paper, namely, the lottery ticket hypothesis (Chen et al., 2021; Frankle & Carbin, 2018; Frankle et al., 2019, 2020; Malach et al., 2020; Tian et al., 2019). The hypothesis says simply that large randomly initialised neural networks tend to have a sub-network with lucky initial weights. This hypothesis may explain why over-parameterised neural networks show better generalisation performance. However, one of the main problems with this hypothesis lies in the implicit supposition of equal potentiality of any sub-networks, which is not necessarily true. Some networks are chosen much frequently, and others have no chance to be chosen, as is the case with the dead neurons in competitive learning, due to initial biases. The present paper aims to eliminate the frequent chance for specific sub-networks to win the competition by making all weights as well as sub-networks equally chosen. In addition, the present paper aims to show that, without a method to win the competition equally, we have much difficulty in extracting an appropriate sub-network. It can be said that the conventional methods can be dependent on the implicit and passive supposition of equal chance.

At this point, we think that the impartiality can explain one of the phenomena inherent to neural networks, where even as a network size becomes larger in terms of components, generalisation performance does not necessarily decrease, contrary to our expectation (Arpit et al., 2017; Zhang et al., 2021). Those phenomena can be explained by the well-known concept of implicit regularisation (Gunasekar et al., 2017). However, this may be explained by using the impartiality. When the network size becomes larger, and the number of connection weights becomes larger, and when the strength of initial weights is considerably small and close to zero, connection weights tend naturally to have smaller and almost equal values in the end. In terms of our potentiality, all individual potentialities of weights become smaller and almost equal. This means that equal potentiality may be realised in this case, but this is not always true in much larger networks, due to the appearance of dead components, as mentioned in the section on the problem of competitive learning. Thus, the implicit regularisation hypothesis in this case seems to be a passive attitude for obtaining higher potentiality, discussed in this paper. Contrary to those passive methods, we try to force networks to have equal potentiality or an equal chance to win the competition at any time when we must face inequality in competition, including in much larger networks. If it is possible to do so, it can be said that, as the network size becomes larger, generalisation becomes correspondingly higher due to the possible and active existence of higher potentiality.

Finally, two limitations should be pointed out: lower potentiality and how to define the potentiality. In the first place, we have the problem of lower potentiality. As mentioned above, competition is a process of transition from a state with higher potentiality to lower potentiality. However, in our approach, focusing on the higher potentiality, a state with lower potentiality is not necessarily considered fully. In our experimental results, the computational procedures tried to increase the potentiality, accompanied by slightly lower potentiality. As might be inferred, a state with lower potentiality is one with fewer connection weights, leading to more interpretable configurations of weights, compared with a state with higher potentiality. Then, higher and lower potentiality are mutually exclusive, and contradictory to each other, and it is desirable to develop how to produce a state in which higher and lower potentiality co-exist more naturally.

Second, one of the main limitations or maybe one of the main shortcomings of the method proposed here is that the potentiality of a network is principally based on the strength of absolute weights. Then, as the strength becomes larger, the corresponding potentiality becomes simply larger. However, as might be easily inferred, the strength of absolute weights does not necessarily represent the importance of the corresponding connection weights. This is the basic shortcoming or limitation of the method. We used the strength of absolute weights as the first approximation of the true potentiality of components of neural networks because it is easy to define and compute it. The problem is that, when the potentiality increases, all connection weights tend to be the same, having no information for inputs. This is because our potentiality measure has been introduced to replace the entropy measure, and the entropy and the potentiality are almost the same in their meaning. When the entropy increases, the potentiality increases correspondingly. However, when entropy increases, all components become uniformly activated, having no information on inputs. Thus, if the potentiality should represent more realistic situations of information flow in neural networks, it might be better to introduce stronger concepts of potentiality instead of the strength of absolute weights. As is well known, in the informax principle by Linsker (1988, 1989, 1992), a pioneer of the information-theoretic method on neural networks, the variance played important roles, and this measure may be of some use for the potentiality.

4. Conclusion

The present paper proposed a new learning method for all components and computational procedures to compete with each other. In addition, we tried to emphasise the importance of the equal chance or potentiality to win the competition, where all should be winners. However, the equal chance to win the competition cannot be easily realised, because the equal chance corresponds to losing specific information for learning. For this problem, we introduced the cost-forced method to increase the potentiality of components to win the competition. The potentiality as well as the corresponding cost were applied, repeating the processes of potentiality and cost maximisation and minimisation. This repeated way of cost augmentation aimed to force the potentiality or chance to win the competition to increase to the extreme point.

The method was applied to a second-language learning data set. Though much difficulty could be observed in using the relative clauses in language acquisition, it was impossible to extract this fact by analysing the data set with natural-language processing systems, because they can deal only with word frequency. This paper supposed that this inability to extract the relations was due to the shortcoming of conventional linear statistical methods in which only direct relations were explored. Our new method on the impartial potentiality tried to seek for both direct and indirect relations by minimising biased information to obtain necessary information. The experimental results showed that the new method was successful in clarifying linear relations as well as non-linear or indirect ones by which we could extract the important factor of relative clauses, which have been considered ones that make second-language learning difficult.

The paper has aimed to seek for the possibility of competitive learning with impartial operations purely for clarifying the meaning of competition without paying any attention to large-scale data sets and corresponding industrial applications. However, two things are clear, which have some significance favourable for industrial application. The method has been developed to simplify the complicated computational procedures, and this simplicity may be applicable to large-scale industrial applications. In addition, the interpretation has aimed to clarify the meaning of collective behaviours of neural networks, meaning that this interpretation may give more stable and fixed interpretation for large-scale and complicated data sets.

For practical application, more studies are needed to make information acquisition and potentiality control more smooth in competitive computing. For example, competition was controlled algorithmically, which should be more softly controlled in extracting important information from a small quantity of information. However, we can say that the present results certainly show a possibility that many complex problems in social and human sciences can be dealt with by neural networks, whose use will make it possible to clarify hidden characteristics in complicated phenomena.

Author's contributions

The author was responsible for all the conception or design of the work; or the acquisition, analysis, or interpretation of data; or the creation of new software used in the work.

Disclosure statement

The author has no competing interests to declare that are relevant to the content of this article.

Data availability statement

The data set was taken from the data sets of the research project of ICNALE: the international corpus network of Asian learners of English; a collection of controlled essays and speeches produced by learners of English in 10 countries and areas in Asia (https://language.sakura.ne.jp/icnale/). The files on Japanese English learners: W_JPN_PTJB1_2.txt  and W_JPN_SMK_B1.txt were taken from the project home page.