Application Of Density-Based Clustering Approaches For Stock Market Analysis

ABSTRACT

Present economy is largely dependent on the precise forecasting of the business avenues using the stock market data. As the stock market data falls under the category of big data, the task of handling becomes complex due to the presence of a large number of investment choices. In this paper, investigations have been carried out on the stock market data analysis using various density-based clustering approaches. For experimentation purpose, the stock market data from Quandl stock market was used. It was observed that the effectiveness of Dynamic Quantum clustering approach were better. This is because it has better adopting capability according of changing patterns of the stock market data. Similarly performances of other density-based clustering approaches like Weighted Adaptive Mean Shift Clustering, DBSCAN and Expectation Maximization and also partitive clustering methods such as k-means, k-medoids and fuzzy c means were also experimented on the same stock market data. The performance of all the approaches was tested in terms of standard measures. It was found that in majority of the cases, Dynamic Quantum clustering outperforms the other density-based clustering approaches. The algorithms were also subjected to paired t-tests which also confirmed the statistical significance of the results obtained.

Introduction

Stock market guides the future business avenues, which, in turn, influences the economy of a country. So, stock market data analysis is an important research issue for stock market prediction. Accurate prediction will guide the economy in healthy direction. Creating portfolios for the various companies of the stock market is a key aspect for investors as investing in companies is quite expensive. Also this is a very complex job. Since stock market data falls under the category of big data, conventional data mining tools does not work efficiently for the extraction of information from it which can help better stock market prediction. Many machine learning-based techniques have been developed for the purpose. The aim to these techniques are to find effective measures to select stocks having distinguished features (Das and Saha 2019; Rasekhschaffe and Jones 2019). Various clustering-based algorithms have been developed by the past researchers for stock market data analysis (Bini and Mathew 2016; Cai, Le-Khac, and Kechadi 2016; Chaudhuri and Ghosh 2016; Lee et al. 2010). But most of them exhibit a lot of limitations and poor performances owing to the high-dimensional and random trait of the time series stock market data. Also, existence of noise in the data often results in unsatisfactory performance using the traditional methodologies (Nair et al. 2017).

Organizational functions nowadays largely depend on stock market data which is of big data (Sivarajah et al. 2017) category. The information embedded in this type of big data is extensively used by the organizations for taking important decisions regarding its future growth (Data and Hub 2016). Many organizations have already tuned themselves to this category of big data in order to gain greater competitive advantages (Zillner et al. 2016). Particularly, financial organizations have turned toward stock market-based big data analytics to make better choice for investments (Prabhu et al. 2019). As these analysis deal with vast historical data, significant challenges exist for extracting the desired information from it (Alexander et al. 2017). The huge volume of this type of data demands for more refined statistical methods for achieving precise results.

Several algorithms were tried by the past researchers to overcome these limitations by analyzing the effect of changes in the parameters on the clustering mechanism (Dempster, Laird, and Rubin 1977; Ester et al. 1996; Ren et al. 2014; Weinstein and Horn 2009). The clustering approaches based on density of the data-points helps to create clusters which are homogeneous in nature with respect to their returns and volatilities (Dinandra, Hertono, and Handari 2019). Although many researchers applied density-based clustering approaches for analyzing different aspects of the stock market data (Gupta et al. 2023; Han, He, and Toh 2023; Parvatha et al. 2023; Rukmi et al. 2019), however, the techniques are limited to DBSCAN algorithm only and the other modern density-based clustering approaches are not yet being tried for stock market data analysis.

Therefore, in this investigation, experiments have been carried out using various density-based clustering approaches (such as Dynamic Quantum Clustering, Weighted Adaptive Mean Shift Clustering along with DBSCAN, and Expectation Maximization) for grouping companies on the basis of their market trends over two years time period. In order to achieve this, the historic returns and volatility are measured for each of the stocks and the same is being used for clustering purpose. From the clustering results, using the concept of moving average, the returns of the stocks are predicted. Most of the classification methods for stock market prediction require large number of training data. Whereas, methods presented in this article is density-based approach which does not require any training samples.

In this paper, the role of density-based clustering approaches were investigated for the purpose of stock market data analysis. The role of Dynamic Quantum Clustering (DQC) in the stock market data analysis has been thoroughly investigated and its performance was compared with other density-based clustering approaches namely, Weighted Adaptive Mean Shift Clustering (WAMSC), Density- based spatial clustering of applications with noise (DBSCAN) and Expectation Maximization (EM), also with three other partitive clustering-based methods, namely, k-means (Nanda, Mahanty, and Tiwari 2010), k-medoids Nakagawa, Imamura, and Yoshida (2019) and fuzzy c means (Esfahanipour and Aghamiri 2010) previously applied on stock market data. It was found that in the event of stock market analysis to evaluate the stock market prediction capability, Dynamic Quantum clustering outperforms all the other density-based clustering approaches.

The advantage of choosing the density-based clustering approaches is that the number of clusters need not be specified but it is calculated automatically by the algorithms based on density of the data. Additionally, since the presented methods in this paper are density based, therefore, it can detect arbitrary shaped (both convex and concave shaped clusters). Also outliers can also be detected quite effectively by these approaches. Particularly, Dynamic Quantum clustering approach is a very novel approach in this field. Moreover, the future trend of the stocks under investigation can be predicted by utilizing the concept of moving averages of the respective stocks on top of the clustering results based on the historic returns and volatility factors of the stocks obtained by the different density-based clustering approaches. The objective of the investigation is to devise mechanisms for analyzing the stock market so as to form effective trading decisions.

The rest of the article is arranged in the following manner. Section 2 gives a glimpse of some of the previous clustering-based investigations performed on the stock market data. Section 3 discusses the various algorithms considered for our work along with the information regarding the stock market dataset used. Section 4 describes the experimental evaluation of the algorithms using various performance measures. Section 5 summarizes the experimental results obtained. Finally, Section 6 provides a brief conclusion of our work with a highlight on the subsequent direction of research.

Related Works

This section highlights some of the works done by previous researchers on the forecasting of the stock market along with their advantages and limitations.

Nanda, Mahanty, and Tiwari (2010) used the k-means for constructing diversified portfolios of stocks using the Bombay Stock Exchange for the fiscal year 2007–2008 from the Capitaline Databases Plus. The results of their study were compared with SOM and fuzzy c means and intraclass inertia was used for their performance analysis. One of the major drawback of the k-means is that it is unable to capture if the structure of the market becomes concave.

In a similar work (Esfahanipour and Aghamiri 2010), Neuro-Fuzzy Inference System adopted on a Takagi–Sugeno–Kang (TSK)-type Fuzzy Rule-Based System was developed for stock price prediction. In this work, fuzzy C means clustering was implemented for specifying number of rules. The dataset used here is the Tehran Stock Exchange Indexes (TEPIX) and performance is measured using mean absolute percentage error (MAPE). The model forecasted results in TSE index and some Tehran Stock Exchange Indexes.

Baser and Saini tried to generate clusters by using some of the well known methods namely, k-means, k-medoids, and fast k-means (Baser and Saini 2015) to help investors for framing optimum portfolio and better risk-return profile. The dataset considered for the purpose was the Nifty 50 corporations from the market. The effectiveness of the three approaches was measured using intra-class inertia which pointed toward the better performance of the k-means algorithm, which produces clusters of compact size in contrast to the clusters produced by the other approaches. The clusters thus produced by the k-means algorithm were then experimented for asset building and portfolio study. The study can be further improved by comparison with other clustering-based methods, so that diverse portfolios with optimum risk-returns can be suggested for capitalists.

In a very preliminary work (Nair et al. 2017), the system enables the investors to determine viable options for profit maximization and also in comprehending the appropriate knowledge from the stock market. In this approach, dimensionality reduction is done using regression trees after which the data is subjected to Self Organizing Maps (SOM) for the purpose of clustering. The model is tested according to the performance measure of training error, testing error, number of rules and number of parameters, which are found to be .035, .072, 6 and 67, respectively. The performance of the model is not very promising and also very less number of data has been used for the experiment.

In another work (Cheong et al. 2017), a portfolio optimization strategy is demonstrated in which the clustering approach is assisted by a genetic algorithm (GA) for efficient handling of portfolios from the KOSPI 200. Initially, a group of portfolio is generated using the k-means clustering technique. After that a GA is applied to optimize the chosen stocks based on their respective weights. Results show that this amalgamated approach of k-means with the GA works efficiently for the Korean stock market. But the working of this approach may fail to capture the actual shape of the market due to the concave nature of the k-means clusters.

The P-spline-based clustering method proposed in Iorio et al. (2018) constitutes in designing penalized spline smoothers for individual time series data frame. Each data frame is then subjected to a clustering approach on the spline coefficients thus computed. Thus segregating the entire dataset makes it possible to choose the assets from the generated clusters. The outcome of this research was a non-fluctuating portfolio of stocks from the market.

The research by (Nagy and Ormos 2018) using an approach established on the idea of spectral clustering demonstrates the dependency of the stock prices on network level data besides the individual firm. The research was done on a exhaustive investigation of 59 stock indices and the efficiency was measured by augmented Dicky-Fuller (ADF) test. The stocks were first clustered and then the equity index graph was generated using the historical daily closing prices.

Nakagawa, Imamura, and Yoshida (2019) used k-medoids algorithms with Indexing Dynamic Time Warping method to predict the number of price fluctuation clusters. The dataset considered here is the TOPIX dataset from the Japan stock market. Using the performance measures of average accuracy and total returns, it was evident that the approach was effective in predicting monthly stock price changes.

(Chandar 2019) introduced a technique established on subtractive clustering-based adaptive neuro fuzzy systems for the purpose of forecasting the prices of apple stock. The input to the neuro fuzzy model for forecasting the stock prices were four technical indicators. For evaluating the effectiveness of this approach, standard criteria like training and testing error, number of rules and parameters were computed. One of the drawbacks of the system was that every time the model has to be re-trained, it has to pause for a full-cycle.

Rukmi et al. (2019) used density-based clustering approach detect patterns of stock trading deviation. The dataset considered here are the stock trading transactions. DBSCAN helps in identifying the outlier transactions which have a very different pattern with respect to the conventional transactions. The primary objective of this work is to detect stock price manipulation as outlier transactions done by securities brokers.

Based on the ideas of Partitioning Around Medoids (PAM) and the fuzzy theory (D’Urso et al. 2020), developed a clustering technique for the stock market data depending on the predicted cepstrum. Cepstrum can be described as the spectrum of the logarithm of the spectral density function. The developed technique has been effective in computing the appropriate weights in the clustering approach, thus auto-tuning the importance of individual coefficients in the clustering process. The performance of the model was evaluated using Sharpe ratio. However, proposed method cannot handle the impact of outliers.

Parvatha et al. (2023) first uses sentiment analysis on raw data from various social media platforms to convert them into valuable data. Then DBSCAN is applied on the data to evaluate the best approach for stock prediction based on the parameters of DBSCAN algorithm. These outcomes obtained are then compared with the approaches like k-means, LSTM, CNN, with the parameters like MAE, MSE, RMSE.

Han, He, and Toh (2023) develops a pairs trading strategy via unsupervised learning and applied it to the US stock market from January 1980 to December 2020. Three common clustering approaches namely, k-means clustering, density-based spatial clustering of applications with noise (DBSCAN), and agglomerative clustering were implemented and results show that the long-short portfolio constructed via the agglomerative clustering earns a statistically significant annualized mean return. The portfolio f stocks thus obtained tends to have similar future price movement and possibilities for shorting overpriced stocks and longing underpriced stocks within the same cluster can be found.

Gupta et al. (2023) presents a perspective of improving the computation time using an unsupervised machine learning approach, namely, PCA and then using DBSCAN for determining the best pair of stocks. The investigation is evaluated on the historical National Stock Exchange data which indicates significant performance of the work with regard to the NIFTY-50 Global benchmark.

Proper analysis of the framework of the global stock market is essential to spread out the risk evenly so that equity portfolios can be constructed effectively. But building of a relevant portfolio is fairly sophisticated. The assumption made in most of the research works that the framework of the stock market is linear is not very useful is obtaining significant outcomes (Erdős, Ormos, and Zibriczky 2011; Maldonado and Saunders 1981; Song et al. 2011). Additionally, influence of external factors have a great influence on the framework of the market, wherein uncorrelated equities can embark upon changing simultaneously (Heiberger 2014). Also, in some cases imbalanced data poses challenges as well as opportunities in the context of credit risk (Mahbobi, Kimiagari, and Vasudevan 2021). In such practical scenarios, the minority classes are most likely to provide the required objectives.

The objective of this research was to investigate appropriate approaches for establishing the structure of the stock market based on correlation rather than linearity so that diversification becomes consistent. Spherical clusters detected by k-means type algorithms (Tan, Steinbach, and Kumar 2016) (k-means, k-mediods, fast k-means) may not often capture the real shape of the stock market data as they may be sometimes concave in nature. Also pre-determining the optimal number of clusters in case of k-means clustering is a fundamental issue.

Using density-based clustering approaches, little investigations (Gupta et al. 2023; Han, He, and Toh 2023; Parvatha et al. 2023) have been carried out in the past in the stock market data analysis those are only limited to DBSCAN clustering approach. However, no other density-based clustering approaches have been applied so far for stock market data analysis. The fundamental idea of density-based concept is to segregate the sample space by utilizing the data based on density rather than mean centroid. This concept helps to determine tangled as well as irregular shaped clusters (Tan, Steinbach, and Kumar 2016). Also, the different shape of the clusters (convex and concave) can also be detected by density-based models (Bhattacharjee and Mitra 2021; Chowdhury, Bhattacharyya, and Kalita 2021). Moreover, density-based techniques have proven to be able to determine optimal number of clusters from relevant data (Scitovski and Sabo 2020), thus enhancing the performance of clustering. Table 1 summarizes the notable works done toward stock market data analysis.

Table 1. Summary of related works.

Author details	Proposed system	Stocks selected	Performance parameters	Results	Limitations
Nanda, Mahanty, and Tiwari (2010)	k-means	Capitaline Databases Plus	Intraclass inertia	Construction of diversified portfolio of stocks	k-means fails to depict the actual shape of the market
Esfahanipour and Aghamiri (2010)	Fuzzy c means	Tehran Stock Exchange Indexes (TEPIX)	MAPE	Forecasted results in TSE index and Some Tehran Stock Exchange Indexes	Training and testing data required
Baser and Saini (2015)	k-means, k-medoids, fast k-means	Nifty 50	Intraclass inertia	Construction of diversified portfolio of stocks	k-means may fails to depict the actual shape of the market
Nair et al. (2017)	Regression trees and self-organizing maps	20 stocks	Training error, testing error, number of rules, number of parameters	.035, .072, 6, 67, respectively	Very less number of data used for the experiment
Cheong et al. (2017)	k-means amalgamated with genetic algorithm	Top 90 companies of the KOSPI 200	Equal weights, Market capital weights, Minimum variance weights, Sharpe ratio	Created portfolio stock weights based on minimized portfolio risk	Fails to capture the actual shape of the market due to concave nature of k-means approach
Iorio et al. (2018)	Penalized spline smoothers with k-means algorithm	S&P 500	Minimum variance weights, Equal weights, Equally-weighted risk contributions	Consistent financial portfolio of stocks	To rebalance the portfolio periodically
Nagy and Ormos (2018)	Spectral clustering-based method	59 stock indices	Augmented Dicky-Fuller (ADF) test	Filter out outliers and find robust clusters	Analyze the evolution of the network to identify patterns to understand the life cycle of hubs
Nakagawa, Imamura, and Yoshida (2019)	k-medoids with Indexing Dynamic Time Warping method	TOPIX	Average accuracy	Number of price fluctuation clusters	Can only detect convex shape clusters but not concave clusters
Chandar (2019)	Subtractive clustering-based adaptive neuro fuzzy approach	Apple	Training error, testing error, number of rules and parameters	Short-term fluctuations in the stock prices	For re-training, the models has to wait a full cycle
Rukmi et al. (2019)	DBSCAN	Indonesia Stock Exchange	Identifying the outlier transactions	To detect stock price manipulation done by securities brokers	Results obtained are shown in only one company
D’Urso et al. (2020)	Partitioning Around Medoids approach and the fuzzy theory	NASDAQ-100 stocks	Sharpe ratio	Stock clusters according to their excess return performance for a given risk level	Effect of the outliers is present
Parvatha et al. (2023)	Sentiment analysis followed by DBSCAN	Raw data from social media platform	MAE, MSE, RMSE	Finding best approach for stock prediction based on the parameters of DBSCAN algorithm.	Absence of standard dataset
Han, He, and Toh (2023)	k-means clustering, DBSCAN, agglomerative clustering	US stock market	Annualized mean return and Sharpe ratio	Effectiveness of long-short portfolio constructed via the agglomerative clustering	Factor models cannot explain the significant profits of the strategies.
Gupta et al. (2023)	PCA followed by DBSCAN	National Stock Exchange data	Computation time	To determine the best pair of stocks.	Only one optimization pair selection criteria is used.

Considering these factors, and as no density-based clustering-based technique except DBSCAN has been applied, therefore, in this research we tried to investigate the clustering of the stock companies using density-based clustering approaches such as Weighted Adaptive Mean Shift Clustering (WAMSC), Dynamic Quantum Clustering (DQC), along with DBSCAN, and Expectation Maximization (EM) algorithms to select the best performing model for diversified stock prediction.

Material and Methods

In this section, we present a short summary of the foundations of our research which deals with evaluation of best dynamic density-based clustering approach for stock market data analysis which has all the big data characteristics. That is we are pursuing to obtain dynamics-based clustering approach which will best suit for the Big data analysis also. We have discussed shortly about the different density-based clustering algorithms experimented. Section 3.1 gives a concise idea about Density-based spatial clustering of applications with noise (DBSCAN). Section 3.2 is associated with the technique of Expectation Maximization (EM). Section 3.3 provides the basics of Weighted Adaptive Mean Shift clustering and section 3.4 outlines Dynamic Quantum clustering whose performance was found to be the best for the purpose. Section 3.5 provides the details of the datasets used for this work.

Density-Based Spatial Clustering of Applications with Noise

Density-based spatial clustering of applications with noise (DBSCAN) is a clustering mechanism which is significantly robust for detecting irregular shaped clusters (Ester et al. 1996). It is capable of achieving satisfactory accuracy in clustering even in presence of noise (Yue et al. 2004).

The two fundamental attributes of DBSCAN are given by Eps and MinPts. Eps represents the cutoff distance of a point from the core point, to be considered a part of a cluster. MinPts represents the least possible number of data points necessary to form a cluster. The significance of these two attributes is that they are the deciding parameters to decide the the number of clusters in the given dataset (Starczewski and Cader 2019), which varies according to the data distribution.

Three types of data points are generated after the execution of the DBSCAN, algorithm. They are the core point, border point and noise. An instance is labeled as a core point if it has atleast, MinPts number of instances within its Eps neighborhood. An instance is termed as border point, if it has number of points less than MinPts within Eps neighborhood and also it must be within the neighborhood of a core point. The instances which cannot be categorized as a core point or a border point are considered as noise.

The determination of Eps in DBSCAN is an important step for determining the number of clusters. For this purpose, we have used an approach which was introduced by Rahmah and Sitanggang (2016). An acceptable value of Eps is determined by first computing the distance to the nearest number of points for each point. Then these distances are sorted and analyzed to see at which point the most distinct changes are observed. The point at which the change is most distinct is selected to be the Eps.

Procedures A1 and A2 provides the pseudo-code of the DBSCAN approach. Procedure A3 gives the working mechanism for determining Eps.

Procedure A1 is used for the formation of the clusters from the dataset by searching the neighborhood of each datapoint in the entire dataset. If accumulation of minimum number of data-points in the neighborhood of a core point is exhausted, then the next cluster is formed taking the subsequent data point as its center. The data which cannot be assigned to any clusters forms the noise.

Procedure A1: DBSCAN

Input:Dataset $\mathit{D}=x_1,x_2,\dots ,x_n$

Eps

Minpts

Output:Cluster centers $\mathit{y}(\mathit{j})$

Steps:Initialize $\mathit{j}=0$

For each data point $x_i$ (i = 1 to n) in dataset $\mathit{D}$

Mark $x_i$ as visited

Calculate NeighborPts = all points within $x_i{}^{‵}s$ Eps

If sizeof(NeighborPts) $<$ Minpts

mark $x_i$ as NOISE

Else

$\mathit{j}=\mathit{j}+1;$

fillCluster($x_i$, NeighborPts, $\mathit{j}$, Eps, Minpts)

End Procedure

Procedure A2 assigns the points which are reachable to the cluster centers as elements of the respective cluster.

Procedure A2: fillCluster($x_i$, NeighborPts, $\mathit{j}$, Eps, Minpts)

Input:Datapoint $x_i$

Neighboring points $\mathit{NeighborPts}$

Cluster number $\mathit{j}$

Eps

Minpts

Output:Labeled dataset $\mathit{p}(\mathit{j})(\mathit{i})$ where $j^{\mathit{th}}$ cluster filled with the $i^{\mathit{th}}$ data points

Steps:Add data point $x_i$ to $\mathit{p}(\mathit{j})(\mathit{i})$

For each point $\mathit{k}$ in NeighborPts

If $\mathit{k}$ is not visited

Mark $\mathit{k}$ as visited

NeighborPts2 = all points within $\mathit{ks}$ Eps

If sizeof(NeighborPts2) $\ge$ Minpts

NeighborPts = NeighborPts

concatenated with NeighborPts2

If $\mathit{k}$ is not yet member of any cluster

Add $\mathit{k}$ to cluster $\mathit{p}(\mathit{j})(\mathit{k})$

End Procedure

Procedure A3 helps to determine the optimal value of Eps, which is an essential step for determining the number of clusters produced by the algorithm DBSCAN.

Procedure A3: determineEps($x_i$)

Input:Datapoint $x_i$

Output:Eps for each varied density

Steps:For each data point $x_i$

Find distance to the other points

$\mathit{d}(\mathit{i},\mathit{j})$ $\leftarrow$ find distance $(x_i,x_j)$

Find the minimum values of distances

End for

Sort distances ascending and plot them to find each value

Eps corresponds to critical change in the curve

End Procedure

Expectation Maximization

The Expectation Maximization (EM) algorithm was originally proposed to detect the maximum likelihood estimates of parameters in probabilistic paradigms (Dempster, Laird, and Rubin 1977). The two principal steps adapted here includes expectation (E) and maximization (M). These two steps are used to discover the variables which are latent in the available observed data (Kurban, Jenne, and Dalkilic 2017). The EM algorithm can be used to construct statistical models from huge datasets (Fayyad, Bradley, and Reina 2001).

In order to achieve the required convergence while clustering large datasets, the EM approach uses the various Gaussian-based models (Verma, Dwivedi, and Sevakula 2015). For the initial parameters, describing each cluster $\mathit{k}$ namely, the mean ${\mu }_k$, variance ${\sigma }_k$ and size ${\pi }_k$, the joint probability distribution ($\mathit{\theta }(\mathit{x})$) of a data point $\mathit{x}$ can be described by

(1) $\mathit{\theta }(\mathit{x})=\sum _k{\pi }_k\mathit{N}(\mathit{x};{\mu }_k,{\sigma }_k)$(1)

where $\mathit{N}$ is the Gaussian probability density function (Aldershof et al. 1995). Then the sample $\mathit{x}$ can initially be assigned to the parametric model, in the present case a particular cluster, as

(2) $\mathit{\theta }(\mathit{x}|\mathit{z}=\mathit{k})=\mathit{N}(\mathit{x};{\mu }_k,{\sigma }_k)\sim f_{{\theta }_k}(\mathit{x})$(2)

where $\mathit{z}$ is latent variable for observing $\mathit{x}$. The presence of the unknown value of $\mathit{z}$ helps to explain the patterns in the value of $\mathit{x}$, for example, groups or clusters.

EM then proceeds iteratively in two steps. In the first step, the expectation or E-step, for the initial Gaussian parameters, mean ${\mu }_k$ and size ${\pi }_k$, responsibility (${\gamma }_{\mathit{i},\mathit{k}}$) of the $i^{\mathit{th}}$ data point ($x_i$) and $k^{\mathit{th}}$ cluster, is computed as

(3) ${\gamma }_{\mathit{i},\mathit{k}}=\frac{{\pi }_k{f}_{{\theta }_k}(x_i)}{{\sum }_{\mathit{k}=1}^K{\pi }_k{f}_{{\theta }_k}(x_i)},\mathrm{\forall i}\in \{1,2,..\mathit{n}\}\mathrm{\forall k}\in \{1,2,\dots \mathit{K}\}$(3)

where $f_{{\theta }_k}(x_i)$ is the initial probability that the point $x_i$ belongs to the cluster $\mathit{k}$.

Equation (3)(3) ${\gamma }_{\mathit{i},\mathit{k}}=\frac{{\pi }_k{f}_{{\theta }_k}(x_i)}{{\sum }_{\mathit{k}=1}^K{\pi }_k{f}_{{\theta }_k}(x_i)},\mathrm{\forall i}\in \{1,2,..\mathit{n}\}\mathrm{\forall k}\in \{1,2,\dots \mathit{K}\}$(3) measures the relative probability of data point $x_i$ belonging to cluster $\mathit{k}$. If $x_i$ is very likely under the $k^{\mathit{th}}$ Gaussian, it gets higher weight. The denominator helps to make $\mathit{\gamma }$’s sum to one.

In the second step of EM, the maximization or M-step, for the fixed assigned responsibilities ${\gamma }_{\mathit{i},\mathit{k}}$, updating of the parameters mean $\mathit{\mu }$ and size $\mathit{\pi }$ is done. For each cluster $\mathit{k}$, updating of its parameter using an estimate weighted by the parameters ${\gamma }_{\mathit{i},\mathit{k}}$ given by Equation (4)(4) $\mathit{\pi }=\mathit{arg}\mathit{ma}{x}_{\pi \sum {\pi }_k=1}\sum _{\mathit{k}=1}^K\mathit{log}[{\pi }_k]\left(\sum _{\mathit{i}=1}^n{\gamma }_{\mathit{i},\mathit{k}}\right)$(4) and Equation (5)(5) ${\theta }_k=\mathit{arg}\mathit{ma}{x}_{\theta }\sum _{\mathit{i}=1}^n{\gamma }_{\mathit{i},\mathit{k}}\mathit{log}{f}_{{\theta }_k}(x_i),\mathrm{\forall k}\in \{1,2,\dots \mathit{K}\}$(5) is done.

(4) $\mathit{\pi }=\mathit{arg}\mathit{ma}{x}_{\pi \sum {\pi }_k=1}\sum _{\mathit{k}=1}^K\mathit{log}[{\pi }_k]\left(\sum _{\mathit{i}=1}^n{\gamma }_{\mathit{i},\mathit{k}}\right)$(4)

(5) ${\theta }_k=\mathit{arg}\mathit{ma}{x}_{\theta }\sum _{\mathit{i}=1}^n{\gamma }_{\mathit{i},\mathit{k}}\mathit{log}{f}_{{\theta }_k}(x_i),\mathrm{\forall k}\in \{1,2,\dots \mathit{K}\}$(5)

Equation (3)(3) ${\gamma }_{\mathit{i},\mathit{k}}=\frac{{\pi }_k{f}_{{\theta }_k}(x_i)}{{\sum }_{\mathit{k}=1}^K{\pi }_k{f}_{{\theta }_k}(x_i)},\mathrm{\forall i}\in \{1,2,..\mathit{n}\}\mathrm{\forall k}\in \{1,2,\dots \mathit{K}\}$(3) gives the weighted mean of the assigned data while Equation (4)(4) $\mathit{\pi }=\mathit{arg}\mathit{ma}{x}_{\pi \sum {\pi }_k=1}\sum _{\mathit{k}=1}^K\mathit{log}[{\pi }_k]\left(\sum _{\mathit{i}=1}^n{\gamma }_{\mathit{i},\mathit{k}}\right)$(4) gives the weighted co-variance (newly calculated co-variance) of the assigned data. It means that if $x_i$ is a strong member of cluster $\mathit{k}$, the value of the assigned weight will be close to 1.

Procedure B provides the pseudo-code for the two fundamental functions of the EM approach. The first step tries to find the missing variables and the second step tries to optimize the estimates of the parameters used in the procedure so that it describes the data better.

Procedure B: Expectation Maximization

Input:Data points $\mathit{D}=x_1,x_2,\dots .x_n$

Parametric model $f_{\theta }$

Output:Clusters ${\theta }^t$

Steps:Choose an initial distribution ${\pi }^0$ in the probability simplex and pick a parameter ${\theta }^0$ randomly in its state space.

Denote ${\theta }^0=({\pi }^0,{\theta }^0)$

For each cluster $\mathit{t}$ = $1$ to $\mathrm{\infty }$ (until convergence)

E step to compute the expected responsibilities for

each observation $\mathit{i}$ = $1$ to $\mathit{n}$

${\gamma }_{i,k}^t=\frac{{\pi }_k^{t-1}{f}_{{\theta }_k}^{\mathit{t}-1}(x_i)}{{\displaystyle {\sum }_{k=1}^K{\pi }_k^{t-1}{f}_{{\theta }_k}^{\mathit{t}-1}(x_i)}}$

${\gamma }_{i,k}^t=\frac{{\pi }_k^{t-1}{f}_{{\theta }_k}^{\mathit{t}-1}(x_i)}{{\displaystyle {\sum }_{k=1}^K{\pi }_k^{t-1}{f}_{{\theta }_k}^{\mathit{t}-1}(x_i)}}$

M step to update the parameters ${\theta }^t=({\pi }^t,{\theta }^t)$

${\pi }^t$ = arg $\mathit{ma}{x}_{\pi \sum {\pi }_k=1}{\sum }_{\mathit{k}=1}^K\mathit{log}[{\pi }_k]({\sum }_i^n={\gamma }_{\mathit{i},\mathit{k}}^t)$

and

$\mathrm{\forall k}\in \{1,2,\dots \mathit{K}\},$

${\theta }_k^t$ = arg $\mathit{ma}{x}_{\theta }{\sum }_{\mathit{i}=1}^n{\gamma }_{\mathit{i},\mathit{k}}^t\mathit{log}{f}_{{\theta }_k}(x_i)$

Return limiting values in ${\theta }^t$

End Procedure

Weighted Adaptive Mean Shift Clustering

The objective of the original mean shift clustering (MSC) methodology (Fukunaga and Hostetler 1975) was to determine clusters by constantly updating the possible candidates for the centroid. The basis for this algorithm is the kernel density estimation (Weglarczyk 2018) function, which, helps to place each point to its respective cluster. The performance of traditional MSC degrades with the increase in the dimensions of data used (Hyrien and Baran 2016).

To address this limitation, Weighted Adaptive Mean Shift Clustering (WAMSC) was proposed (Ren et al. 2014). This is basically an extension of the MSC algorithm. This adaptation of weights is done by approximating the importance of features for each data point $x_i$ and then computing the distance between $x_i$ to the remaining data points in the resulting subset ($\mathit{x}$) of important features (Ren et al. 2014). Given a subspace $w_t$, the distance of a point $x_t$ from the rest of the points in $\mathit{x}$ is calculated as

(6) $D_{\mathit{wt}}(x_t,\mathit{x})=\frac{{\sum }_{\mathit{i}=1}^d{w}_{\mathit{il}}|x_{\mathit{il}}-x_l|}{s_l}$(6)

where $s_l$ is the average $l^{\mathit{th}}$ attribute distance of all pairs of points in $\mathit{x}$, which is given by

(7) $s_l=\frac{1}{\left(\begin{array}{c}\mathit{n}\\ 2\end{array}\right)}\sum _{\mathit{i}\ne \mathit{j}}|x_{\mathit{il}}-x_{\mathit{jl}}|$(7)

Equation 8 gives the values of the density estimation points

(8) $y_{\mathit{t}+1}=\frac{{\displaystyle {\sum }_{i=1}^n\frac{x_i}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}{{\displaystyle {\sum }_{i=1}^n\frac{1}{h_t^{d+2}}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}$(8)

where $h_t$ denotes the distance between $x_t$ and its $k^{\mathit{th}}$ nearest neighbor and $\mathit{g}(\mathit{x})=-\mathit{D}{ }^{\mathrm{\prime }}(x_t,\mathit{x})$, provided that the derivative of $\mathit{D}$ exists.

The corresponding weighted mean shift vector is evaluated using Equation 9,

(9) $\mathit{m}(y_t)=\frac{{\displaystyle {\sum }_{i=1}^n\frac{x_i}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}{{\displaystyle {\sum }_{i=1}^n\frac{1}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}-y_t$(9)

Equation 9(9) $\mathit{m}(y_t)=\frac{{\displaystyle {\sum }_{i=1}^n\frac{x_i}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}{{\displaystyle {\sum }_{i=1}^n\frac{1}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}-y_t$(9) ensures that this weighted mean shift algorithm performs effectively while dealing with high-dimensional data.

Procedure C outlines the steps of WAMSC. It attempts to perform clustering by identifying the relevant sub-space and then incorporating this information within the Mean Shift vector. This will result in a methodology which is capable of enhancing the clustering speed without compromising accuracy.

Procedure C: Weighted Adaptive Mean Shift Clustering

Input:Data points $\mathit{D}=x_1,x_2,\dots .x_n$

Output:Cluster centers $\mathit{y}(\mathit{t})$

Steps:Initialize $\mathit{i}=0$, $\mathit{t}=1$ and $y_0\leftarrow x_i$

For each cluster $\mathit{t}$ = $1$ to $\mathrm{\infty }$ (Until convergence)

Calculate the density estimation points $y_{\mathit{t}+1}$

$y_{\mathit{t}+1}=\frac{{\displaystyle {\sum }_{i=1}^n\frac{x_i}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}{{\displaystyle {\sum }_{i=1}^n\frac{1}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}$

Calculate the corresponding Mean Shift Vector $\mathit{m}(y_t)$

$\mathit{m}(y_t)=\frac{{\displaystyle {\sum }_{i=1}^n\frac{x_i}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}{{\displaystyle {\sum }_{i=1}^n\frac{1}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}-y_t$

Return $\mathit{y}(\mathit{t})$

End Procedure

Dynamic Quantum Clustering

The Quantum clustering approach was introduced by Horn and Gottlieb (Horn and Gottlieb 2002). This technique is based on the fundamentals of quantum mechanics. The algorithm basically calculates the potential function from the Schrödinger potential equation. It then places each data point to its respective cluster using the gradient descent mechanism (Horn and Gottlieb 2002). The drawback of this algorithm is that it becomes computationally expensive with the increase in the volume of the data.

To overcome this limitation, Dynamic Quantum Clustering (DQC) algorithm was proposed by Weinstein and Horn (Weinstein and Horn 2009). This approach is effective in handling large datasets. The DQC algorithm, uses time-dependent part of the Schrodinger equation, for allocating data points to its corresponding cluster, thus making it suitable for clustering big data (Weinstein et al. 2013).

The algorithm starts by assigning a Gaussian $\mathit{\psi }$ to a data point $\mathit{x}$,

(10) $\mathit{\psi }(\mathit{x})=\sum _{\mathit{i}=1}^n{e}^{-\frac{{\left(x-x_i\right)}^2}{2{\sigma }^2}}$(10)

with width $\mathit{\sigma }$ to the $\mathit{n}$ data points.

Now, from the assigned Gaussian kernel, $\mathit{\psi }$, of the data points and the width, $\mathit{\sigma }$, between each data points, the potential function ($\mathit{V}(\mathit{x})$) is calculated as (Deodhar et al. 2008):

(11) $\mathit{V}(\mathit{x})=\frac{1}{2{\sigma }^2\mathit{\psi }}\sum _{\mathit{i}=1}^n{\left(x-x_i\right)}^2{e}^{-\frac{{\left(x-x_i\right)}^2}{2{\sigma }^2}}$(11)

The cluster centers for the set of points, $x_i$, are then determined from the local minima of the potential function $\mathit{V}(\mathit{x})$ (Deodhar et al. 2008). If $\mathit{\psi }(x_i)$ is the Gaussian localized around the the $i^{\mathit{th}}$ data point $x_i$, then the relation of $x_i$ to its respective cluster center, $\mathit{V}(\mathit{x})$, is derived using the following time-dependent trajectory as

(12) $x_i=⟨\mathit{\psi }(x_i)|\mathit{x}|\mathit{\psi }(x_i)⟩$(12)

If $\mathit{\psi }(x_i)$ follows a narrow Gaussian, then it can be ascertained that each data point moves toward the nearest minimum of the potential. The main advantage of this methodology is that it is capable of searching the connections among the closely related data-points even in huge spaces.

Procedure D describes the dynamic quantum clustering approach which can be used for clustering big data to find associated groups from it. The parameter $\mathit{\sigma }$ regulates the proximity of the data points in the respective clusters. The smaller the value of $\mathit{\sigma }$, higher will be the number clusters with less number of instances in the individual clusters. For significantly smaller values of $\mathit{\sigma }$, all the instances are grouped in their own distinct clusters. On the other hand, for larger values of $\mathit{\sigma }$, all the instances tends to be in the same cluster.

Procedure D: Dynamic Quantum Clustering

Input:Data points $\mathit{D}=x_1,x_2,\dots .x_n$

Output:Cluster centers $\mathit{y}(\mathit{i})$

Steps:$\mathit{n}$ = the number of data points

Calculate the Gaussian

$\mathit{\psi }(\mathit{x})={\sum }_{\mathit{i}=1}^n{e}^{-\frac{{\left(x-x_i\right)}^2}{2{\sigma }^2}}$

For each data point $\mathit{i}$ = $1$ to $\mathit{n}$

Calculate the Potential Function for each $x_i$ as per Equation (11)(11) $\mathit{V}(\mathit{x})=\frac{1}{2{\sigma }^2\mathit{\psi }}\sum _{\mathit{i}=1}^n{\left(x-x_i\right)}^2{e}^{-\frac{{\left(x-x_i\right)}^2}{2{\sigma }^2}}$(11)

for $\mathit{i}$ = $1$ to $\mathit{n}$

Assign data to the clusters as per Equation (12)(12) $x_i=⟨\mathit{\psi }(x_i)|\mathit{x}|\mathit{\psi }(x_i)⟩$(12)

Return $\mathit{y}(\mathit{i})$

End Procedure

Dataset Description

Here we have used the historical time series data of S&P 500 obtained from the Quandl stock exchange data (https://www.quandl.com/) (accessed on $4^{\mathit{th}}$ January 2017). The S&P 500 (https://fred.stlouisfed.org/series/SP500) (accessed on $5^{\mathit{th}}$ April 2018) contains the most prevailing stocks traded on American stock exchanges. In this investigation, the period of stock market data under consideration was spanning from $4^{\mathit{th}}$ January 2016 to $4^{\mathit{th}}$ January 2018.

The dataset consists of all the facts concerning the day-to-day properties of the stock quotes, consisting of

• Date of the complete stock exchange time period

• The starting price at the opening of the time period

• Maximum price reached during the time period

• Minimum price attained during the time period

• Ending price of the given time period

• Entire number of transactions during this time period

• Rectified closing price

The fundamental format of the dataset is given in Table 2. The prominent discriminating characteristics of this data is that the arrangement of the dataset matters, on account of their appended time tags.

Table 2. Structure of the stock market dataset.

Date	Open	High	Low	Close	Adj Close	Volume
2017-05-01	2388.50	2394.49	2384.83	2388.33	2388.33	3.19E+09
2017-05-02	2391.05	2392.93	2385.82	2391.17	2391.17	3.81E+09
2017-05-03	2386.50	2389.82	2379.75	2388.13	2388.13	3.89E+09
2017-05-04	2389.797	2391.43	2380.35	2389.52	2389.52	4.36E+09
2017-05-05	2392.37	2399.29	2389.38	2399.29	2399.29	3.54E+09

In the domain of stock market, volatility (Day and Lewis 1992) illustrates the fluctuations in the prices of the stocks for a specific time span. Volatility is determined by constantly computing the standard deviation of the prices for the time period under consideration. From the dataset described in Table 2, we first calculate the returns of each stock using the standard method of dividing each closing price by the closing price of the day before and subtracting one (Fan and Yao 2017). The standard deviation of all the returns of a particular stock for the considered time frame is then computed as the respective volatilities for each stock.

Experimental Evaluation

The aim of this work is to cluster the companies of the same business stature using density based clustering methods. This is done using the data available from the stock market depending on their market movement. The two important steps in this regard is the assessment of the market movement and the determination of number of clusters, which is discussed in sub sections 4.2 and 4.1 respectively. The experiment is discussed in brief in sub section 4.3.

Determination of Number of Clusters

Since all the clustering methods applied are density based, therefore the number of clusters is automatically determined by the respective algorithms depending upon their parameter values as described below.

The number of clusters determined by the DBSCAN algorithm which is dependent on the value Eps, is determined by the approach given by Rahmah and Sitanggang (2016), which is explained in Procedure A3. Based on the average optimal value of Eps, the parameter Eps generated produced 4 clusters with outliers at a few points. The number of clusters in case of the EM clustering algorithm is the limiting values of ${\theta }^t$ as given in Equation (5)(5) ${\theta }_k=\mathit{arg}\mathit{ma}{x}_{\theta }\sum _{\mathit{i}=1}^n{\gamma }_{\mathit{i},\mathit{k}}\mathit{log}{f}_{{\theta }_k}(x_i),\mathrm{\forall k}\in \{1,2,\dots \mathit{K}\}$(5) which is found to be 3. The modes $y_{\mathit{t}+1}$ calculated by the WAMSC as given in Equation (8)(8) $y_{\mathit{t}+1}=\frac{{\displaystyle {\sum }_{i=1}^n\frac{x_i}{h_t^{d+2}}\mathit{g}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}{{\displaystyle {\sum }_{i=1}^n\frac{1}{h_t^{d+2}}{\left(\frac{D_w\mathit{t}(x_t,y_t)}{h_t}\right)}^2}}$(8) gives the number of clusters which is computed to be 3. Finally, for DQC the number of clusters determined by Equation (11)(11) $\mathit{V}(\mathit{x})=\frac{1}{2{\sigma }^2\mathit{\psi }}\sum _{\mathit{i}=1}^n{\left(x-x_i\right)}^2{e}^{-\frac{{\left(x-x_i\right)}^2}{2{\sigma }^2}}$(11) is found to be 5.

For the partitive clustering algorithms, the number of optimal clusters by implementing the k-means technique described in (Nanda, Mahanty, and Tiwari 2010) is computed to be 4, the k-medoids (Nakagawa, Imamura, and Yoshida 2019) is computed to be 6 and using the fuzzy c means technique described in (Esfahanipour and Aghamiri 2010) is computed to be 5 when applied on the stock market dataset considered in this study.

Assessment of the Market Movement

For experimentation purpose, the pricing data for the 3000 stocks from the S&P 500 has been taken into consideration. The stocks are grouped into distinct clusters depending upon their historic returns and volatilities. As the time tag is attached with each observation in the dataset, the clusters are derived depending on volatility ((Day and Lewis 1992)), which are basically the variations of the stock prices with respect to time.

If $r_x$ is the average rate of return of a stock $\mathit{x}$, then volatility of $\mathit{x}$ is defined as

(13) $\mathit{volatilit}{y}_x=\mathit{StdDev}(r_x)$(13)

where $\mathit{StdDev}(r_x)$ is the standard deviation on the return of $\mathit{x}$, which basically represents the difference in historic returns of a stock ((Day and Lewis 1992)).

Experimentation

Thus the list of stocks from the S&P 500 are assigned to its corresponding clusters based on their returns and volatility characteristics. Stocks belonging to clusters having high volatility, is chosen for investment choices. It is based on the fundamental rule that the performance of stocks with high volatility are significantly higher than the stocks with low volatility (Pandey and Sehgal 2017).

From the cluster having high volatility, two of the stocks were chosen for investment choices. And for evaluation of the results some of the most commonly used metrics for quantifying clustering applications were used for the two chosen stocks. These validity measures include Normalized Mutual Information ($\mathit{NMI}$) (Estévez et al. 2009), Adjusted Rand Index ($\mathit{ARI}$) (Yeung and Ruzzo 2001), Jaccard Index ($\mathit{JI}$) (Halkidi, Batistakis, and Vazirgiannis 2001) and Fowlkes Mallows Index ($\mathit{FM}$) (Campello 2007). The details of these performance measures are discussed in sub-section 4.4.

In order to frame the trading strategies, the two of the stock companies from the cluster having high volatility are chosen. The concept of moving averages have been applied to them. The concept of moving averages helps in identification of stock trends. Moving average (Chang et al. 2018) is a technical indicator, generally used for computing the future trend of a stock based on its past prices. It is calculated by averaging the closing prices spanning over the one year period. The moving average values, thus calculated, are cascaded to create a singular flowing line, depicting the market trend.

Validity Measures

Normalized Mutual Information

To find the efficiency of clustering, Normalized mutual information ($\mathit{NMI}$) follows the principle of entropy-based measures (McDaid, Greene, and Hurley 2011). Entropy is useful for evaluating the measure of orderliness of the information in all the partitions.

Entropy of a clustering, $\mathit{C}$, is measured as:

(14) $\mathit{H}(\mathit{C})=-\sum _{\mathit{i}=1}^r{p}_{C_i}log{p}_{C_i}$(14)

where $\mathit{r}$ is the total number of clusters obtained and $p_{C_i}$ is the probability of the $i^{\mathit{th}}$ cluster ($C_i$). $p_{C_i}$ is calculated as:

(15) $p_{C_i}=\frac{n_i}{\mathit{n}}$(15)

which is basically the number of instances ($n_i$) in $i^{\mathit{th}}$ particular cluster divided by the total number of instances ($\mathit{n}$).

Then entropy of the actual partitioning of the dataset, $\mathit{T}$, is defined in a similar way:

(16) $\mathit{H}(\mathit{T})=-\sum _{\mathit{j}=1}^k{p}_{T_j}log{p}_{T_j}$(16)

where $\mathit{k}$ is the actual number of clusters.

Here $p_{T_j}$ is the probability of the actual $j^{\mathit{th}}$ partition ($T_j$). $p_{T_j}$ is calculated as:

(17) $p_{T_j}=\frac{n_j}{\mathit{n}}$(17)

where the $n_j$ is the number of instances in the actual $j^{\mathit{th}}$ partition and $\mathit{n}$ is the total number of instances.

Exploiting the idea of the entropy-based measure, Mutual information, $\mathit{I}(\mathit{C},\mathit{T})$ quantifies the aggregate information similar amidst obtained clustering C and actual partitioning T using the following equation:

(18) $\mathit{I}(\mathit{C},\mathit{T})=\sum _{\mathit{i}=1}^r\sum _{\mathit{j}=1}^k{p}_{\mathit{ij}}log\frac{{\mathit{p}}_{\mathit{ij}}}{p_{C_i}{p}_{T_j}}$(18)

where $\mathit{r}$ is the number of clusters obtained and $\mathit{k}$ is the number of actual clusters. $p_{\mathit{ij}}$ measures the observed joint probability of $\mathit{C}$ and $\mathit{T}$.

Normalized mutual information, $\mathit{NMI}(\mathit{C},\mathit{T})$ to find how accurate the obtained outcome is to the actual knowledge is calculated as:

(19) $\mathit{NMI}(\mathit{C},\mathit{T})=\frac{\mathit{I}(\mathit{C},\mathit{T}}{\sqrt{\mathit{H}(\mathit{C}).\mathit{H}(\mathit{T})}}$(19)

Equation 19(19) $\mathit{NMI}(\mathit{C},\mathit{T})=\frac{\mathit{I}(\mathit{C},\mathit{T}}{\sqrt{\mathit{H}(\mathit{C}).\mathit{H}(\mathit{T})}}$(19) returns the normalized mutual information which is a value ranging between .0 and 1.0. $\mathit{NMI}(\mathit{C},\mathit{T})$ values close to 1 infers that the cluster labeling is accurate.

Adjusted Rand Index

The Adjusted Rand Index ($\mathit{ARI}$) gives an estimate of similarity of the two clustering assignments for the predicted and actual values for the entire set of data points (Yeung and Ruzzo 2001).

Let $\mathit{U}=\{u_1,\dots ,u_R\}$ be the set of actual clusters and $\mathit{V}=\{v_1,\dots ,v_C\}$ be the set of clusters obtained for a set of $\mathit{n}$ objects $\mathit{S}=\{o_1,\dots o_n\}$.

If $n_{\mathit{ij}}$ be the number of objects present in both the actual cluster $u_i$ and the obtained cluster $v_j$; and the number of objects in the actual cluster $u_i$ and obtained cluster $v_j$ are $n_i$ and $n_j$ respectively, then the Adjusted Rand Index is calculated as:

(20) $\mathit{ARI}=\frac{{\sum }_i{\sum }_j{ }^{{\mathit{n}}_{\mathit{ij}}}{C}_2-\frac{\left[{\sum }_i{ }^{n_i}{C}_2{\sum }_j{ }^{n_j}{C}_2\right]}{\mathit{n}{C}_2}}{\frac{\frac{1}{2}\left[{\sum }_i{ }^{n_i}{C}_2+{\sum }_j{ }^{n_j}{C}_2\right]-\left[{\sum }_i{ }^{n_i}{C}_2{\sum }_j{ }^{n_j}{C}_2\right]}{\mathit{n}{C}_2}}$(20)

where ${ }^n{C}_r$ represents the combination of $\mathit{r}$ objects from $\mathit{n}$.

Equation (20)(20) $\mathit{ARI}=\frac{{\sum }_i{\sum }_j{ }^{{\mathit{n}}_{\mathit{ij}}}{C}_2-\frac{\left[{\sum }_i{ }^{n_i}{C}_2{\sum }_j{ }^{n_j}{C}_2\right]}{\mathit{n}{C}_2}}{\frac{\frac{1}{2}\left[{\sum }_i{ }^{n_i}{C}_2+{\sum }_j{ }^{n_j}{C}_2\right]-\left[{\sum }_i{ }^{n_i}{C}_2{\sum }_j{ }^{n_j}{C}_2\right]}{\mathit{n}{C}_2}}$(20) gives the adjusted random index which returns a value between −1.0 and 1.0. A value of 1.0 means a perfect match for the clustering labels while a value of −1.0. means random labeling.

Jaccard Index

To calculate the consistency between the results of actual and obtained clustering, the Jaccard Index ($\mathit{JI}$) (Yang 2016) is calculated as

(21) $\mathit{JI}=\frac{{\sum }_i{\sum }_{{\mathit{j}}^{{\mathit{n}}_{\mathit{ij}}}}{C}_2}{{\sum }_i{ }^{n_i}{C}_2+{\sum }_j{ }^{n_j}{C}_2-{\sum }_i{ }^{{\mathit{n}}_{\mathit{i},\mathit{j}}}{C}_2}$(21)

where $n_{\mathit{ij}}$ is the number of objects in both the actual cluster $u_i$ and the obtained cluster $v_j$ and the number of objects in cluster $u_i$ and cluster $v_j$ being $n_i$ and $n_j$, respectively.

Equation 21(21) $\mathit{JI}=\frac{{\sum }_i{\sum }_{{\mathit{j}}^{{\mathit{n}}_{\mathit{ij}}}}{C}_2}{{\sum }_i{ }^{n_i}{C}_2+{\sum }_j{ }^{n_j}{C}_2-{\sum }_i{ }^{{\mathit{n}}_{\mathit{i},\mathit{j}}}{C}_2}$(21) returns the Jaccard index which calculates the similarity between the two sets of data. The range of this index is between .0 to 1.0. Greater the value of this indicator, higher the similarity between the actual and obtained results.

Fowlkes Mallows Index

The Fowlkes-Mallows ($\mathit{FM}$) index estimates the efficiency of the clustering algorithm. Here the second clustering is assumed to be the ground-truth (Wagner and Wagner 2007).

For $\mathit{n}$ number of instances and $\mathit{k}$ number of clusters in the clustering $\mathit{A}$ and $\mathit{B}$, the Fowlkes-Mallows index is calculated as:

(22) $\mathit{F}{M}_k=\frac{T_k}{\sqrt{P_k{Q}_k}}$(22)

where$T_k=\left({\sum }_{\mathit{i}=1}^k{\sum }_{\mathit{j}=1}^k{m}_{\mathit{ij}}^2\right)-\mathit{N}$

$P_k=\left({\sum }_{\mathit{i}=1}^k{\left({\sum }_{\mathit{j}=1}^k{m}_{\mathit{ij}}\right)}^2\right)-\mathit{N}$

$Q_k=\left({\sum }_{\mathit{j}=1}^k({\sum }_{\mathit{i}=1}^k{m}_{\mathit{ij}}{)}^2\right)-\mathit{N}$ and

$m_{\mathit{ij}}$ is the number of instances in the $i^{\mathit{th}}$ $j^{\mathit{th}}$ position of the ($\mathit{kXk}$) matrix which is constructed by the number of instances that lie in the $i^{\mathit{th}}$ cluster of clustering algorithm A and the $j^{\mathit{th}}$ cluster of a clustering algorithm B.

In simpler terms, Fowlkes-Mallows index can also be expressed as:

(23) $\mathit{FM}=\frac{\mathit{TP}}{\sqrt{(}\mathit{TP}+\mathit{FP})(\mathit{TP}+\mathit{FN})}$(23)

where

TP denotes the count of true-positive cases

FP denotes the count of false-positive cases

FN denotes the count of false-negative cases

The value produced by Equation (23)(23) $\mathit{FM}=\frac{\mathit{TP}}{\sqrt{(}\mathit{TP}+\mathit{FP})(\mathit{TP}+\mathit{FN})}$(23) ranges from .0 to 1.0. A higher score of FM infers a conformity between the ground truth and the obtained results.

Results and Discussion

The four density-based clustering approaches namely, density-based spatial clustering of applications with noise (DBSCAN), Expectation Maximization (EM), Weighted Adaptive Mean Shift Clustering (WAMSC) and Dynamic Quantum Clustering (DQC) were applied on the time series data. This was found to be very effective in creating portfolios having a collection of stocks with diverse properties. Figure 1 illustrates the results of clustering of the stocks into different similarity groups (represented with different colors) on the basis of their returns and volatilities using the four clustering algorithms. Experimental results suggest that the algorithms can detect the noisy points/outliers correctly as shown in Figure 1 where the black color dots represent the outliers which are situated far away.

Figure 1. Results of clustering of stocks into different similarity groups on the basis of their returns and volatilities formed by (a) DBSCAN (b) EM (c) WAMSC (d) DQC algorithm.

The models are used for executing statistically informed trade decisions. It is clear from the figures that the results obtained from DBSCAN and EM are somewhat skewed and this makes it quite difficult to infer a logical explanation of the market. On the contrary, WAMSC and DQC shows stock clusters with both high and low coverage.

From the clusters thus generated from the Dynamic Quantum clustering approach based on the returns and volatilities, we identified the stocks belonging to cluster with the highest volatility. The reason being that greater the amount of volatility, the greater the potential return. From such a pool of stocks, two of the stocks selected in the present case were Apple (AAPL) and Microsoft (MSFT).

Table 3 summarizes the performance of the DBSCAN, expectation maximization (EM), weighted adaptive mean shift clustering (WAMSC) and dynamic quantum clustering (DQC) on the stock returns of AAPL and MSFT based on four performance metrics, namely, $\mathit{NMI}$, $\mathit{ARI}$, $\mathit{JI}$ and $\mathit{FM}$. Also, for comparing the results with counterpart techniques from the literature, namely, k-means (Nanda, Mahanty, and Tiwari 2010),k-medoids Nakagawa, Imamura, and Yoshida (2019) and fuzzy c means (Esfahanipour and Aghamiri 2010) applied on the same stock market dataset. The standard deviations of accuracies obtained from 10 simulations are also shown using $\pm$ sign corresponding to each percentage accuracy in Table 3. From the table, it is evident that in terms of all the metrics DQC outperformed all the other compared methods.

Table 3. Summary of the average experimental results (in term of accuracy $\mathit{NMI}$, $\mathit{ARI}$, $\mathit{JI}$ and $\mathit{FM}$) on the two stock market dataset, namely, AAPL and MSFT.

Dataset	Methods	Accuracy (%)	$\mathit{NMI}$	$\mathit{ARI}$	$\mathit{JI}$	$\mathit{FM}$
	DBSCAN	$82.0\pm 1.41$	$0.872\pm 0.007$	$0.852\pm 0.006$	$0.849\pm 0.008$	$0.864\pm 0.009$
	EM	$72.0\pm 2.31$	$0.742\pm 0.011$	$0.695\pm 0.011$	$0.719\pm 0.008$	$0.738\pm 0.010$
	WAMSC	$66.0\pm 2.21$	$0.869\pm 0.008$	$0.832\pm 0.006$	$0.792\pm 0.007$	$0.847\pm 0.011$
	DQC	$84.4\pm 1.77$	$0.719\pm 0.009$	$0.760\pm 0.006$	$0.725\pm 0.007$	$0.741\pm 0.007$
	k-means (Nanda, Mahanty, and Tiwari 2010)	$72.8\pm 1.2$	$0.732\pm 0.016$	$0.725\pm 0.011$	$0.700\pm 0.008$	$0.743\pm 0.010$
AAPL	k-medoids (Nakagawa, Imamura, and Yoshida 2019)	$71.5\pm 1.4$	$0.729\pm 0.016$	$0.713\pm 0.009$	$0.699\pm 0.014$	$0.722\pm 0.019$
	Fuzzy c means (Esfahanipour and Aghamiri 2010)	$70.0\pm 1.81$	$0.722\pm 0.013$	$0.705\pm 0.012$	$0.689\pm 0.006$	$0.715\pm 0.012$
	DBSCAN	$84.4\pm 2.22$	$0.884\pm 0.015$	$0.864\pm 0.022$	$0.866\pm 0.025$	$0.859\pm 0.009$
	EM	$74.0\pm 1.49$	$0.685\pm 0.037$	$0.670\pm 0.024$	$0.670\pm 0.016$	$0.679\pm 0.028$
	WAMSC	$66.5\pm 3.43$	$0.876\pm 0.019$	$0.847\pm 0.015$	$0.851\pm 0.014$	$0.843\pm 0.009$
	DQC	$86.6\pm 2.46$	$0.760\pm 0.016$	$0.739\pm 0.012$	$0.746\pm 0.014$	$0.741\pm 0.011$
	k-means (Nanda, Mahanty, and Tiwari 2010)	$75.4\pm 3.31$	$0.73\pm 0.031$	$0.720\pm 0.033$	$0.731\pm 0.023$	$0.704\pm 0.016$
MSFT	k-medoids (Nakagawa, Imamura, and Yoshida 2019)	$73.5\pm 1.6$	$0.72\pm 0.012$	$0.719\pm 0.021$	$0.724\pm 0.018$	$0.679\pm 0.013$
	Fuzzy c means (Esfahanipour and Aghamiri 2010)	$71.0\pm 1.31$	$0.70\pm 0.013$	$0.672\pm 0.013$	$0.669\pm 0.018$	$0.661\pm 0.021$

Figures 2 and 3 depicts the box plots (Cox 1987) of percentage accuracy, $\mathit{NMI}$, $\mathit{ARI}$, $\mathit{JI}$ and $\mathit{FM}$ achieved by the different methods performed with 10 simulations on the two stock market datasets namely AAPL and MSFT respectively. From the Figures 2 and 3, it is observed that the median values obtained by DQC are significantly higher than the other methods considered. This infers that Dynamic Quantum Clustering approach has better performance than the other approaches in terms of different evaluation indices.

Figure 2. Box plots of (a) %Accuracy, (b) $\mathit{NMI}$, (c) $\mathit{ARI}$, (d) $\mathit{JI}$ and (e) $\mathit{FM}$ index obtained using different methods, namely, DQC, DBSCAN, EM and WAMSC performed on AAPL dataset.

Figure 3. Box plots of (a) %Accuracy, (b) $\mathit{NMI}$, (c) $\mathit{ARI}$, (d) $\mathit{JI}$ and (e) $\mathit{FM}$ index obtained using different methods, namely, DQC, DBSCAN, EM and WAMSC performed on MSFT dataset.

Statistical Significance Test

From Table 3, it is seen that the performance of dynamic quantum approach on the stock market data analysis is better than that of other density-based approaches in case of Big data analysis like stock market data. The results are statistically verified by performing paired t-test (Cox 1987) on DQC versus the other counterpart methods for the two stocks AAPL and MSFT at 5% level of significance. The results of the paired t-tests in terms of p-score for AAPL and MSFT are given in Tables 4 and 5 accordingly. Results are treated as noteworthy when the p-values are less than or equal to .05 (at 5% significance level) which are represented by bold font. In a summary, out of total 30 paired tests as shown in Tables 4 and 5, in 27 cases DQC achieved statistically significant improvements compared to its other counterpart techniques. Although in remaining 3 cases (shown in normal font in Tables 4 and 5), DQC achieved better results compared to other methods but that better results are not statistically significant.

Table 4. Paired t-test results in terms of p score values performed by the best performing method DQC versus other methods for AAPL dataset.

t-test done on	DQC vs DBSACN	DQC vs EM	DQC vs WAMSC
Accuracy	$\mathbf{1}\mathbf{.2012X1}{\mathbf{0}}^{-\mathbf{7}}$	$\mathbf{3}\mathbf{.4306X1}{\mathbf{0}}^{-\mathbf{10}}$	$\mathbf{2}\mathbf{.5588X1}{\mathbf{0}}^{-\mathbf{5}}$
NMI	$\mathbf{1}\mathbf{.3000X1}{\mathbf{0}}^{-\mathbf{11}}$	$\mathbf{5}\mathbf{.2604X1}{\mathbf{0}}^{-\mathbf{10}}$	NaN
ARI	$\mathbf{4}\mathbf{.3538X1}{\mathbf{0}}^{-\mathbf{10}}$	$\mathbf{1}\mathbf{.0316X1}{\mathbf{0}}^{-\mathbf{11}}$	$\mathbf{2}\mathbf{.9610X1}{\mathbf{0}}^{-\mathbf{7}}$
Jaccard Index	$\mathbf{4}\mathbf{.9251X1}{\mathbf{0}}^{-\mathbf{12}}$	$\mathbf{1}\mathbf{.0720X1}{\mathbf{0}}^{-\mathbf{10}}$	$\mathbf{1}\mathbf{.4267X1}{\mathbf{0}}^{-\mathbf{8}}$
FM Index	$\mathbf{3}\mathbf{.2775X1}{\mathbf{0}}^{-\mathbf{10}}$	$\mathbf{1}\mathbf{.1499X1}{\mathbf{0}}^{-\mathbf{9}}$	$\mathbf{0}\mathbf{.0020}$

Table 5. Paired t-test results in terms of p score values performed by the best performing method DQC versus other methods for MSFT dataset.

t-test done on	DQC vs DBSACN	DQC vs EM	DQC vs WAMSC
Accuracy	$\mathbf{2}\mathbf{.8707X1}{\mathbf{0}}^{-\mathbf{10}}$	$\mathbf{6}\mathbf{.0453X1}{\mathbf{0}}^{-\mathbf{9}}$	$\mathbf{8}\mathbf{.4978X1}{\mathbf{0}}^{-\mathbf{5}}$
NMI	$\mathbf{3}\mathbf{.9693X1}{\mathbf{0}}^{-\mathbf{8}}$	$\mathbf{1}\mathbf{.6490X1}{\mathbf{0}}^{-\mathbf{7}}$	$0.3183$
ARI	$\mathbf{9}\mathbf{.7944X1}{\mathbf{0}}^{-\mathbf{8}}$	$\mathbf{4}\mathbf{.0152X1}{\mathbf{0}}^{-\mathbf{9}}$	$\mathbf{0}\mathbf{.0058}$
Jaccard Index	$\mathbf{1}\mathbf{.1625X1}{\mathbf{0}}^{-\mathbf{7}}$	$\mathbf{8}\mathbf{.1733X1}{\mathbf{0}}^{-\mathbf{9}}$	$0.1336$
FM Index	$\mathbf{1}\mathbf{.7520X1}{\mathbf{0}}^{-\mathbf{9}}$	$\mathbf{3}\mathbf{.0455X1}{\mathbf{0}}^{-\mathbf{8}}$	$\mathbf{0}\mathbf{.0020}$

Implication of Results

While building portfolios in stock market, diversification is a major step as investing in all the stocks may not be possible due to financial constraints. Also not all stocks are worth investing in and thus an efficient plan is required to choose only the stocks that behave differently. This strategy helps in minimizing risks by selecting stocks from different sectors but in a more data-driven way. So, clustering algorithms applied here identifies the different clusters of stocks from which best performing stocks can be chosen.

In this work, the stocks has been divided into distinct groups based upon said returns and volatilities. Dividing stocks into groups with similar characteristics can help in portfolio construction to ensure that the chosen stocks have sufficient diversification between them. Such diversification prevents the portfolio suffering greatly from downturn of a specific type of companies during a period. Also, volatility clustering has critical implications in financial risk management (Kim and Song 2020).

Effectiveness of the Density-Based Clustering Approaches

Now to justify the performance of the various density-based clustering approaches for identifying stock trends, we applied the concept of moving averages (Chang et al. 2018) to the stocks AAPL and MSFT, obtained from the same cluster. Simple Moving Average (SMA) refers to a stock’s average closing price over a specified period (Fifield, Power, and Knipe 2008).

SMA can be calculated by taking the arithmetic mean of a given set of values over a specified period (Metghalchi, Marcucci, and Chang 2012). A set of numbers, or prices of stocks, are added together and then divided by the number of prices in the set. The formula for calculating the simple moving average of a stock is as follows:

(24) $\mathit{SMA}=(A_1+A_2+..+A_n)/\mathit{n}$(24)

where A is average of the stock prices in period n and n is the number of time periods. Moving averages are calculated to identify the trend direction of a stock.

To visualize the clustering results, stocks with the highest volatility values are plotted in Figure 4 based on the historical returns in vertical axis with respect to the dates (horizontal axis). The historic return is basically the difference between the current price of a stock and the previous day’s closing price (Fan and Yao 2017). Similarly, the historical returns of the stocks from the others clusters can also be visualized.

Figure 4. Historical correlation between the stocks AAPL and MSFT from the same cluster.

Qualitative Analysis of the Results

The results from the experimental evaluation implies that density-based clustering approaches, particularly, Dynamic Quantum clustering, are quite effective in creating portfolios for the stock market data. From the portfolios thus created, stocks showing different behavior can be selected for planning trading strategies. Results from statistical significance test confirm the validity of the Dynamic Quantum clustering approach. This better models the time series stock market data which results in better stock prediction accuracy and thus provides the useful profitable trades prediction tool.

Conclusion

Stock market prediction is an interesting topic which uses the methodologies related to extraction of knowledge from big data. With this objective in mind, the present empirical study tries to analyze the applicability of different density-based clustering techniques for the stock market data analysis. Under this category, Dynamic Quantum clustering (DQC), Density-based spatial clustering of applications with noise (DBSCAN), Expectation Maximization (EM) and Weighted Adaptive Mean Shift Clustering (WAMSC) were used. The stock market data used was the S&P 500 trading data for a time span of two years. Also for comparison purposes, partitive clustering methods, namely, k-means, k-medoids and fuzzy c means was implemented on the same stock market data.

The effectiveness of the techniques were validated on two stocks, namely AAPL and MSFT, in terms of different validity measures namely, NMI, ARI, Jaccard index and FM index. From the performance measures, DQC is found to be superior than the other density-based approaches as well as the partitive-based approaches. The statistical significance of the better achievement accomplished by DQC method in contrast to the other methods is confirmed from the paired t-test results for most of the evaluation indices.

The box plots of the results for the two stocks on AAPL and MSFT also suggest better median value of the evaluation measure produced by DQC method in comparison to other counter part methods. Thus indicating the superiority of DQC in stock market data analysis and prediction than its competitive techniques.

These stocks when introduced to the concept of moving average acts as a technical indicator of price trends and helps in the forecasting on the next-day stock limit predictions. The forecasting ability of the DQC turns out to be much better than its other competitors. Thus it can be inferred with sufficient proof that DQC is one of the best tool for the stock market data analysis. Since stock market data is of the category of big data, it can be safely inferred that DQC is a suitable tool for other variety of big data analysis also.

As the clustering approaches considered here are not limited to spherical shapes but based on density, so they can handle arbitrary shaped clusters (both concave and convex shaped clusters) and in turn handle noisy data quite effectively. However, as the data size increases, it takes a toll in the running time of the algorithms. So, in the future, we consider to increase the efficiency of the algorithms by decreasing the processing time of the data. Also trend detection with the help of moving averages works fairly well during smooth conditions. But the drawback is that if the prices of the stocks becomes uneven, multiple trade signals are generated which creates confusion. These issues can be addressed in the future work.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Publicly available stock market datasets are used in this article and therefore data are already publicly available.