Combined visual and spatial-temporal information for appearance change person re-identification

Abstract

Person re-identification (ReID) seeks to identify the same individual across different cameras by matching their corresponding images. The current ReID datasets are limited in size and diversity, especially in terms of clothing changes, making traditional techniques vulnerable to appearance variations. Further, current approaches rely heavily on appearance features for discrimination, which is unreliable when a person’s appearance changes. We hypothesize that the ReID accuracy can be enhanced by training the ReID model on a large volume of diversified training data and combining multiple features for discrimination. We use the image channel shuffling data augmentation method, producing a large volume of diversified training data. Also, a two-stream visual and spatial-temporal method is proposed to learn the feasible features for appearance change scenarios. The appearance features obtained from the visual stream are combined with spatio-temporal information to discriminate between two people. The proposed approach is evaluated for its robustness on short-term and on long-term datasets. The presented two-stream approach outperforms earlier methods by achieving Rank1 accuracy of 98.6% on Market1501, 95.52% on DukeMTMC-reID, 76.21% on LTCC, and 91.76% on VC-Clothes, respectively.

Keywords:

• Person re-identification
• video surveillance
• person retrieval
• deep learning
• appearance features
• data augmentation

1. Introduction

Person Re-identification (ReID) is the task of associating the same person’s image captured from a different camera or in the same camera at a different time. Deep learning has revolutionized the field of person re-identification by enabling the development of highly accurate and robust models (Bilakeri & Karunakar, 2023; Bilakeri & Kotegar, 2022; Movassagh et al., 2021) for this task. The ReID task is a key element of surveillance systems and a variety of other intelligent applications, e.g., forensic video investigation, crime investigation, and person tracking (Bilakeri & K, 2022). The most challenging part of ReID is appearance variation, which refers to the possibility that two people could appear to be the same person or that the same person could appear differently. The appearance variation occurs when a person arrives in a surveillance network after a short time gap (a few hours) or a long time gap (a few days, weeks, or years). The causes for the short-term appearance variation are illumination change, pose and viewpoint variation, background change, etc., and in the long-term appearance, the cloth change is the common cause. Each person’s images in Market1501 (L. Zheng et al., 2015) and DukeMTMC-reID (Z. Zheng et al., 2017) datasets are collected during a short period (on the same day). Therefore, these are considered short-term datasets. In contrast, the LTCC (Qian et al., 2020) and VC-cloth (Wan et al., 2020) are long-term datasets because these datasets are collected over a considerably longer period of time. Because of these short-term and long-term appearance variations, the same person appears to be different, or two people appear to be the same. Consistent re-identification under such variations remained unsolved.

There has been tremendous growth in the ReID performance with the advent of deep learning for ReID tasks, which has dominated the other techniques (e.g. handcrafted feature engineering) in extracting discriminative features. Recently, by utilising deep learning for feature extraction, cutting-edge person ReID algorithms (G. Wang et al., 2016; Zhong, Zheng, Cao, et al., 2017; Zhuo et al., 2018) significantly improved the results. However, the performance is inefficient when tested on the same person with different appearance variations or when two persons are in the same cloth pattern. The first reason that affected the performance is that most of the benchmark datasets Market1501, DukeMTMC-reID have a large number of classes with limited samples per class and a lack of diversity. The existing ReID methods addressed this issue using different data augmentation methods such as random cropping (Krizhevsky et al., 2012), random flipping (Simonyan & Zisserman, 2014), random erasing (Zhong, Zheng, Kang, et al., 2017), etc. However, these methods will enhance the sample size without encoding class-sensitive information during transformation. Recently, Generative Adversarial Network (GAN) has been widely used for augmentation purposes. However, the training is computationally expensive and takes a lot of time to converge the training model. The second reason for ReID’s performance drop in the appearance change scenario is the lack of an efficient feature learning approach.

Recent studies employed structural features such as body size, the shape of a person, horizontal/vertical person’s body part features, etc. These structural features helped to exploit complete body features and part level features for the ReID task. Various attention and part-based methods have also been proposed to achieve efficient performance. However, most of the attention mechanisms were based on a pose estimation model, and detecting key points in the data considered for the study is time-consuming. Instead of appearance features, various methods such as video-based or image-to-video ReID methods (G. Wang et al., 2017; Li et al., 2018) targets to acquire visual representations that are invariant to spatial and temporal factors. Nevertheless, these methods solely prioritize the visual attributes; most prior arts have ignored spatial-temporal constraints between multiple cameras. For instance, an individual observed by Camera 1 during a specific time instance t should not be visible on Camera 2 located at a distance from Camera 1 during a different time interval t + $\mathrm{\Delta }$t. A large portion of the irrelevant target images is removed from the gallery by such a spatial-temporal constraint, significantly reducing the appearance ambiguity issue.

In light of the above discussion, we present a novel approach that uses image channel shuffling to generate images with significant appearance variations. Furthermore, we propose a two-stream approach for discriminative feature learning. The two-stream approach considers both visual and spatial-temporal information. The visual stream extracts content and edge features, and the Spatio-temporal stream uses effortless information such as the time-stamp of each frame. The visual stream features have more discriminative capability than the spatial-temporal information. Therefore, when both are combined, the visual stream features contribute more towards improving the recognition rate, and adding spatial-temporal information improves the overall performance. The major contributions of our study are as follows:

1. We show that the image channel shuffling data augmentation increases the sample size and introduces diversity in terms of the appearance change of a person.

2. We validate that the inclusion of image channel shuffling has been helpful in improving the model generalization capability.

3. We show that the integration of visual features (edge, content) and spatio-temporal information makes the proposed method robust for the appearance change scenario.

4. The experimental results obtained on short-term and long-term datasets show that our method outperforms prior arts.

5. We have presented experimental investigation of cross-domain evaluation that signifies the proposed method is more generalized to cross domain re-identification.

The remainder of the paper is structured as follows. Section 2 reviews related works. Section 3 presents the proposed combined visual and spatio-temporal approach to generate more discriminative features. Experimental results and discussions are given in Section 4. Finally, conclusions are drawn in Section 5.

2. Related works

The current person ReID techniques focus on deep learning for appearance representation. Given the vast variety of studies on person ReID, we review the methods most closely related to our work. The existing works are reviewed as short-term and long-term person re-identification in our study.

2.1. Short-term person re-identification

The challenge of ReID task is drawing increasing attention as surveillance systems become more common in the real world. As previously stated, practically all of the ReID datasets that are currently available (L. Zheng et al., 2015; Z. Zheng et al., 2017) were collected over a short period. Therefore, the clothes’ appearances are more or less consistent for the same person. The data collected in a short time exhibits an appearance change caused due to illumination, pose, viewpoint, and occlusion. In the era of deep learning, considerable work has gone into creating methods for automatically re-identifying people using robust distance measurements or discriminative features. However, the models trained on the current dataset (L. Zheng et al., 2015; Z. Zheng et al., 2017) are not robust when a person comes with a changed cloth. Therefore, there is a need to supervise the model on large and diversified datasets. Aiming at this, Yu et al (Z. Yu et al., 2021). proposed a semi-supervised clothing-invariant feature learning framework for discriminative embedding learning with a GAN that simulates clothing. To make the ReID model invariant to external factors,(Deng et al., 2018). suggested the use of a similarity-preserving GAN to address domain discrepancies through image style transfer. The work presented by Wei et al (Wei et al., 2018). transformed the person from a source domain to a target domain to handle appearance changes caused by environmental effects. Recently, several deep learning models for person Re-identification (ReID) with pose guidance have been introduced, aimed at accounting for changes in the appearance of individuals. For example, Qian et al (Qian et al., 2018). developed an image generator capable of producing images of the same person in various poses. At the same time, Lv et al (Lv et al., 2018). proposed a method for transferring view-invariant representations from source data to target data using asymmetric multi-task dictionary learning. However, these works consider GAN for image construction to have an appearance change effect. The time complexity of the GAN is considerably high, which limits its application in a real-world scenario. Some of the works have aimed to propose appearance-invariant models by considering discriminative features. For example, Bhujel et al (Bhujel et al., 2020). and Sun et al (W. Sun et al., 2021). focused on local and global body features and part features such as horizontal and vertical. Also, the attention mechanism has profound potential in the field of person ReID. Inspired by the attention mechanism, various methods are presented (Fan et al., 2020; M. Zheng et al., 2019; Si et al., 2019; Xu et al., 2021). Xu et al (Xu et al., 2021). introduced a dual attention network that involves pose-guided attention and an activation-based attention model to solve the ReID problem. Zang et al (Zhang et al., 2021). presented a part-guided attention network that exploits the human body’s key points. Based on the key points, horizontal parts are generated. Then, inter and intra-local parts relations are exploited for the ReID task. However, part-based and attention-based approaches (Xu et al., 2021; Zhang et al., 2021) highly depend on the pose estimation model, which is an extra burden to estimate ground truth key points. Another research team focused on spatial-temporal data in order to create the appearance invariant ReID model. In the context of person ReID from images to videos, Wang et al (G. Wang et al., 2017). devised a point-to-set technique. For the purpose of identifying several differentiating parts of a person in a video, Li et al (Li et al., 2018). created a spatial-temporal attention model. Wang et al (G. Wang et al., 2019). proposed two-stream networks which involve visual features from horizontal body part features and spatial-temporal information from frame sequences. However, equal horizontal partition methods will not solve misalignment, thereby increasing misclassification. Also, the model is trained on minimal samples; therefore, this model shows inefficiency when a person’s appearance changes due to illumination, viewpoint, and on different background changes. The consideration of suitable data augmentation techniques that improves the sample size and adds the appearance variation effect to a dataset is essential. Also, more discriminative features such as content, edge, and spatio-temporal information combined with the right data augmentation would better solve the person’s re-identification in an appearance change scenario.

2.2. Long-term person re-identification

Due to its widespread applicability, the topic of changing clothes has received increased attention in recent years. In long-term person re-identification, a person changes his/her clothes very often. Recent person ReID methods focused on exploring appearance invariant features for discrimination. Yang et al. (Yang et al., 2019) present a technique to capture the contour sketch of the body’s features. This work primarily concentrates on learning invariant and discriminative ReID features by considering clothing information. Chen et al. (J. Chen et al., 2021) exploit the texture-insensitive 3D shape learning from 2D images to re-identify the person in cloth changing. Hong et al. (Hong et al., 2021) presented a two-stream network to extract body shape cues to handle cloth change challenges. However, it is observed that body shape cues are ineffective when two individuals’ body shapes are similar. Yu et al. (S. Yu et al., 2020) offer a two-branch network to retrieve biometric data and clothing properties through two submodels, respectively. They also introduce COCAS, a large-scale dataset focusing on pedestrian clothing modification status. Recently, Gao et al. (Z. Gao, H. Wei, W. Guan, J. Nie, et al., 2022) proposed human semantic attention and a visual cloth shielding model jointly supervised end-to-end to extract more discriminative and robust features irrelevant to appearance change. Gu et al. (Gu et al., 2022) presented a cloth adversarial loss function to extract irrelevant cloth representations by penalizing the predictive capability of the ReID model w.r.t. clothes. Goa et al. (Z. Gao, H. Wei, W. Guan, W. Nie, et al., 2022) proposed a multi-granular representation (MGR) technique, multilevel and multi-granular feature information is adaptively extracted, and a cloth desensitization network (CDN) is created to increase feature robustness for people wearing various outfits. However, performance improvement is still needed when someone arrives with changed clothes. The model performance can be improved by training it on a larger sample size, and a combination of multiple discriminative features would better solve the problem of long-term person re-identification.

In the proposed work, we bring to light a two-stream framework in which the combination of visual and spatiotemporal information is considered to make the model invariant to short-term and long-term appearance changes. Although conceptually related, our work differs by several aspects compared to (F. Chen et al., 2021; G. Wang et al., 2019): (i) We address ReID under both long-term and short-term appearance variation. In (F. Chen et al., 2021; G. Wang et al., 2019), only the short-term ReID is focused. (ii) We use the channel shuffling data augmentation method to increase the sample size and produce more diversified training samples on both long-term and short-term benchmarks. Whereas in (F. Chen et al., 2021), the channel shuffling is applied to a short-term dataset. (iii) In the proposed two-stream approach, the visual stream extracts content and edge features that are combined with the spatial-temporal information obtained from the spatial-temporal stream for discrimination. In (G. Wang et al., 2019), the two-stream framework extracts Spatial-temporal information and visual features from a part-based convolution model, which is inefficient for spatial misalignment. Our approach extracts multiple features (content, edge) from the visual stream and appearance invariant spatial-temporal information that is efficient and comprehensive for appearance change scenarios.

3. Combined visual and spatial-temporal information approach

The proposed ReID framework shown in Figure 1 is a two-branch network involving Visual Feature Stream (VFS) and Spatio-temporal Stream (STS). The two main components of VFS are data augmentation and feature disentangling networks. The data augmentation method takes RGB images as input and performs channel shuffling to generate the same image in six colors. This method increases sample size and diversity in terms of appearance change. Next, the randomly picked augmented image and the original image are passed to a feature-disentangling network, which contains a content and an edge encoder modules. The encoders mine the appearance and edge features from the input image. The two encoders are trained to minimize the reconstruction error between the original and reconstructed images. While also maximizing the distance between the identity-related features of different people using identification loss and contrastive loss. The training is not necessary for the spatio-temporal stream since the spatio-temporal information of each person in each camera is available from the dataset. During inference, pairs of an image are passed to VFS and STS. The VFS extracts and compares two images’ content and edge features to generate a similarity score. Parallelly, STS gives the probability distribution of two images being positive (images of the same person) between disjoint cameras. We use a joint metric that considers the class probability of two input images from the visual stream and the probability of a positive image pair from the spatial-temporal stream to classify whether both images are of the same person. In the subsequent subsections, there is a comprehensive description of every module is provided.

Figure 1. The Conceptual framework of the proposed method.

3.1. Visual feature stream

3.1.1. Data augmentation and soft label assignment

Increasing the training sample size by including additional images that contain appearance variations is essential to improve the generalization ability of the ReID method. With this aim, we utilize an image channel shuffling (F. Chen et al., 2021) data augmentation technique. In channel shuffling data augmentation, an input image is split into its channels (i.e., R G B), after which the order of the channels is changed (i.e., G R B), and finally, the channels are combined to create a new image. As shown in Figure 2, we generate five images for each input image, each with a distinct channel order (i.e., RBG, GRB, GBR, BRG, BGR). After data augmentation, the generated images are labeled using the soft label assignment. We take channel order and the connections between the set of generated images into account for soft label assignment. Consider P denotes the number of original identities. The group of generated images will have wider appearance variation; hence, we define the label vector of 6P bits. Let p correspond to identities 1, 2, … 6P of labeled data, and y is the position of 1, same as in the one-hot encoding. The ground truth label is defined based on equation 1.

Figure 2. Samples generated using channel’s order shuffling approach.

(1) $\mathit{Id}(\mathit{p})=\left\{\begin{array}{cc}0,& \mathit{p}\ne \mathit{y}\\ 1,& \mathit{p}=\mathit{y}\end{array}\right.$(1)

If the class identification number (p) and the position (y) in the one hot encoding are the same, that returns 1 (correct ground truth); otherwise, 0. In the soft label method, the label Id(.) is determined by three components formulated as in equation 2.

(2) $\mathit{Id}(\mathit{P})=\mathit{I}{d}_m(\mathit{P})+\mathit{I}{d}_c(\mathit{P})+\mathit{I}{d}_e(\mathit{P}).$(2)

Where $\mathit{I}{d}_m(\mathit{P})$, $\mathit{I}{d}_c(\mathit{P})$, $\mathit{I}{d}_e(\mathit{P})$ are the main component, content information, and edge information, the main component gives the training sample’s identity information. Content information represents the association between the images in the same channel sequence. Edge information measures how similar the generated image resembles the original. The soft label distribution is the probability distribution and is defined as follows in equation 3.

(3) $E_m+E_c+E_e=1$(3)

Where $E_m$, $E_c$, and $E_e$ are the weights assigned to the main, content, and edge components, respectively. The $\mathit{I}{d}_m(.)$ is defined using equation 4.

(4) $\mathit{I}{d}_m(\mathit{P})=\left\{\begin{array}{cc}0,& \mathit{p}\ne \mathit{y}\\ E_m,& \mathit{p}=\mathit{y}\end{array}\right.$(4)

$\mathit{I}{d}_c(.)$ focuses on channel order consistency. The content weight $E_c$ represents the relation of content information among images of the same channel sequence. For every image, P identities will have identical channel sequences. Since there are 6P identities, $\mathit{Ec}$ in the soft label is evenly split by P bits. It is defined by:

(5) $\mathit{I}{d}_c(\mathit{P})=\left\{\begin{array}{cc}{E}_c/\mathit{P},& \mathrm{p},\mathrm{y}\mathrm{have}\mathrm{the}\mathrm{same}\mathrm{channel}\mathrm{order}.\\ 0,& \mathrm{otherwise}.\end{array}\right.$(5)

$\mathit{I}{d}_e(.)$ denotes the edge resemblance among the set of generated images. The edge weight represents the correlation between the original and generated image. The generated six images of each identity will have a similar edge. Therefore, $E_e$ is equally split by 6 bits in the soft label encoding, which is defined by equation 8.

(6) $\mathit{I}{d}_e(\mathit{P})=\left\{\begin{array}{cc}{E}_e/6,& \mathrm{p}\mathrm{and}\mathrm{y}\mathrm{are}\mathrm{from}\mathrm{same}\mathrm{group}\\ 0,& \mathrm{o}\mathit{therwise}\end{array}\right.$(6)

The soft label assignment is flexible for extracting the desired information, such as content and edge. The reason for the soft label being flexible is content and edge information can be extracted by setting $E_c$ and $E_e$ to a larger value. The considered data augmentation relies solely on internally available information and can extract distinguishing features from image samples using ID labels. The original image and augmented and soft-labeled images are passed to the feature disentangling method to extract content and edge features.

3.1.2. Feature disentangling

The feature disentangling module takes two input images: the original image ($O_I$) and a randomly chosen generated image ($G_I$). A pair of input images are processed using two encoders, namely $E_c$ and $E_e$, which are structured similarly but have different parameters. This results in the extraction of content ($O_c$, $G_c$) and edge feature maps ($O_e$, $G_e$) from the images. The encoders are built on DenseNet121 (G. Huang et al., 2017) backbone network. Followed by the shared decoders reconstruct the images according to the feed from the encoder’s output feature maps. The model is supervised on reconstruction, contrastive, and identification loss functions for better and fast convergence.

The decoder network reconstructs the original and generated images by concatenating their respective extracted features (content and edge) to ensure that the generated features capture representative information. Then, pixels wise self re-constructive L2 loss is computed to check the similarity between the original ($O_I$, $O_G$) and reconstructed ($R_{O_I}$, $R_{O_G}$) images. Equation 7 is for self re-constructive loss ($L_{\mathit{rec}-\mathit{self}}$) computation.

(7) $L_{\mathit{rec}-\mathit{self}}=\sum _{\mathit{x}\in (\mathit{O},\mathit{G})}{∥I_x-\mathit{D}(E_c(I_x),E_e(I_x))∥}_2$(7)

where x is either the original or generated image, $\mathit{D}(E_c(I_x),E_e(I_x))$ denotes the reconstructed images ($R_{\mathit{oi}}$) by combining the content and edge information of an original image($O_c$, $O_e$), same definition follows to reconstruct the generated image. We hypothesize that the generated and the original images will share a common edge and different content information. To prove the hypothesis, the original and generated images’ content information is swapped, and these feature maps are represented as ($O_c$, $G_e$). That means the content features of the original image are used with the edge features of generated one to reconstruct a new image, which is represented as R($G_e$, $O_c$). The function ($G_c$, $O_e$) defines the content features of generated image and is used with the original image’s edge features to reconstruct a new image named R($O_e$,$G_c$). The reconstructed images (R($G_e$, $O_c$), R($O_e$,$G_c$)) are expected to be the same as the original images ($O_I$, $O_G$). The re-constructive swap loss ($L_{\mathit{rec}-\mathit{swap}}$) is computed using equation 8.

(8) $L_{\mathit{rec}-\mathit{swap}}=\sum _{(\mathit{x},\mathit{y})\in (\mathit{O},\mathit{G})\mathrm{\&x}\ne \mathit{y}}{∥I_x-\mathit{D}(E_c(I_x),E_e(I_y))∥}_2$(8)

Where, $\mathit{D}(E_c(I_x),E_e(I_y))$ denotes the reconstructed image by concatenating the content of the original image ($I_x$) with the edge of the generated image ($I_y$). The re-constructive loss ($L_{\mathit{rec}}$) is the sum of self and swap re-constructive losses, as defined in equation 9.

(9) $L_{\mathit{rec}}=L_{\mathit{rec}-\mathit{self}}+L_{\mathit{rec}-\mathit{swap}}$(9)

Also, the ReID model is trained using contrastive loss ($L_{\mathit{con}}$) for faster convergence. The contrastive loss is computed separately for content ($L_{\mathit{con}-\mathit{c}}$) and edge features ($L_{\mathit{con}-\mathit{e}}$) as defined in equation 10 and equation 11. The total contrastive loss is computed using equation 12.

(10) $L_{\mathit{con}-\mathit{c}}=\mathit{max}(\mathit{margi}{n}_c-{∥f_o^c-f_g^c∥}_2)$(10)

(11) $L_{\mathit{con}-\mathit{e}}=L_{\mathit{con}-\mathit{c}}={∥f_o^c-f_g^c∥}_2)$(11)

Where $f_o^c$ and $f_g^c$ are feature vectors of original, and augmented images, respectively and $\mathit{margi}{n}_c$ is the hyper-parameter representing the least gap between $f_o^c$ and $f_g^c$.

(12) $L_{\mathit{con}}={\alpha }_c{L}_{\mathit{con}-\mathit{c}}+{\alpha }_e{L}_{\mathit{con}-\mathit{e}}$(12)

where ${\alpha }_c$ and ${\alpha }_e$ are the trade-off parameters. To boost the identification ability of the learned feature, the ID loss is computed on the extracted content and edge features of the original and the generated images. The ID loss ($L_{\mathit{id}}$) is defined as per equation 13.

(13) $L_{\mathit{id}}=\sum _{\mathit{x}\in (\mathit{O},\mathit{G})}{L}_{\mathit{id}-\mathit{c}-\mathit{x}}+\sum _{\mathit{x}\in (\mathit{O},\mathit{G})}{L}_{\mathit{id}-\mathit{e}-\mathit{x}}$(13)

Where, $L_{\mathit{id}-\mathit{c}-\mathit{x}}$ and $L_{\mathit{id}-\mathit{e}-\mathit{x}}$ are the cross-entropy loss between the predicted and the ground truth probability.

Overall, the visual feature stream is trained using re-constructive, contrastive, and ID loss. As in equation 14, the final loss ($L_{\mathit{total}}$) is the weighted sum of all three defined loss functions where $\mathit{\beta }$ and $\mathit{\gamma }$ are weights assigned to contrastive and ID loss to manage different tasks.

(14) $L_{\mathit{total}}=L_{\mathit{con}}+\mathit{\beta }{L}_{\mathit{rec}}+\mathit{\gamma }{L}_{\mathit{id}}$(14)

During the testing phase, the 512-dimensional features related to content and edges are integrated to make a final person representation in the form of a 1024-dimensional vector. The correlation between the gallery and query image is computed using the Euclidean distance metric. To boost the discriminative power of our proposed method, we also estimate the spatio-temporal probability for the same two images. The computation of spatio-temporal probability is described in the following subsection.

3.2. Spatial-Temporal stream

A spatial-temporal stream is employed in our work to obtain spatial-temporal harmonizing data to aid the visual feature stream. Spatial-temporal distribution is computed using the non-parametric parzen window approach (Parzen, 1962) (Histogram parzen window method). This stream does not require training; we only train a visual stream. Considering $I_i$, $I_j$ are two images of $\mathit{I}{D}_i$, $\mathit{I}{D}_j$, captured by $C_i$, $C_j$ at time $t_i$, $t_j$, in order. We generate rough spatial-temporal frequency distribution to represent the likelihood of a positive image pair by equation 15.

(15) $\hat{P}(\mathit{Y}=1|\mathit{k},C_i,C_j)=\frac{n_{c_i{c}_j}^{\mathit{k}}}{{\sum }_l{n}_{c_i{c}_j}^{\mathit{l}}}$(15)

Where k is the $k^{\mathit{th}}$ bin of the histogram whose time interval is $t_i$-$t_j$ $\in$ ((k-1)$\mathrm{\Delta }$t, k$\mathrm{\Delta }$t). Where $n_{c_i{c}_j}$is the count of a pair of a person whose time gaps are at $k^{\mathit{th}}$ bin from $c_i$, $c_j$ and $\mathrm{\Delta }$t is a small value. The term Y = 1 denotes $I_i$, $I_j$ are of the same person image. We smooth the histogram using the Parzen Window approach defined by equation 16.

(16) $\hat{P}(\mathit{Y}=1|\mathit{k},C_i,C_j)=\frac{1}{Z}\sum _l\hat{P}(\mathit{Y}=1|\mathit{l},C_i,C_j)\mathit{K}(\mathit{l}-\mathit{k})$(16)

Where k(.) is kernel and normalized vector is $\mathit{Z}={\sum }_k\mathit{P}(\mathit{Y}=1|\mathit{k},C_i,C_j)$. In this work, the Gaussian function is used as a kernel K which is defined by:

(17) $\mathit{K}(\mathit{x})=\frac{1}{\sqrt{2\mathit{\pi \sigma }}}{e}_{2{\sigma }^2}^{-x^2}$(17)

where, x denotes the data sample fall in bin k, $\mathit{\sigma }$ is standard deviation, and ${\sigma }^2$ is the variance.

The spatio-temporal stream evaluates the likelihood of two images pertaining to the same individual. The probability estimates from VFS and STS are considered in the joint metric phase for classification.

3.3. Joint metric

The spatial-temporal distribution is independent of visual similarity score due to the two diverse patterns. The joint probability is defined as follows:

(18) $\mathit{P}(\mathit{y}=1|x_i,x_j,\mathit{k},c_i,c_j)=\mathit{s}(x_i,x_j)\mathit{P}(\mathit{y}=1|\mathit{k},c_i,c_j)$(18)

where $\mathit{s}(x_i,x_j)$ represents visual similarity of two person $(x_i,x_j)$ and $\mathit{P}(\mathit{y}=1|\mathit{k},c_i,c_j)$ is spatial-temporal probability. Equation 18 neglects two things: (i) consideration of visual similarity score itself as visual similarity probability and (ii) neglecting the unreliable and uncontrollable spatial-temporal probability (people may appear from anywhere and at any time). Directly employing $\mathit{P}(\mathit{y}=1|\mathit{k},c_i,c_j)$ leads to lower recall. For example, given a query image, the gallery image has a 0.9 visual similarity score and 0.01 spatial-temporal distribution, and another gallery image has a 0.3 visual similarity score and 0.1 spatial-temporal distribution. As per equation 18, the second gallery image is retrieved. The image with less spatial-temporal probability is treated as an irrelevant image which is not acceptable in real-time applications. The reason is the thief will have a fast walking speed compared to a normal person. It is quite obvious to get the low spatial-temporal probability for the thief, thereby failing to retrieve the thief. To overcome this complexity, the visual similarity score is transformed into visual probability. First, Laplace smoothing is employed to adjust the probability of rare events (zero frequency problem), which alleviates the unreliable probability estimation. Secondly, we apply the logistic function for binary classification as defined in equation 19.

(19) $\mathit{f}(\mathit{x},\mathit{\lambda },\mathit{\gamma })=\frac{1}{1+\mathit{\lambda }{\mathit{e}}^{-\mathit{\gamma x}}}$(19)

where $\mathit{\lambda }$, $\mathit{\gamma }$ are smoothing and shrinking factors, respectively. Using both the Laplace smoothing and the logistic functions together is known as the logistic smoothing strategy. Equation 18 is modified as follows

(20) $P_{\mathit{joint}}=\mathit{f}(\mathit{s},{\lambda }_0,{\gamma }_0),\mathit{f}(P_{\mathit{st}},{\lambda }_1,{\gamma }_1)$(20)

Where $p_{\mathit{joint}}$ is $\mathit{P}(\mathit{Y}=1|(X_i,X_j,\mathit{K},C_i,C_j))$, s is defined as $\mathit{s}((X_i,X_j)\in (-1,1)$, and $P_{\mathit{st}}$ is defined as $\mathit{P}(\mathit{Y}=1|(\mathit{K},C_i,C_j)\in (0,1)$. The $\mathit{\lambda }$ and $\mathit{\gamma }$ are the smoothing and shrinking factor, respectively. Employing the logistic function to convert the similarity score into a binary classification probability is straightforward and self-evident.

4. Results and discussions

4.1. Implementation details

DenseNet121 is used as encoder (content and edge encoder) $E_c$, $E_e$, which has a larger receptive field than VGG, ResNet50, that helps to extract better global features of a pedestrian (F. Chen et al., 2021). As the max pooling layer extracts more relevant features than the average pooling layer, the global average pooling layer is swapped out with an adaptive max-pooling layer, and two fully connected layers (fc1, fc2) are added. The fc1 is to map a feature map to a 512-dimensional feature vector, and fc2 is to generate class probability. The output neuron is set to six times the training IDs. The decoder structure is mentioned in the Table 1.

Table 1. Decoder structure

Layer name	Parameter size	Input size	Output size
ConvTranspose0	[3 × 3, 512, 2, 1]	2048 × 8 × 4	512 × 16 × 8
ConvTranspose1	[3 × 3, 128, 2, 1]	512 × 16 × 8	128 × 32 × 16
ConvTranspose2	[3 × 3, 64, 2, 1]	128 × 32 × 16	64 × 64 × 32
ConvTranspose3	[3 × 3, 32, 2, 1]	64 × 64 × 32	32 × 128 × 64
ConvTranspose4	[3 × 3, 3, 2, 1]	32 × 128 × 64	3 × 256 × 128

This work follows the empirically tested hyperparameter values for the loss function and the soft label computation from (F. Chen et al., 2021). All the images are re-sized to $256\times 128$. The SGD optimizer trains the network with 24 as batch size and 0.5 dropout rate. The 0.01 learning rate is set for the initial layers. Then, every 35 epochs, the learning rate is decayed by 0.1. The model is trained for 90 epochs. For the spatial-temporal stream, $\mathrm{\Delta }$t is set to 100, and for the Gaussian kernel, the $\mathit{\sigma }$ value is defined as 50. In joint metric, ${\lambda }_1$, ${\lambda }_2$, ${\gamma }_1$, and ${\gamma }_2$ are set to 1, 2, 5, and 5, respectively. The spatial-temporal stream does not require training since the spatial-temporal information is generated from the image annotation.

The model is tested on public benchmark datasets such as Market1501, DukeMTMc-reID, LTCC, and VC-cloth. The dataset’s details are summarized in Table 2.

Table 2. Datasets detail

Datasets	Market1501	DukeMTMC- reID	VC-cloths	LTCC
Training images	12,936	16,522	9449	9,576
Training IDs	751	702	256	77
Testing images	19,732	19,989	9611	7,050
Testing IDs	750	702	256	75
#Views	6	8	4	12

4.2. Comparison with the prior arts

Our method is evaluated on Market1501 and DukeMTMC-reID datasets to analyse the performance in short-term settings. The performance attained by our approach on short-term datasets is compared with several cutting-edge techniques as listed in Table 3. Also, we verify the performance on the long-term datasets (LTCC and VC-clothes), and the outcomes are compiled in Table 4. It has been discovered that the performance is superior to existing approaches in both short and long-term scenarios.

Table 3. The performance comparison with prior-arts on market1501 and DukeMTMC-reID. The best results are highlighted in bold

Methods	Market1501		DukeMTMC-reID
	Rank1(%)	mAP(%)	Rank1(%)	mAP(%)
PGFA (Che et al., 2022)	94.6	81.4	85.7	72.7
Per-former (Pervaiz et al., 2022)	89.4	74.5	81.1	61.6
DAReID (Xu et al., 2021)	94.6	87.0	88.9	78.4
MRNet (X. Wang et al., 2020)	95.67	88.5	89.32	78.99
PGCN (Zhang et al., 2021)	98.01	94.8	91.1	85.2
PUCNN (W. Sun et al., 2021)	81.23	70.73	70.34	51.23
PB-GCN (Y. Wang et al., 2021)	94.1	83.8	84.7	71.9
FDReID (F. Chen et al., 2021)	94.72	86.74	89.05	78.02
ST-ReID (G. Wang et al., 2019)	98.1	87.6	94.5	83.9
Pos-Neg (X. Jia et al., 2022)	96.2	90.5	92.3	83.3
CAL (Gu et al., 2022)	94.7	87.5	79.7	57.3
proposed	98.6	93.03	95.52	86.59
Proposed(re-rank)	98.68	96.83	93.47	93.8

Table 4. The performance comparison with prior-arts on LTCC and VC-clothes. The obtained results are for cloth change settings

Methods	LTCC		VC-Clothes
	Rank1(%)	mAP(%)	Rank1(%)	mAP(%)
GI-ReID (Jin et al., 2021)	73.59	36.07	64.5	57.8
FSAM (Hong et al., 2021)	38.5	16.2	78.6	78.9
ASSP (J. Chen et al., 2021)	31.3	14.8	79.9	81.2
PCB (Y. Sun et al., 2018)	23.5	10	62.0	62.2
MVSE (Gao et al., 2021)	70.5	33.0	-	-
Part-aligned (Suh et al., 2018)	-	-	69.4	67.3
3APF (Wan et al., 2020)	-	-	90.2	82.1
Proposed	76.21	59.63	91.76	84.61

The PGCN (Zhang et al., 2021) and DAReID (Xu et al., 2021) methods require a pose estimation model to obtain ground truth joint points, which is an extra burden. Then, the PGCN model focused more on de-noising images than extracting robust features for the ReID task; also, most images are discarded based on uncertainty criteria. In our work, the data augmentation increased the dataset size and introduced more appearance change diversity. The content and edge feature and the complementary spatial-temporal information effectively handled the ReID task with an appearance change scenario. The efficient visual stream and complimentary spatial-temporal streams together enhanced the ReID performance compared to the existing techniques. The best results are highlighted in bold; also, the result without and with re-ranking is represented.

In order to assess the robustness of the proposed technique for appearance change scenarios, it is evaluated on the LTCC and VC-Clothes datasets. Our strategy outperforms cutting-edge approaches, as shown by the findings given in Table 4.

We visualized the top 10 retrieved results on Market1501 and LTCC datasets. Figure 3(a,c) shows retrieved results considering only the visual features stream, and Figure 3(b,d) shows the retrieved results considering both visual stream and spatial-temporal features (combined). We can see that the combined features are more favorable in retrieving the correct gallery image than considering visual features alone. Figure 4(a) is the top 10 retrieved images considering the visual feature stream alone, and Figure 4(b) is the retrieved results considering both the visual stream and spatial-temporal stream on the LTCC dataset. The retrieved images are not resized, directly retrieved, and illustrated from the gallery set. Therefore, we could see a varied image size in Figure 4. The proposed model performs more efficiently both on long-term and short-term datasets. Figure 3. Market1501 dataset : Top 10 retrieved gallery images for the given query image (Figure 4). LTCC dataset: Top 10 retrieved gallery images for the given query image.

Figure 3. Market1501 dataset : Top 10 retrieved gallery images for the given query image.

Figure 4. LTCC dataset : Top 10 retrieved gallery images for the given query image.

The limitation of the proposed method is that to integrate the visual and spatio-temporal information; the input data need to be provided with a time-stamp of a person in each camera view.

4.3. Ablation study and discussions

To check the effectiveness of data augmentation (DA), feature disentangling (FD), and spatial-temporal (ST) modules, we analyzed them by systematic experiment under the same dataset settings. The ablation study is conducted on Market1501 and DukeMTMC-reID datasets.

Baseline

The baseline is the simple DenseNet121 network with a fully connected layer to perform classification. The number of neurons is set to the number of identities in the considered dataset. The model is trained with identification loss.

Baseline+DA

The channel shuffling data augmentation is applied to all the images in the training set. The generated images are assigned new IDs using the soft-label assignment approach. The number of neurons in the fully connected layers is set to six times the original identities. The model is trained with identification loss.

Baseline+DA+FD

The feature disentangles method is employed to extract the content and edge features by setting different parameters such as $E_m$ = 0.7, $E_c$ = 0.2, and $E_e$ = 0.1. The model is trained with ID loss, contrastive loss, and reconstruction loss. The number of neurons in the fully connected layer is six times that of the original IDs. The model developed with Baseline+DA+FD represents the visual stream.

Baseline+DA+FD+ST

To complement the visual stream, spatial-temporal information is used. The joint metric method is applied to evaluate the performance of the model.

The ablation study results are represented in Table 5. The addition of each component one by one shows performance improvement. Additionally, the interaction of these aspects is advantageous for improving overall performance.

Table 5. Ablation study conducted with combination of different feature

Methods	Datasets
	Market1501		DukeMTMC-reID
	mAP(%)	Rank1(%)	mAP(%)	Rank1(%)
Baseline	76.89	92.01	66.07	82.91
Baseline+DA	82.00	93.72	73.95	86.18
Baseline+DA+FD	86.71	94.29	77.84	88.62
Baseline+DA+FD+ST	93.03	98.63	86.59	95.52

4.3.1. Effect of visual stream, spatial-temporal stream, and joint metric

The performance is analyzed considering visual and spatial-temporal streams separately, then the combination of both features is utilized in the joint metric.

Effectiveness of visual stream

To check the benefit of the visual stream, we discarded the visual stream and considered only spatial-temporal information for re-identification. It is observed that the very large marginal performance dropped from 98.63% to 12.01% and 95.52% to 8.13% Rank1 accuracy on Market1501 and DukeMTMC-reID datasets, respectively. The result is listed in Table 6. The visual features play a very important role in this approach.

Table 6. Analysis on the effectiveness of visual, spatial-temporal and joint-metric on the ReID performance

Steam	Market1501		DukeMTMC-reID
	mAP	Rank1	mAP	Rank1
Visual	86.71	94.29	77.84	88.62
Spatial-temporal	7.25	12.01	5.25	8.13
Joint Metric	93.03	98.63	86.59	95.52

Effectiveness of spatial-temporal stream

The benefit of having an STS is analyzed by detaching this module. The performance is assessed while simply taking into account the visual stream, and the findings are shown in Table 6. We observed the Rank1 accuracy drop from 98.63% to 94.29% on Market1501 and 95.52% to 88.62% Rank1 accuracy drop on DukeMTMC-reID datasets, respectively.

Effectiveness of Joint Metric

The joint metric uses visual and spatial-temporal probability to discriminate between two individuals. The combined features achieved an improved Rank1 recognition rate of 98.63% on Market1501 and 95.52% on DukeMTMC-reID datasets, respectively.

It is observed from Table 6 that the visual stream plays a vital role, and the spatial-temporal stream is complementary in enhancing the ReID performance further. The joint metric module helps in adjusting the probability of the rare event.

4.3.2. Cross-dataset ReID

There is a considerable domain gap between various ReID datasets, which results in a significant decline in performance when a model trained on one dataset is directly applied to another. To showcase the ability of our method to generalize across domains, we carry out a comparative analysis in a cross-domain environment. The cross-domain ReID is conducted by training the model on Market1501 and testing it on the DukeMTMC-reID dataset and visa-versa. The obtained results are summarized in Table 7. We have made two observations on the result; one is that all the models suffer from major performance degradation, and secondly, our approach is superior to the existing methods.

Table 7. The performance of the proposed model on cross domain is compared with state-of-the-art methods. M$\to$d indicates trained on market1501 and tested on DukeMTMC-reID, visa-versa

Method	M$\to$D		D$\to$M
	mAP	Rank1	mAP	Rank1
DAReID (Xu et al., 2021)	30.3	51.1	33.0	61.7
FDReID (F. Chen et al., 2021)	23.13	39.3	27.5	54.9
PCB (Y. Sun et al., 2018)	21.2	37.1	25.4	52.9
DG-Net (Z. Zheng et al., 2019)	24.2	42.62	26.83	56.12
EANet (H. Huang et al., 2018)	36.7	56.1	35.8	66.1
DFEF (Z. Jia et al., 2022)	34.4	56.9	36.3	68.3
Proposed	40.69	67.54	38.35	70.57

5. Conclusion

To address the problem of scarcity of training data and lack of diversity in it, in this paper, we have explored the image channel shuffling data augmentation approach that has increased the sample size and introduced the appearance variation effect into benchmark datasets. The model trained with this augmentation technique has shown improved generalization capability on the unseen domain. Further, combining multiple features is more robust than using the appearance feature alone. Aiming at this, we proposed a two-stream network that combines the visual (content, edge) and spatio-temporal features. The considered multiple features have shown their significance by consistently re-identifying a person under long-term and short-term appearance variations. The presented two-stream approach outperforms earlier methods by achieving Rank1 accuracy of 98.6% on Market1501, 95.52% on DukeMTMC-reID, 76.21% on LTCC, and 91.76% on VC-Clothes, respectively. Our method outperforms the prior arts and the results evidence that our method is robust in re-identifying a person in an appearance change scenario.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

No funding was received for conducting this study.