Spatio-temporal intention learning for recommendation of next point-of-interest

ABSTRACT

Next point-of-interest (POI) recommendation has been applied by many internet companies to enhance the user travel experience. Recent research advocates deep-learning methods to model long-term check-in sequences and mine mobility patterns of people to improve recommendation performance. Existing approaches model general user preferences based on historical check-ins and can be termed as preference pattern models. The preference pattern is different from the intention pattern, in that it does not emphasize the user mobility pattern of revisiting POIs, which is a common behavior and kind of intention for users. An effective module is needed to predict when and where users will repeat visits. In this paper, we propose a Spatio-Temporal Intention Learning Self-Attention Network (STILSAN) for next POI recommendation. STILSAN employs a preference-intention module to capture the user’s long-term preference and recognizes the user’s intention to revisit some specific POIs at a specific time. Meanwhile, we design a spatial encoder module as a pretrained model for learning POI spatial feature by simulating the spatial clustering phenomenon and the spatial proximity of the POIs. Experiments are conducted on two real-world check-in datasets. The experimental results demonstrate that all the proposed modules can effectively improve recommendation accuracy and STILSAN yields outstanding improvements over the state-of-the-art models.

KEYWORDS:

• Point-of-Interest (POI)
• recommendation
• spatial pretrained model
• self-attention
• revisiting intention

1. Introduction

In the past few decades, we have seen the rapid development of location-based social networks (LSBNs). Users on LSBNs post their check-ins and share life experiences through mobile devices. A check-in record usually includes point-of-interest (POI) visited in a specific spatio-temporal context, which reflect users’ mobility pattern. The next POI recommendation is one of the most important applications of LBSNs (Li, Westerholt, and Zipf 2018; Aslam et al. 2021). It recommends the next POI for users to visit based on historical check-in records.

Recent research advocates deep learning methods to model long-term check-in sequences and mine mobility patterns of people to improve recommendation performance. POI recommendation is different from the general recommendation task in that it typically considers the spatial, temporal, preference, and intention influence. Existing approaches model general user preferences based on historical check-ins and can be termed as preference pattern models. For example, the self-attention mechanism (Vaswani et al. 2017) has been used to model user historical behavior sequences (Kang and McAuley 2018). The model calculates attention weights between the next item and other items in the historical sequence, and thus, long-term dependencies can be captured effectively. As a result, existing research shows that self-attention can achieve good performance on the next POI recommendation (Guo and Qi 2020; He, Qi, and Ramamohanarao 2020; Huang et al. 2020; Lian et al. 2020; Luo, Liu, and Liu 2021).

However, the preference pattern is different from the intention pattern, in that it does not emphasize the user mobility pattern of revisiting POIs, which is a common behavior and kind of intention for users. RepeatNet (Ren et al. 2019) incorporates the repeat-explore mechanism into the gated recurrent units (GRUs) for session-based recommendation. The purpose of the work is to remind users of items that they have seen or consumed. Similarly, revisiting the same place is a common behavior of users. Existing approaches seem to ignore the learning of such a mobility intention pattern. Compared with other recommendation fields, such as product recommendation, repeated behaviors are more obvious in POI recommendation scenarios and have spatio-temporal characteristics. We take the visited POI which was revisited by the users as intentional behavior. For example, users arrive at work or home during the morning and evening rush hour. Meanwhile, exploring new POIs that may be of interest is regarded as a preference behavior. For instance, the users are willing to go to some new gourmet restaurants during the weekend. Therefore, when making a POI recommendation at the next moment, deep learning not only learns user preferences for POIs but also needs to consider that users with travel intentions may go to places visited before.

Although long-term dependencies have been captured by the self-attention mechanism, the spatial influence could be improved in the modeling. For example, Tobler’s first law of geography (Tobler 1970) points out that nearby things are more closely related. Existing research has demonstrated that spatial proximity affects check-in behavior. Users tend to visit places nearby. Therefore, it is necessary to encode geo-coordinates of POI based on distance. However, embedding learning for each POI position will greatly increase the cost of model training. How to balance training cost with effective modeling of POI spatial features is a major challenge.

To address the aforementioned issues, we propose the Spatio-Temporal Intention Learning Self-Attention Network (STILSAN) with spatial clustering-based embedding for next POI recommendation, which yields outstanding improvements over the state-of-the-art models. STILSAN includes the following: The first phase is to capture user revisiting intention. A spatial-temporal revisiting mechanism is used to learn user revisiting intention that can calculate the recommendation probability of revisiting POIs in the intention pattern. The second phase discusses the improvement of POI recommendation performance by spatial factor. We first construct a set of virtual centroids by clustering POI locations. Each POI is associated with a virtual centroid. Then, we pretrain a spatial feature model for virtual centroids via variational graph auto-encoders (VGAE) (Kipf and Welling 2016), whereby spatial embeddings are learning for each POI.

The contributions of this paper can be summarized as follows:

• We propose the STILSAN for the next POI recommendation. To the best of our knowledge, STILSAN is the first model that explicitly models the user mobility pattern of intention.

• A spatial feature pretrained model based on VGAE is proposed to learn POI spatial embedding. This model simulates the spatial clustering phenomenon and preserves the spatial proximity of the POIs.

• An optimal embedding combination and form are explored in the embedding module. This can provide heuristics for self-attention-based models in next POI recommendation task.

• Tested on two real POI recommendation datasets, the STILSAN model achieves the SOTA performance compared to other state-of-the-art methods.

2. Related work

In this section, we summarize related work from two aspects, which are next POI recommendation and self-attention for recommendation.

2.1. Next POI recommendation

Early next POI recommendation models mainly include Markov chains (Rendle 2010; Rendle, Freudenthaler, and Schmidt 2010) and its extended models (Cheng et al. 2013; Feng et al. 2015), but Markov chains cannot capture contextual information well. Later on, the Recurrent Neural Network (RNN) has become the mainstream method for modeling the POI check-in sequences (Sun et al. 2020; Zhao et al. 2020a). ST-RNN (Liu et al. 2016) applies an extended RNN to the location prediction problem. ST-LSTM (Huang et al. 2021) proposes an attention-based LSTM network to model sequential patterns of check-in behavior. Although RNN effectively improves the recommendation performance, it still suffers from the long-term dependency problem (Chung et al. 2014). Compared to other scenarios, POI recommendation usually utilizes temporal and spatial relations to assist in modeling user dynamic preferences. Nitu, Coelho, and Madiraju (2021) take into account a user’s most recent interest by incorporating time-sensitive recency weight into the model. Liu et al. (2016) model spatio-temporal contexts to improve model performance using time-specific transition matrices and distance-specific transition matrices. Time-LSTM (Zhu et al. 2017), ST-LSTM (Zhao et al. 2018), STGN (Zhao et al. 2020b), and RTPM (Liu et al. 2022) adopt similar ideas, which add time gate and distance gate to the structure of LSTM to better capture user preferences in different spatio-temporal contexts. HGMAP (Zhong et al. 2020), GARG (Wu et al. 2020), GGLR (Chang et al. 2020), and DGCN (Wang et al. 2022b) take a similar approach, which independently learns geographic influences by graph neural network or representation learning. Others, for example, Yu et al. (2020) consider the POI categorical influence. Yang, Liu, and Zhao (2022) develop a novel time-aware category context embedding to capture the diverse temporal patterns of POI categories. The aforementioned models all demonstrate that the combination of various influences can effectively improve the recommendation performance.

2.2. Self-attention for recommendation

In recent years, self-attention has been introduced into the field of sequential recommendation and achieved state-of-the-art results (Kang and McAuley 2018; Huang et al. 2020; Li, Westerholt, and Zipf 2018). Self-attention has been shown to be effective in next POI recommendation. Most of the studies integrate the spatio-temporal patterns of user check-ins into the self-attention mechanism to improve POI recommendation. Considering the spatio-temporal effect of non-adjacent locations and nonconsecutive check-ins, Luo, Liu, and Liu (2021) propose a spatio-temporal self-attention-based model called STAN. Lian et al. (2020) design a geography-aware POI sequential recommendation model based on self-attention so that some spatial phenomena can be effectively captured. Wang et al. (2022a) measure the spatial-temporal interval as users’ acceptance to distance and time with self-attention. He, Qi, and Ramamohanarao (2020) focus on the impact of temporal contexts and proposed a time-aware self-attention network for next POI recommendation. Compared to RNN, self-attention usually has better performance because it calculates the correlation of each check-in with other check-ins in the sequence. Therefore, the self-attention mechanism has obvious advantages in learning user long-term preferences

Our approach to next POI recommendation differs from the works listed earlier. STILSAN considers not only the spatio-temporal pattern but also the user behavior pattern of revisiting POIs. At the same time, STILSAN adopts advanced spatial feature pretrained model to achieve further improvements in recommendation accuracy.

3. Problem formulation

Let $\mathit{U}=\left\{u_1,u_2,\dots ,u_{\left|\mathit{U}\right|}\right\}$ and $\mathit{L}=\left\{l_1,l_2,\dots ,l_{\left|\mathit{L}\right|}\right\}$ denote a set of users and a set of POIs. Here $\left|\mathit{U}\right|$ and $\left|\mathit{L}\right|$ are the number of users and POIs. Given the user $\mathit{u}\in \mathit{U}$, the tuple of its historical check-in sequence is denoted as $S^u=\left\{⟨l_1,c_1,t_1⟩,⟨l_2,c_2,t_2⟩,\dots ,⟨l_n,c_n,t_n⟩\right\}$, where $c_i=\left(\mathit{lo}{n}_i,\mathit{la}{t}_i\right)$ is the geo-coordinates to which POI belongs, $t_i$ is the timestamp and $\mathit{i}$ is the position of this check-in in the sequence. The next POI recommendation task is formulated as calculating the recommendation score of the $\left(\mathit{n}+1\right)$-th POI, which is defined as $\mathit{P}\left(l_{\mathit{n}+1}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}{S}^u\right)$. By introducing the spatial-temporal revisiting mechanism which assumes that the user behavior pattern is controlled by preference pattern and revisiting intention pattern, the conditional probability is factorized as

(1) $\mathit{P}\left(l_{\mathit{n}+1}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}{S}^u\right)=\mathit{P}\left(\mathit{p}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}{S}^u\right)\mathit{P}\left(l_{\mathit{n}+1}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}\mathit{p},S^u\right)+\mathit{P}\left(\mathit{r}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}{S}^u\right)\mathit{P}\left(l_{\mathit{n}+1}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}\mathit{r},\mathit{\hspace{0.17em}}{S}^u\right)$(1)

where $\mathit{p}$ and $\mathit{r}$ denote check-in preference pattern and revisiting intention pattern, respectively. $\mathit{P}\left(\mathit{p}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}{S}^u\right)$ and $\mathit{P}\left(\mathit{r}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}{S}^u\right)$ represent probabilities of spatial-temporal transition. $\mathit{P}\left(l_{\mathit{n}+1}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}\mathit{p},S^u\right)$ and $\mathit{P}\left(l_{\mathit{n}+1}\mathit{\hspace{0.17em}}|\mathit{\hspace{0.17em}}\mathit{r},\mathit{\hspace{0.17em}}{S}^u\right)$ represent the recommendation probabilities of the $\left(\mathit{n}+1\right)$-th POI in the preference pattern and the revisiting intention pattern.

4. Approach

In this section, we introduce our model in detail. Figure 1 shows the overall architecture of STILSAN, including the embedding module, preference-intention module, and spatial encoder module. First, in the embedding module, we concatenate POI ID embedding, user embedding, and spatial embedding by the noninvasive method as the input and feed it into the preference-intention module. We get the spatial embedding using a pretrained model in the spatial encoder module. Then, we calculate the recommendation probabilities for all POIs in the preference-intention module. Finally, we recall the POIs from the candidate set that are most likely to be recommended. Each module will be elaborated in the following sections.

Figure 1. The architecture of our STILSAN model.

4.1. Embedding module

4.1.1. Noninvasive embedding

We transform the check-in sequences of all users to a fixed length $\mathit{m}$ sequence, where $\mathit{m}$ represents the maximum sequence length that the model can handle. If the number of check-ins in the original sequence $n^u$ is greater than $\mathit{m}$, we select $\mathit{m}$ check-ins in the most recent time. If not, we pad in zeros on the left side of the sequence until the sequence length is $\mathit{m}$. We randomly initialize the feature embedding of POI ID and user ID, defined as $e^l\in {{\mathbb{R}}^d}^{\mathit{d}}$ and $e^u\in {\mathbb{R}}^d$ where $\mathit{d}$ is the feature dimension.

To incorporate the representation of POI into the spatial feature, we combine the POI spatial representation $\mathit{Z}$ from the spatial encoder module. In existing work, the ways to combine multiple features mainly include feature concatenation. It may be inappropriate to input the combined results directly into the self-attention module. This behavior is considered as an invasive method, which may confuse the feature space of POI ID and reduce the recommendation performance. We interpret the reasons for confusing the feature space of POI ID as: (1) combining additional features into the feature of POI ID will overwhelm the POI ID information; and (2) feeding the combined features into self-attention will output a mixed feature space. This output is combined with the POI candidate set that represents the hypothesis space to compute the recommendation ranking. The feature space of the POI candidate set is consistent with POI ID but inconsistent with output by the invasive method. Therefore, it may be confusing and difficult to calculate the recommendation ranking.

To solve this problem, we design a noninvasive embedding module for self-attention-based models in next POI recommendation. The noninvasive embedding module largely keeps the feature space of output and the candidate set the same. Specifically, we redefine the query $\mathit{Q}$, key $\mathit{K}$, and value $\mathit{V}$ in self-attention. The embedding of $\mathit{Q}$ and $\mathit{K}$ is defined as $e^{\mathit{qk}}=\mathit{concat}\left(e^l,e^Z,e^u\right){\mathit{W}}^{\mathit{qk}}$, where $\mathit{concat}(\cdot )$ represents concatenation operation, $e^Z$ is the spatial embedding corresponding to POI $\mathit{l}$. The embedding of $\mathit{V}$ is defined as $e^V=e^l{\mathit{W}}^{\mathit{v}}$. ${\mathit{W}}^{\mathit{qk}}\in {\mathbb{R}}^{3\mathit{d}\times \mathit{d}}$, ${\mathit{W}}^{\mathrm{v}}\in {\mathbb{R}}^{\mathit{d}\times \mathit{d}}$ are learnable parameters. We show a comparison between the traditional self-attention structure in Figure 2(a) and the noninvasive self-attention structure in Figure 2(b).

Figure 2. Comparison of self-attention by invasive method and noninvasive method.

Some work considers temporal features in the embedding module. For example, Luo, Liu, and Liu (2021) divide the week into 7 × 24 = 168 h. Therefore, the index size of temporal embedding is 168. However, we drop the temporal feature, because we find that it will reduce the performance of the model in experiments.

4.1.2. Spatio-temporal intervals embedding

In this paper, we calculate the time intervals and distance intervals between check-ins in the sequence that are regarded as the displayed spatial-temporal relation. For $S^u$, ${\mathbf{\Delta }}_{\mathit{i}\mathit{j}}^{\mathit{s}}\in {\mathbb{R}}^{\mathit{n}\times \mathit{n}}$ and ${\mathbf{\Delta }}_{\mathit{i}\mathit{j}}^{\mathit{t}}\in {\mathbb{R}}^{\mathit{n}\times \mathit{n}}$ represent the distance interval and time interval between $\mathit{i}$-th and $\mathit{j}$-th check-ins, respectively. Then, we directly adopt a method that can learn spatio-temporal transition matrices based on a linear interpolation method proposed by STAN (Luo, Liu, and Liu 2021). For $S^u$, spatial and temporal intervals embeddings ${\mathit{T}}_{{\mathbf{\Delta }}^{\mathit{t}}},{\mathit{S}}_{{\mathbf{\Delta }}^{\mathit{s}}}\in {\mathbb{R}}^{\mathit{n}\times \mathit{n}}$ can be calculated as:

(2) $\left\{\begin{array}{c}{\mathit{T}}_{{\mathbf{\Delta }}^{\mathit{t}}}=\frac{{\mathit{T}}_{\mathit{L}\left({\mathbf{\Delta }}^{\mathit{t}}\right)}\left(\mathit{U}\left({\mathbf{\Delta }}^{\mathit{t}}\right)-{\mathbf{\Delta }}^{\mathit{t}}\right)+T_{\mathit{U}\left({\mathbf{\Delta }}^{\mathit{t}}\right)}\left({\mathbf{\Delta }}^{\mathit{t}}-\mathit{L}\left({\mathbf{\Delta }}^{\mathit{t}}\right)\right)}{\mathit{U}\left({\mathbf{\Delta }}^{\mathit{t}}\right)-\mathit{L}\left({\mathbf{\Delta }}^{\mathit{t}}\right)}\\ {\mathit{S}}_{{\mathbf{\Delta }}^{\mathit{s}}}=\frac{{\mathit{S}}_{\mathit{L}\left({\mathbf{\Delta }}^{\mathit{s}}\right)}\left(\mathit{U}\left({\mathbf{\Delta }}^{\mathit{s}}\right)-{\mathbf{\Delta }}^{\mathit{s}}\right)+S_{\mathit{U}\left({\mathbf{\Delta }}^{\mathit{s}}\right)}\left({\mathbf{\Delta }}^{\mathit{s}}-\mathit{L}\left({\mathbf{\Delta }}^{\mathit{s}}\right)\right)}{\mathit{U}\left({\mathbf{\Delta }}^{\mathit{s}}\right)-\mathit{L}\left({\mathbf{\Delta }}^{\mathit{s}}\right)}\\ \mathrm{\Delta }={\mathit{T}}_{{\mathbf{\Delta }}^{\mathit{t}}}+{\mathit{S}}_{{\mathbf{\Delta }}^{\mathit{s}}}\end{array}\right.$(2)

where $\mathbf{\Delta }$ is the spatio-temporal intervals embedding, $\mathit{U}\left(\cdot \right)$ and $\mathit{L}\left(\cdot \right)$ denote the upper bound and lower bound of ${\mathbf{\Delta }}^{\mathit{t},\mathit{s}}$, $T_{\mathit{L}\left(\cdot \right)}$ and $T_{\mathit{U}\left(\cdot \right)}$ are the learnable parameters, which denote the unit embeddings of the upper bound and lower bound.

4.2. 4.2. Preference-intention module

We integrate the spatial-temporal revisiting mechanism into the self-attention model. Specifically, we subdivide user mobility behaviors into check-in preference pattern and revisiting intention pattern. The preference pattern models the user preferences based on historical check-ins. The revisiting intention pattern enables the model to recognize the user's intention to revisit some specific POIs. We design a preference encoder and an intention encoder to calculate the recommendation probabilities for POIs in the two patterns, respectively. At the same time, the spatio-temporal transition probabilities are proposed as a soft switch to decide which pattern the next check-in is. Then, we can get the final recommendation scores by combining the spatio-temporal transition probabilities and the recommendation probabilities. Finally, we rank the recommendation scores. The POIs with higher scores in the candidate set are more likely to be accessed by users in the next moment.

4.2.1. 4.2.1 Preference pattern encoder

We design a preference pattern encoder based on self-attention to model the check-in sequences. The input is the latent representation of the check-in sequences through the embedding module, which is defined as $\mathit{E}=\left\{e_1,e_2,\dots ,e_m\right\}\in {\mathbb{R}}^{\mathit{m}\times \mathit{d}}$. Here, $\mathit{e}$ is a tuple $⟨e^{\mathit{qk}},e^v⟩$. Then, $\mathit{E}$ is linearly transformed into $\mathit{E}{\mathit{W}}^{\mathit{Q}}$, $\mathit{E}{\mathit{W}}^{\mathit{K}}$, $\mathit{E}{\mathit{W}}^{\mathit{V}}$. Three of them represent query, key, and value, respectively. ${\mathit{W}}^{\mathit{Q}}$, ${\mathit{W}}^{\mathit{K}}$, ${\mathit{W}}^{\mathit{V}}\in {\mathbb{R}}^{\mathit{d}\times \mathit{d}}$ are learnable parameters. Based on the self-attention mechanism, the new representation of check-in sequences denoted as ${\mathit{E}}^{\prime }=\left\{{e_1}^{\prime },{e_2}^{\prime },\dots ,{e_m}^{\prime }\right\}\in {\mathbb{R}}^{\mathit{m}\times \mathit{d}}$ is calculated through $\mathit{SA}\left(\cdot \right)$ and $\mathit{FFN}\left(\cdot \right)$, which is defined as follows:

(3) $\mathit{O}=\mathit{SA}\left(\mathit{Q},\mathit{K},\mathit{V}\right)=\mathit{s}\mathrm{oftmax}\left(\frac{\mathit{Q}{\mathit{K}}^{\mathrm{T}}}{\sqrt{\mathit{d}}}\right)\mathit{V}$(3)

Then, the output $\mathit{O}$ is fed into a point-wise two-layer Feed-Forward Network (FFN), which is defined as follows:

(4) $\mathit{E}{ }^{\mathrm{\prime }}=\mathit{FFN}\left(\mathit{O}\right)=\mathrm{ReLU}\left(\left(\mathit{O}{\mathit{W}}^1+b^1\right){\mathit{W}}^2+b^2\right)$(4)

where ${\mathit{W}}^1$, ${\mathit{W}}^2\in {\mathbb{R}}^{\mathit{d}\times \mathit{d}}$ are the learnable parameters, and $b^1$, $b^2$ are the biases.

Next, we employ dot product to compute the recommendation probabilities of the $\mathit{i}$-th POI in the candidate set as follows:

(5) $\mathit{P}\left(l_{\mathit{n}+1}|\mathit{p},S^u\right)=\mathit{\hspace{0.17em}}\frac{exp\left({\mathrm{c}}_i^T{e^{\prime }}_{\mathit{m}}\right)}{{\sum }_{i=1}^{\left|L\right|}\mathit{exp}\left({\mathrm{c}}_i^T{e^{\prime }}_{\mathit{m}}\right)}\mathit{\hspace{0.17em}}$(5)

where $c_i$ is the embedding of the $\mathit{i}$-th POI ID in the candidate set.

It is worth noting that in preference pattern encoder we need two masks to shield the sequence positions that affect the attention calculation. The first mask is used to adapt to user check-in sequences of different lengths. Its size is $\mathit{m}-n^u$, which is the length of zero padding on the left side of the sequence. Another mask shields future check-in behaviors related to the predicted POI.

4.2.2. Revisiting intention pattern encoder

We adopt additive attention to calculate the recommendation probabilities of revisiting POIs in the intention pattern encoder. The probabilities corresponding to POIs that do not appear in historical check-ins are set to 0. Therefore, the recommendation probabilities of revisiting the $\left(\mathit{n}+1\right)$-th POI is defined as:

(6) $a_{\tau }^r={\mathit{V}}_r^{\mathrm{T}}\tanh \left({\mathit{W}}_1^r{e_m}^{\prime }+{\mathit{W}}_2^r{e_{\tau }}^{\prime }+\mathbf{\Delta }\right)$(6)

(7) $\mathit{P}\left(l_{\mathit{n}+1}|\mathit{r},S^u\right)=\left\{\begin{array}{c}\mathit{\lambda }\frac{{\sum }_{{\mathit{l}}_{\mathit{n}+1}}\mathit{exp}\left(a_{\tau }^r\right)}{{\sum }_{\tau =1}^m\mathit{exp}\left(a_{\tau }^r\right)}{l}_{\mathit{n}+1}\in S^u\\ 0\mathit{other}\end{array}\right.$(7)

where ${\mathit{V}}_{\mathrm{r}}$, ${\mathit{W}}_{\mathbf{1}}^{\mathit{r}}$, ${\mathit{W}}_{\mathbf{2}}^{\mathit{r}}$ are the learnable parameters, ${e_{\tau }}^{\prime }$ ($\mathit{\tau }\in \left[1,\mathit{m}\right]$) is the $\mathit{\tau }$-th embedding in the $\mathit{E}\prime$. $\mathit{\lambda }$ is a penalty coefficient to prevent overfitting. $\sum _{{\mathit{l}}_{\mathit{n}+1}}\mathit{exp}\left(a_{\tau }^r\right)$ represents the sum of the recommendation probabilities of POI $l_{\mathit{n}+1}$, because the same revisited POIs will appear multiple times in the historical sequence.

4.2.3. Recommendation with preference and revisiting intention

We consider probabilities of spatial-temporal transition, which are used to measure whether the next visit is more biased toward the preference pattern or the intention pattern. We compute probabilities of spatial-temporal transition through a spatial-temporal attention mechanism. First, we obtain the last embedding $e_m^{\prime }$ in the ${\mathit{E}}^{\prime }$ that is the output of the preference pattern encoder. Next, we compute the additive attention scores between the last embedding and the other embedding in the ${\mathit{E}}^{\prime }$ based on the additive attention mechanism. At the same time, we introduce the spatial-temporal relation matrix into the additive attention mechanism, which makes the spatial-temporal transition to fully consider the influence of different spatio-temporal contexts, as shown in Eq. 8(8) $P_p={\mathit{V}}^{\mathit{T}}\tanh \left({\mathit{W}}_{\mathbf{1}}^{\mathit{p}}{e_m}^{\prime }+{\mathit{W}}_{\mathbf{2}}^{\mathit{p}}{e_{\tau }}^{\prime }+\mathbf{\Delta }\right)$(8) . Finally, we transform the additive attention scores to probabilities of spatial-temporal transition by a linear variation and a softmax operation, as shown in Eq. 9(9) $\left[\mathit{P}\left(\mathit{p}|S^u\right),\mathit{P}\left(\mathit{r}|S^u\right)\right]=\mathrm{softmax}\left({\mathit{W}}_{\mathbf{3}}^{\mathit{p}}{P}_p\right)$(9) .

(8) $P_p={\mathit{V}}^{\mathit{T}}\tanh \left({\mathit{W}}_{\mathbf{1}}^{\mathit{p}}{e_m}^{\prime }+{\mathit{W}}_{\mathbf{2}}^{\mathit{p}}{e_{\tau }}^{\prime }+\mathbf{\Delta }\right)$(8)

(9) $\left[\mathit{P}\left(\mathit{p}|S^u\right),\mathit{P}\left(\mathit{r}|S^u\right)\right]=\mathrm{softmax}\left({\mathit{W}}_{\mathbf{3}}^{\mathit{p}}{P}_p\right)$(9)

where $\mathit{V}$, ${\mathit{W}}_{\mathbf{1}}^{\mathit{p}}$, ${\mathit{W}}_{\mathbf{2}}^{\mathit{p}}\in {\mathbb{R}}^{\mathit{d}\times \mathit{d}}$, ${\mathit{W}}_{\mathbf{3}}^{\mathit{p}}\in {\mathbb{R}}^{2\times \mathit{d}}$ are learnable parameters, $\mathbf{\Delta }$ is spatio-temporal interval embedding.

4.3. Spatial encoder module

The modeling of spatial dependencies is an important influencing factor for next POI recommendation. Learning the spatial embedding of each POI directly may lead to excessive model training cost. To this end, we first map POIs to virtual centroids through a spatial clustering algorithm and then learn the spatial embedding of each virtual centroid based on VGAE. We treat the spatial encoder module as a pretrained model for POI spatial embedding. This supports downstream recommendation tasks.

4.3.1. Virtual centroids clustering

We adopt the K-means clustering algorithm to aggregate the POIs according to their spatial distribution. Each POI is associated with a cluster. Then, we compute the mean of the geo-coordinates of all POIs within each cluster and define it as the virtual centroid $\mathit{V}=\left\{v_1,v_2,\dots ,v_{\left|\mathit{V}\right|}\right\}$, where $\left|\mathit{V}\right|$ is the prespecified optimal number of clusters. In particular, we try different clustering algorithms to achieve the same goal, including K-means, DBSCAN (Ester et al. 1996), and OPTICS (Ankerst et al. 1999). In contrast to K-means, which is a distance-based clustering algorithm and tends to generate spherical clusters, DBSCAN and OPTICS are density-based clustering algorithms that can find clusters of arbitrary shapes and identify noisy points. Although DBSCAN and OPTICS have the advantage of not needing to set the number of clusters, it is difficult to obtain satisfactory clustering results. One reason is that DBSCAN and OPTICS do not work well for clusters of different densities. The spatial distribution of POI locations is usually inhomogeneous. The other reason is that virtual centroids are more suitable to reflect the spatial feature of POIs within spherical clusters.

4.3.2. Spatial embedding learning

Spatial distance is the most important factor for modeling correlations between POIs. We propose a spatial embedding learning model using the VGAE. VGAE is a framework for unsupervised learning on graph-structured data based on a variational auto-encoder. Based on VGAE, the spatial encoder module trains a spatial latent embedding influenced by location distance. Specifically, we propose the spatial embedding learning model including three steps of graph construction, encoder by Graph Convolutional Network (GCN), and decoder by reconstruction.

First, we use the distance to construct the virtual centroids graph $\mathit{G}=\left(\mathit{V},\mathit{E}\right)$, where $\mathit{E}$ is the set of edges in the $\mathit{G}$. An edge $e_{\mathit{i},\mathit{j}}$ denotes the distance feature between virtual centroids $\mathit{i}$ and $\mathit{j}$. We calculate the distance feature between virtual centroids as follows:

(10) $e_{\mathit{i},\mathit{j}}=\frac{1}{\mathit{Distance}}$(10)

where $\mathit{Distance}$ stands for the distance between virtual centroids $\mathit{i}$ and $\mathit{j}$. The closer the distance, the greater the value of $e_{\mathit{i},\mathit{j}}$. Then, we construct the adjacency matrix $\mathit{A}\in {\mathbb{R}}^{\left|\mathit{V}\right|\times \left|\mathit{V}\right|}$ from the virtual centroids graph $\mathit{G}$.

Second, the encoder of VGAE is denoted by $\mathit{Z}=\mathit{f}\left(\mathit{X},\mathit{A}\right)$, where $\mathit{X}\in {\mathbb{R}}^{\left|\mathit{V}\right|\times \mathit{d}}$ is a randomly initial node embedding matrix, $\mathit{d}$ is the size of node embedding. $\tilde{\mathit{A}}={\mathit{D}}^{-\frac{1}{2}}\mathit{A}{\mathit{D}}^{-\frac{1}{2}}$ is the normalized adjacency matrix, where $\mathit{D}$ is the degree matrix of $\mathit{A}$. Here, $\mathit{Z}\in {\mathbb{R}}^{\left|\mathit{V}\right|\times \mathit{d}}$ is the output of the encoder representing the node embedding, $\mathit{d}$ is the size of the embedding. $\mathit{f}\left(\cdot \right)$ calculates the mean $\mathit{\mu }$ and variance $\mathit{\sigma }$ by two-layer GCN as follows:

(11) $\mathit{\mu }=\mathit{\hspace{0.17em}}\mathit{GC}{N}_{\mathit{\mu }}\mathit{(}\mathit{X},\tilde{\mathit{A}})$(11)

(12) $log\mathit{\sigma }=\mathit{\hspace{0.17em}}\mathit{GC}{N}_{\mathit{\sigma }}\mathit{(}\mathit{X},\tilde{\mathit{A}})$(12)

(13) $\mathit{GCN}\mathbf{(}\mathit{X},\tilde{\mathit{A}})=\tilde{\mathit{A}}\mathrm{ReLU}\left(\tilde{\mathit{A}}\mathit{X}{\mathit{W}}_{\mathbf{0}}\right){\mathit{W}}_1$(13)

where ${\mathit{W}}_{\mathbf{0}}$ is shared in $\mathit{GC}{N}_{\mathit{\mu }}$ and $\mathit{GC}{N}_{\mathit{\sigma }}$, but ${\mathit{W}}_{\mathbf{1}}$ is different.

Next, we sample the mean $\mathit{\mu }$ and variance $\mathit{\sigma }$ using reparameterization as follows:

(14) $\mathit{Z}=\mathit{\mu }+\text{isin}\mathit{\sigma }$(14)

where $\text{isin}$ is the noise variable that follows a normal distribution.

Third, the decoder of VGAE uses the output $\mathit{Z}$ of the encoder to reconstruct the virtual centroids graph through the inner product as follows:

(15) $\hat{\mathit{A}}=\mathit{\sigma }\left(\mathit{Z}{\mathit{Z}}^T\right)$(15)

where $\hat{\mathit{A}}$ is the reconstructed adjacency matrix, and $\mathit{\sigma }$ is the logistic sigmoid function.

To better understand the spatial pretrained model, the pseudocodes are shown in Pseudocode 1.

Pseudocode 1. The spatial pretrained model

Table

Input: $\mathit{G}=\left(\mathit{V},\mathit{E}\right)$, where $\mathit{V}$ is the set of nodes that represent the virtual centroids, $\mathit{E}$ is the set of edges that represent the distance feature between virtual centroids.

(1) $\mathit{X}$ $\leftarrow$ Randomly initial node embedding matrix

(2) $\mathit{A}$ $\leftarrow$ Construct the adjacency matrix of $\mathit{G}$, the elements of the adjacency matrix are computed by the reciprocal of the distance between the virtual centroids

(3) $\mathit{D}$ $\leftarrow$ Construct the degree matrix of $\mathit{A}$

(4) $\tilde{\mathit{A}}$ $\leftarrow$ Construct the normalized adjacency matrix of A by $\tilde{\mathit{A}}={\mathit{D}}^{-\frac{1}{2}}\mathit{A}{\mathit{D}}^{-\frac{1}{2}}$

Encoder by GCN

(5) $\mathit{\mu }$ $\leftarrow$ Generate the matrix of mean vectors by $\tilde{\mathit{A}}\mathrm{ReLU}\left(\tilde{\mathit{A}}\mathit{X}{\mathit{W}}_0\right){\mathit{W}}_1^{\mathit{\mu }}$

(6) $log\mathit{\sigma }$ $\leftarrow$ Generate the matrix of variance vectors by $\tilde{\mathit{A}}\mathrm{ReLU}\left(\tilde{\mathit{A}}\mathit{X}{\mathit{W}}_0\right){\mathit{W}}_{\mathbf{1}}^{\mathit{\sigma }}$

(7) $\mathit{Z}$ ← Sample the mean μ and variance σ by $\mathit{\mu }+\text{isin}\mathit{\sigma }$

Decoder by reconstruction

(8) $\hat{\mathit{A}}$ ← Reconstruct the $\mathit{G}$ by $\mathit{\sigma }\left(\mathit{Z}{\mathit{Z}}^T\right)$

Model Training

repeat

Execute the encoder by GCN

Execute the decoder by reconstruction

Calculate the reconstruction error $L_{\mathit{spatial}}$ according to Eq. 16(16) $L_{\mathit{spatial}}=L_{\mathit{bce}}+L_{\mathit{KL}}$(16) , Eq. 17(17) $L_{\mathit{bce}}={\mathbb{E}}_{\mathit{q}\left(\mathit{Z}|\mathit{X},\mathit{A}\right)}\left[log\mathit{p}\left(\mathit{A}|\mathit{Z}\right)\right]$(17) , and Eq. 18(18) $L_{\mathit{KL}}=\mathit{KL}\left[\mathit{q}\left(\mathit{Z}|\mathit{X},\mathit{A}\right)||\mathit{p}\left(\mathit{Z}\right)\right]$(18)

Update parameters using full-batch gradient descent

until convergence

Output: $\mathit{Z}$ as the POI spatial feature obtained from the pretrained model

4.4. Model training

4.4.1. Pretraining for spatial encoder

For VGAE, the purpose of model training is to hope that the reconstructed graph is as similar as possible to the original graph. The loss function is defined as follows:

(16) $L_{\mathit{spatial}}=L_{\mathit{bce}}+L_{\mathit{KL}}$(16)

where $L_{\mathit{bce}}$ is the binary cross-entropy loss function that is used to measure the difference between the reconstructed graph and the original graph as follows:

(17) $L_{\mathit{bce}}={\mathbb{E}}_{\mathit{q}\left(\mathit{Z}|\mathit{X},\mathit{A}\right)}\left[log\mathit{p}\left(\mathit{A}|\mathit{Z}\right)\right]$(17)

where $L_{\mathit{KL}}$ is the Kullback–Leibler (KL) divergence which measures the difference between the node embedding distribution $\mathit{q}\left(\mathit{Z}|\mathit{X},\mathit{A}\right)$ and the normal distribution $\mathit{p}\left(\mathit{Z}\right)$ as follows:

(18) $L_{\mathit{KL}}=\mathit{KL}\left[\mathit{q}\left(\mathit{Z}|\mathit{X},\mathit{A}\right)||\mathit{p}\left(\mathit{Z}\right)\right]$(18)

4.4.2. Training for STILSAN

STILSAN is trained cyclically, and its training times are the length of the training data $l_{\mathit{train}}$ at each step. Each cycle uses the first $\mathit{n}$ POIs to predict the $\left(\mathit{n}+1\right)$-th POI until $\mathit{n}+1=l_{\mathit{train}}$. Therefore, there is only one ground truth per cycle. We take the final recommendation scores $\left\{r_{\mathit{n}+1,1},r_{\mathit{n}+1,2},\dots ,r_{\mathit{n}+1,\left|\mathit{L}\right|}\right\}$ as output and a POI ground truth $l_q$ as expected output. For the positive sample $l_q$, we randomly sample $\mathit{j}$ POIs as negative samples that neither appear in the historical sequence. These POIs are defined as $\mathit{N}=\left\{N_1,N_2,\dots ,N_{\mathrm{j}}\right\}$. The cross-entropy loss function of POI is defined as:

(19) $-{\sum }_{S^u\in \mathit{S}}\left[log\left(\mathit{\sigma }\left(r_{\mathit{n}+1,\mathit{q}}\right)\right)+{\sum }_{N_i\in \mathit{N}}^{\mathit{j}}log\left(1-\mathit{\sigma }\left(r_{\mathit{n}+1,{\mathit{N}}_{\mathit{i}\mathit{\hspace{0.17em}}}}\right)\right)\right]$(19)

5. Experiments

5.1. Experimental setup

5.1.1. 5.1.1 Datasets

We use two real-world check-in datasets in New York City (NYC) and Tokyo (TKY) from 12 April 2012 to 16 February 2013 provided by Yang et al. (2015). We filter out POIs visited fewer than 10 times. We delete those users who check in with less than 10 POIs in sequence. Table 1 shows the number of users, POIs, and check-ins in each dataset after preprocessing. We take the first 80% check-ins as the training set, the latter 10% as the validation set, and the other real check-ins as the test set.

Table 1. Statistics of datasets.

Datasets	User	POI	Check-in
NYC	918	4259	132935
TKY	2156	6982	433145

5.1.2. Baselines

We select several representative models as the baselines from multiple subdivisions, including RNN-based model, self-attention-based model, category-aware model, and time-aware model. We summarize the baseline models and their characteristics as follows:

• STRNN (Liu et al. 2016) extends the RNN method which takes into account local temporal and spatial contexts.

• LSTPM Sun et al. 2020) is a long-term and short-term sequence model that uses a nonlocal network and geo-dilated RNN to model user long-term and short-term preferences.

• CatDM (Yu et al. 2020) incorporates POI category and geographical influence.

• SASRec (KangKang and McAuley 2016) uses the self-attention mechanism to model a user's historical behavior sequence that does not consider temporal and spatial factors.

• RepeatNet (Ren et al. 2019) incorporates the repeat-explore mechanism into the GRU. RepeatNet implements the general session-based recommendation task without considering the spatial-temporal features.

• STAN (Luo, Liu, and Liu 2021) is the first model to apply self-attention to POI recommendations, which considers spatio-temporal features.

5.1.3. Parameter and metrics

We set the maximum sequence length $\mathit{m}$ as 100, which represents the maximum sequence length that the model can handle. We observe that the median check-in number of users in the NYC and TKY datasets is 97 and 133, respectively. Therefore, it is reasonable to set the parameter $\mathit{m}$ to 100, which can balance the impact of retaining and padding operations in Section 4.1.1. There are two kinds of hyperparameters: (1) common hyperparameters that are shared by all models; (2) unique hyperparameters that depend on our proposed STILSAN. To other common hyperparameters for our model, we train these hyperparameters on a simple self-attention network and then apply them to our models, which helps reduce the training burden and find the optimal hyperparameter settings quickly. The learning rate, drop-out rate, penalty coefficient, the number of negative samples, and batch size are set to 0.001, 0.2, 0.02, 10, and 25. The POI, spatial, user, and other embedding dimensions are set to 40 and 25 in NYC and TKY, respectively. The unique hyperparameter in our model is the number of virtual centroids. We set the number of virtual centroids in TKY $\left|V_{\mathit{NYC}}\right|$ and NYC $\left|V_{\mathit{TKY}}\right|$ are set to 150 and 200, respectively. We discuss the set of hyperparameters $\left|V_{\mathit{NYC}}\right|$ and $\left|V_{\mathit{TKY}}\right|$ for different datasets and the corresponding influences in Section 5.3. We use an Adam optimizer to optimize our model. For other baseline models, we follow the optimal hyperparameter settings suggested by the authors. In the experiment, Recall@$\mathit{k}$ and NDCG@$\mathit{k}$ are used to evaluate the recommendation performance of the model. Recall@$\mathit{k}$ evaluates whether there are correct POIs in the top $\mathit{k}$ recommendation results. Normalized Discounted Cumulative Gain (NDCG) is a measure of ranking quality. We set $\mathit{k}$ = 5, and 20 in the experiment.

5.2. Overall performance

On the TKY and NYC datasets, our proposed model outperforms all baseline models on recall, as shown in Table 2. Compared with the optimal baseline, STILSAN has increased by 5.5% and 8% on the two datasets in terms of Recall@5, respectively. This improvement is significant. First, based on the experimental results, we find that all self-attention-based models are better than non-self-attention models. The reason is that self-attention can better model long-term dependence than RNN. LSTPM is better than STRNN because LSTPM considers the impact of user short-term and long-term preferences for the next POI. Meanwhile, we observe that STRNN performs worst. As an RNN-based POI recommendation model, the STRNN may lie in insufficient modeling of spatio-temporal information. Although CatDM additionally introduces POI category information, it is lower than STILSAN in all indicators. The possible reason is that the user revisiting behavior is more obvious during check-in activities. In addition, we also observe that a large number of check-ins in the two datasets are concentrated in specific POI categories, such as home and subway. This uneven data distribution makes it difficult to model category preferences. CopyNet achieves considerable performance because it models repeat behavior. However, due to the lack of spatio-temporal modeling and the insufficient ability of the GRU to capture long-term preferences, CopyNet does not achieve optimal performance. Moreover, we conclude from comparative experiments that STAN is better than the STRNN and LSTPM, but the recommendation performance is lower than the STILSAN. For example, in terms of Recall@$\mathit{k}$, STILSAN is almost 5.8%–8% higher than STAN on the NYC dataset and 4.5%–5.5% higher than STAN on TKY dataset. We attribute the improvement in recommendation performance to the embedding module, preference-intention module, and spatial encoder module that we designed.

Table 2. Comparison of STILSAN with baselines in Recall@$\mathit{k}$ and NDCG@$\mathit{k}$.

	TKY				NYC
	Recall@5	Recall@20	NDCG@5	NDCG@20	Recall@5	Recall@20	NDCG@5	NDCG@20
STRNN	0.1133	0.1848	0.0869	0.1142	0.1376	0.2162	0.1071	0.1314
LSTPM	0.219	0.3497	0.1775	0.2342	0.3188	0.4758	0.2538	0.2997
CatDM	0.1835	0.2691	0.1382	0.1615	0.2107	0.3083	0.1588	0.2077
SASRec	0.1976	0.3182	0.1475	0.1901	0.2996	0.5523	0.2218	0.3799
CopyNet	0.26557	0.3535	0.1958	0.2254	0.4053	0.4844	0.3018	0.3572
STAN	0.2637	0.385	0.1909	0.2320	0.4318	0.5786	0.3450	0.3971
STILSAN	0.319	0.4300	0.2487	0.2770	0.512	0.637	0.3898	0.4225

5.3. Stability study

For the K-means algorithm, we need to set the number of virtual centroids in advance. Therefore, it is necessary to study the effect of the number of virtual centroids on the model performance. We set the number of virtual centroids $\left|V_{\mathit{NYC}}\right|$ and $\left|V_{\mathit{TKY}}\right|$ to 50, 100, 150, 200, and 250 while keeping other optimal hyperparameters unchanged. Figure 3 presents specific changes in the model performance as the number of virtual centroids changes. First, it can be observed that the performance of the two different datasets is greatly different. This is due to the difference in the spatial distribution and density of POIs between the TKY and NYC datasets. The number of POIs in the TKY dataset is much more than that in the NYC dataset. While the spatial extent of the TKY dataset is smaller than that of the NYC dataset. It is not difficult to know that the POI density in the TKY dataset is greater than that in the NYC dataset. Second, the model shows low performance when $\left|V_{\mathit{TKY}}\right|$ and $\left|V_{\mathit{NYC}}\right|$ are small. This is because when the number of virtual centroids is small, POIs within a large area are aggregated into a cluster which makes the spatial feature between distant POIs not easy to be distinguished. Thirdly, we can observe that the performance of the model hits the peak when $\left|V_{\mathit{TKY}}\right|=150$ and $\left|V_{\mathit{NYC}}\right|=200$. And then, as the number of virtual centroids increases, the performance of our model begins to decline. This is because when the number of virtual centroids is large, the adjacent POIs are separated into different clusters, which makes the model underfit and yield suboptimal performance.

Figure 3. The impact of the number of clusters.

5.4. Stability study

5.4.1. Embedding combinations and forms

The embedding module is considered in almost recommendation models. However, few studies have specifically compared the model performance between different embedding combinations and forms. Specifically, embedding forms refer to invasive method and noninvasive method. To find the optimal embedding combination and form, we conduct a comparison experiment. We expect to provide more general heuristics to all other self-attention-based models for next POI recommendation. We define the Original Model (OM), which is without the preference-intention module. The OM only uses POI ID embedding in the embedding module. We consider several variants of embedding module based on the OM as shown in Table 3. For example, variant (VIII) concatenates spatial embedding and user embedding with POI ID embedding by noninvasive method. Table 4 shows the experimental results for all variants of the embedded module.

• Finding 1: Noninvasive method tends to perform better compared to invasive methods. For example, when comparing these two variants (I, II), the noninvasive method achieves up to 1.2%–2.8% and 2.4%–3.7% improvement over the invasive method, on TKY and NYC, respectively. Comparing variants (III, IV) and variants (V, VI), the noninvasive method also outperforms the invasive method. The reasons have been explained in Section 4.1.1.

• Finding 2: Adding user embedding, and time embedding by invasive methods will degrade model performance. We observe that directly fusing other embeddings into the POI ID embedding leads to performance degradation by comparing OM with variants (I, III). A similar conclusion has also been demonstrated in another paper (Lian et al. 2020). Lian et al. (2020) propose the GeoSAN model whose embedding module adopted invasive method. They also demonstrated that adding user embedding or time embedding did not lead to performance improvement. We observe that adding user embedding by invasive method has the greatest negative impact on performance, while spatial embedding by invasive method has a minimal negative impact. The possible reason is that the spatial embedding represents the inherent property of POI.

• Finding 3: Adding user embedding and spatial embedding by noninvasive method will get a significant performance boost. Although user embedding performs poorly on invasive method, it performs well on noninvasive method. Adding user embedding by noninvasive method not only avoids the mismatch of feature space but also realizes personalized modeling check-in sequences. We find that it produces a similar effect to the revisiting mechanism. Adding spatial embedding by noninvasive also has a good improvement. The reason might be that POIs that have spatial proximity to POIs of historical check-ins are given higher attention scores in the self-attention model.

• Finding 4: The optimal embedding combination is ASUEN (VIII). Comparing these two variants (VIII, IX), time embedding is not suggested to be added to our combination. Comparing these two variants (VII and VIII), we again demonstrate that the performance gap between noninvasive and invasive methods is significant. Our embedding combination by noninvasive method dramatically improves the accuracy of recommendation.

Table 3. The variants of embedding module.

Approach	User embedding	Time embedding	Spatial embedding	Noninvasive method
I. AUEI	√	×	×	×
II. AUEN	√	×	×	√
III. ATEI	×	√	×	×
Ⅳ. ATEN	×	√	×	√
Ⅴ. ASEI	×	×	√	×
Ⅵ. ASEN	×	×	√	√
Ⅶ. ASUEI	√	×	√	×
Ⅷ. ASUEN	√	×	√	√
Ⅸ. ASUTEN	√	√	√	√

Table 4. Comparison of the OM with their variants in Recall@$\mathit{k}$ and NDCG@$\mathit{k}$.

	TKY				NYC
Model	Recall@5	Recall@20	NDCG@5	NDCG@20	Recall@5	Recall@20	NDCG@5	NDCG@20
OM	0.2882	0.3943	0.2248	0.2532	0.4801	0.6038	0.3644	0.3974
I. AUEI	0.2673	0.3850	0.2023	0.2341	0.4637	0.5877	0.3424	0.3822
II. AUEN	0.2961	0.3976	0.2285	0.2569	0.4983	0.6256	0.3729	0.4066
III. ATEI	0.2769	0.3907	0.2233	0.2520	0.4741	0.5818	0.3614	0.3931
Ⅳ. ATEN	0.2855	0.3908	0.2252	0.2533	0.4882	0.6180	0.3682	0.4020
Ⅴ. ASEI	0.2865	0.3956	0.2227	0.2519	0.4941	0.6179	0.3637	0.3992
Ⅵ. ASEN	0.2965	0.4110	0.2285	0.2598	0.4943	0.6237	0.3711	0.4081
Ⅶ. ASUEI	0.2673	0.3851	0.2028	0.2329	0.4757	0.6133	0.3373	0.3778
Ⅷ. ASUEN	0.3069	0.4163	0.2368	0.2670	0.5013	0.6284	0.3712	0.4096
Ⅸ. ASUTEN	0.3067	0.4120	0.2353	0.2658	0.4999	0.6271	0.3704	0.4056

5.4.2. 5.4.2 Preference-intention module and spatial encoder module

To verify the effectiveness of key parts designed in our model, including the preference-intention module and spatial encoder module, we conduct an ablation study. Our base model is ASUEN (VIII), which has optimal embedding combination and form. We consider the following variants of the model:

• X. Add Preference-intention Module (APM): Based on the OM, we add the preference-intention module.

• XI. STILSAN: The variant (XI) is the complete model we proposed called STILSAN, which adds the preference-intention module to ASUEN.

• XII. Adopt Randomly Initialized Embedding (ARIE): Based on STILSAN (XI), we replace pretrained spatial embedding with randomly initialized embedding.

• XIII. Intention Learning Self-Attention Network (ILSAN): Based on STILSAN (XI), we remove spatio-temporal interval embedding so that revisiting mechanism does not consider spatial-temporal relation.

Table 5 shows the results of the ablation study. We can get the following findings.

• Finding 5: Preference-intention module leads to performance improvement. Comparing models (OM, APM) and variants (ASUEN, STILSAN), the models with preference-intention module all show higher performance. We can conclude that the preference-intention module is beneficial and plays an indispensable role in the model. This is because we capture the human mobility pattern of revisiting the POIs. Comparing ILSAN (XIII) and STILSAN (XI), we find that the model with spatial-temporal revisiting mechanism has higher performance than the model with revisiting mechanism. This is because the spatio-temporal intervals are explicitly incorporated into the preference-intention module, which helps to better mine user mobility patterns for revisiting.

• Finding 6: Spatial encoder module boosts the performance. Take Recall@20 as an example. STILSAN outperforms APM (Ⅹ) by about 2.5% and 1.3% on TKY and NYC, respectively. Meanwhile, STILSAN outperforms ARIE (Ⅻ) by about 0.98% and 1.7% on TKY and NYC, respectively. Therefore, spatial embedding, which is aware of spatial clustering phenomenon and distance, is crucial for improving recommendation performance. Randomly initialized embedding cannot effectively represent the spatial distance features of POIs.

Table 5. Comparison of the base model with their variants in Recall@$\mathit{k}$ and NDCG@$\mathit{k}$.

	TKY				NYC
Model	Recall@5	Recall@20	NDCG@5	NDCG@20	Recall@5	Recall@20	NDCG@5	NDCG@20
OM	0.2882	0.3943	0.2248	0.2532	0.4801	0.6038	0.3644	0.3974
Ⅹ. APM	0.2946	0.4040	0.2304	0.2594	0.4983	0.6161	0.3723	0.4060
Ⅷ. ASUEN	0.3069	0.4163	0.2368	0.2670	0.5013	0.6284	0.3712	0.4096
Ⅻ. ARIE	0.3099	0.4211	0.2366	0.2689	0.4948	0.6231	0.3705	0.4081
XIII. ILSAN	0.3174	0.4221	0.2460	0.2724	0.5069	0.6286	0.3740	0.4101
Ⅺ. STILSAN	0.3197	0.4300	0.2487	0.2770	0.5120	0.6370	0.3898	0.4226

5.5. Interpretability study

To understand the mechanism of preference-intention module, we adopt an interpretability study to prove the necessity and correctness of the preference-intention module. Figure 4 is based on a slice of a real user trajectory example. The sequential sequence of the trajectory examples is shown in Figure 4(a). The POI $l_6$ and POI $l_1$ as ground truths will be visited by the user at $t_7$ and $t_8$, respectively. We visualized the trajectory of the user on OpenStreetMap. Due to the large number of candidate POIs, we are unable to show them all on the map. Therefore, we select some typical candidate POIs including the historical POIs and non-historical POIs. POI $l_1$ to POI $l_5$ are historical POIs, and POI $l_6$ to POI $l_{10}$ are non-historical POIs which indicate that the user has not visited these POIs. Moreover, we collect the recommendation probabilities of intention and preference generated by STILSAN when the models reach convergence. To better visualize, we standardize the recommendation probabilities of the above 10 candidate POIs and took the standardized result as the score. Then, we display the preference scores and intention scores using vertical bars in the lower part of Figure 4. The height of each vertical bar denotes the size of recommendation scores, i.e., a candidate POI with higher vertical bars indicates that is more likely to be accessed by users in the next moment.

Figure 4. The Interpretability study of STILSAN. (a) a slice of a real user trajectory example. (b) the recommendation scores for 10 candidate POIs at the time $t_7$. (c) the recommendation scores for 10 candidate POIs at the time $t_8$. The POI with higher vertical bars indicates that are more likely to be accessed by users in the next moment.

By query of the exact GPS, we find that the POI $l_1$ and $l_5$ are different gyms. The POI $l_2$, $l_4$, and $l_6$are drink shop, delicatessen, and Italian restaurant, respectively, which are related to dining. The rest of the POIs are of other categories. It is important to note that POI categories are only used to better understand user movement scenarios. The proposed STILSAN emphasizes the intention to revisit historical POIs that have been visited but not the intention of POI categories. We can observe that the user often goes to the POI $l_1$ around noon, which belongs to the gym, as shown in Figure 4(a). After a trip to the gym, the user is more likely to go to a location associated with the restaurant. We can observe from the user trajectory segment that the user explored different gourmet restaurants. The vertical bars in Figure 4(b) show that POI $l_6$ as ground truth gets more preference scores and is recommended preferentially. At the same time, the user usually revisits the gym he has visited before. For example, we can observe that the user has the travel intention to go to the POI $l_1$ at the exact time. The vertical bars in Figure 4(c) show that POI $l_1$ as ground truth gets a lot of intention scores and a few preference scores. One reason is that preference and intention are often intertwined and both would be reflected by the human mobility, and the other reason is that it is difficult for the model to accurately distinguish the user’s movement pattern at each step. By comparison, we can observe that the proportion of intention score in Figure 4(c) is larger than that in Figure 4(b). This means that the model successfully identifies that the user’s movement behavior is more inclined to preference at the time $t_7$ and more inclined to intention at the time $t_8$. These examples further illustrate that STILSAN can capture both the user mobility pattern of intention and preference successfully and it has the capability to automatically recognize different scenarios.

6. Conclusions

In this paper, we propose a model called STILSAN for next POI recommendation task. In this model, spatio-temporal intention learning is integrated into the self-attention network. The user mobility pattern of revisiting POIs can be modeled explicitly. This work puts forward a new spatial encoder to model spatial feature so that spatial clustering phenomenon and distance can be considered in modeling. Experimental results indicate that STILSAN shows more competitive recommendation performance than existing state-of-the-art models on two real-world datasets. Besides, different embedding combinations and forms are evaluated to examine their influence on the recommendation performance. It reveals that adding spatial embedding and user embedding by noninvasive method is the optimal embedding combination and form. In addition, through ablation study, we show that the preference-intention module and spatial encoder module can improve recommendation performance significantly. One of the shortcomings of our model is that it is difficult to deal with the extremely cold-start issue in next POI recommendation. So, in the future, more context information such as POI category and social network is worth being incorporated into our model to mitigate the cold-start problem.

Acknowledgements

The authors appreciate the efforts of the anonymous reviewers and the editor. The authors thank Jian Li, Kai Yan, and Fan Yu for their contributions to the revision of the manuscript. The authors thank Chongqing Changan Automobile Co., Ltd., Dongfeng Motor Corporation, and Dongfeng Changxing Tech. Co., Ltd. for their technical guidance

Disclosure statement

No potential conflict of interest was reported by the authors.

Data availability statement

The data that support the findings in this study, as well as the code to process it with the methods presented in this paper, are available on Github at https://github.com/leehommlee/STILSAN.

Additional information

Funding

This work is supported by Chongqing Technology Innovation and Application Development Project [grant number cstc2021jscx-dxwtBX0023], and funding from Chongqing Changan Automobile Co., Ltd., Dongfeng Motor Corporation, and Dongfeng Changxing Tech Co., Ltd.

Notes on contributors

Hao Li

Hao Li is a PhD candidate at the School of Remote Sensing and Information Engineering, Wuhan University. His research interest is POI recommendation and knowledge graph.

Peng Yue

Peng Yue is a Professor at the School of Remote Sensing and Information Engineering, Wuhan University. He serves as the deputy dean at the School of Remote Sensing and Information Engineering, the director at the Hubei Province Engineering Center for Intelligent Geoprocessing (HPECIG), and the director at the Institute of Geospatial Information and Location Based Services (IGILBS), Wuhan University. His research interests include Earth science data and information systems, Web GIS and GIServices, and GIS software and engineering.

Shangcheng Li

Shangcheng Li is an engineer at Dongfeng Changxing Tech. Co., Ltd. His research interest is travel data mining.

Chenxiao Zhang

Chenxiao Zhang is an associate research fellow in the School of Remote Sensing and Information Engineering at Wuhan University. He received a PhD from the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University. His research interests include geographic information system, remote sensing, and deep learning.

Can Yang

Can Yang is a postdoctoral researcher in the School of Remote Sensing and Information Engineering, Wuhan University. He received a PhD from KTH, the Royal Institute of Technology. His research interests include trajectory data mining and movement analysis.