Exploring latent weight factors and global information for food-oriented cross-modal retrieval

Abstract

Food-oriented cross-modal retrieval aims to retrieve relevant recipes given food images or vice versa. The modality semantic gap between recipes and food images (text and image modalities) is the main challenge. Though several studies are introduced to bridge this gap, they still suffer from two major limitations: 1) The simple embedding concatenation only can capture the simple interactions rather than complex interactions between different recipe components. 2) The image feature extraction based on convolutional neural networks only considers the local features and ignores the global features of an image, as well as the interactions between different extracted features. This paper proposes a novel method based on Latent Component Weight Factors and Global Information (LCWF-GI) to learn the robust recipe and image representations for food-oriented cross-modal retrieval. This proposed method integrates the textual embeddings of different recipe components into a compact embedding to represent the recipes with the latent component-specific weight factors. A transformer encoder is utilised to capture the intra-modality interactions and the importance of different extracted image features for enhanced image representations. Finally, the bi-directional triplet loss is further used to perform retrieval learning. Experimental results on the Recipe 1M dataset show that our LCWF-GI method achieves competent improvements.

KEYWORDS:

• Food-oriented cross-modal retrieval
• image-recipe retrieval
• cross-modal food domain retrieval
• recipe representation
• image representation

1. Introduction

With the exponential growth of food-related multimodal resources on the Web, large-scale food domain information resources, such as cooking videos, food images, recipe texts, etc., can be easily accessed. These information resources are not only relevant to people’s privacy but also closely related to their health and diet (Liang et al., 2022b; Zhang et al., 2023). As an emerging research field, food computing has attracted wide attention (Min et al., 2017). Several relevant tasks in food computing have been explored, including but not limited to: food-oriented cross-modal retrieval (Chen et al., 2021; Fu et al., 2020; Salvador et al., 2017; Salvador et al., 2021; Wang et al., 2019), food classification (Krizhevsky et al., 2017; Martinel et al., 2018; Min et al., 2017), recipe recommendations (Elsweiler et al., 2017; Sanjo & Katsurai, 2017; Trattner & Elsweiler, 2017), and recipe ingredient analysis (Kagaya et al., 2014; Matsuda et al., 2012). Food-oriented cross-modal retrieval, as an essential way to acquire recipe or food image information, has been widely concerned by researchers. In this paper, we focus on this retrieval task, whose purpose is to retrieve recipe text with the help of food images or vice versa. Since it involves two modalities (recipe text and food image) in this task, there exists a semantic gap between them (Zhao et al., 2023). This leads to a sharp decline in the performance of food-oriented cross-modal retrieval compared with single-modal retrieval (Peng et al., 2017). How to bridge the semantic gap between recipes and images is the main challenge.

To address this challenge, the existing studies have proved that learning the consistent semantic representations of images and recipes is a possible resolution. Generally, a recipe text contains three components, including title, ingredient, and cooking instruction. Most existing studies commonly utilise popular deep learning techniques, such as long short term memory (LSTM) and its variants (Carvalho et al., 2018; Fu et al., 2020; Salvador et al., 2017), as well as Transformer encoders (Salvador et al., 2021), to learn the textual embeddings of three different recipe components and directly concatenate their textual embeddings to generate the final recipe representations. Then a convolutional neural network (such as the ResNet-50 model (He et al., 2016)) is used to learn image representations and the ranking learning function is leveraged to compute the semantic similarity between image and recipe representations. These studies can bridge the semantic gap between recipes and images to some degree and achieve reasonable retrieval performance. However, there exist two main limitations.

1. The simple and direct embedding concatenation of different recipe components only can model the simple interactions rather than the complex interactions between them. Compared with complex interactions, the simple and straightforward concatenation will result in the final undesirable recipe representations.

2. The image feature extraction based on a convolutional neural network (ResNet-50 model) only considers the local features of an image. It ignores the global features of this image and the interactions between different image features.

To deal with these issues, this paper proposes a novel method based on Latent Component Weight Factors and Global Information (referred to as LCWF-GI) for food-oriented cross-modal retrieval. This proposed method utilises three Transformer encoders with different parameters to extract textual features of three different recipe components (title, ingredient, and cooking instruction) respectively. In fact, the three recipe components closely correlate with each other, for instance, the cooking steps in the cooking instructions depend on the ingredients and quantity listed in the ingredient component. Tensor decomposition and tensor representation can be used to capture the relatedness between these recipe components and make full use of semantic information and features from them to guide robust recipe representation learning. Since a tensor is constructed by the outer product over different inputs, we use the outer product over textual embeddings of three recipe components in recipes to construct a tensor. Each order of this constructed tensor represents one kind of recipe component. We exploit tensor decomposition to obtain latent weight factors of three different recipe components, which represent the corresponding semantic features of these components. Then our LCWF-GI method leverages latent component weight factors and tensor representation (Fukui et al., 2016; Liu et al., 2018; Zadeh et al., 2017) to fuse and integrate these textual embeddings into the final compact recipe representation with richer semantic information. As we mentioned before, the previous studies directly concatenated the textual embeddings of three recipe components to acquire the final recipe representations. The embedding concatenation only can capture the simple interactions between these three recipe components by directly combining embeddings together. However, it may miss some important information. For example, if two textual embeddings of these recipe components are similar to each other in some dimensions and have unique information in other dimensions, the direct embedding concatenation may hide or cover up their similarity. Unlike the simple embedding concatenation, our proposed method utilises the element-wise product over textual embeddings of these three different components and their corresponding component-specific weight factors to represent the final recipe representations. This method can capture complex interactions because the element-wise product allows all elements of the component embeddings to interact with each other in their dimensions. The element level operation not only can retain its own information, but also can produce the new embedding through this element interaction between each other. Furthermore, after extracting the local image features with the ResNet-50 model (He et al., 2016), we further introduce a Transformer encoder to capture global information of images and interactions between different image features for enhanced image semantic representations. Finally, our proposed LCWF-GI method uses a bi-directional triplet loss function to perform the semantic matching between recipe and image representations. To make full use of the textual information of different recipe components and reduce the dependence on recipe-image pairs, our LCWF-GI method still considers the implicit semantic alignment between different recipe components. Several experiments are conducted on the standard Recipe 1M dataset. Compared with the representative food-oriented cross-modal retrieval baseline methods, experimental results prove that our LCWF-GI method can effectively improve retrieval performance.

The contributions can be summarised as follows:

1. We propose a novel method based on Latent Component Weight Factors and Global Information to learn consistent modality representations for food-oriented cross-modal retrieval. This proposed method utilises latent component weight factors to fuse the textual embeddings of different recipe components for the final recipe representations with deep semantic information. The generated representations can capture the complex interactions between the textual information of different components.

2. Based on the ResNet-50 model, we further introduce a Transformer encoder to learn the enhanced image semantic representations, which can capture the global information of images and the interaction relationships between the extracted image features.

3. Experimental results conducted on the Recipe 1M dataset show that our proposed method is superior to the representative baseline methods.

2. Related work

2.1. Food computing

With the exponential growth of multimodal food data that is closely relevant to the daily diet and people’s health available on the Internet, food computing is becoming an emerging research area. There are many popular research tasks in this area, such as food-oriented cross-modal retrieval (Chen et al., 2021; Fu et al., 2020; Salvador et al., 2017; Salvador et al., 2021; Wang et al., 2019), food classification (Krizhevsky et al., 2017; Martinel et al., 2018; Min et al., 2017), recipe recommendation (Elsweiler et al., 2017; Sanjo & Katsurai, 2017; Trattner & Elsweiler, 2017), and recipe ingredient analysis (Kagaya et al., 2014; Matsuda et al., 2012). Due to the power of deep learning techniques, they are widely used in many application domains (Li et al., 2022; Liang et al., 2022a; Long et al., 2022). Similarly, in these tasks, researchers also tend to use deep learning techniques or machine learning methods to extract the semantic information of recipes and images. For example, the research studies (Kawano & Yanai, 2014; Liu et al., 2016; Yanai & Kawano, 2015) used convolutional neural network-based methods to analyze or identify food in images. Kusmierczyk et al. leveraged Latent Dirichlet Algorithm (LDA) (Zhou et al., 2022a; Zhou et al., 2022b) to discover potential topic information in cooking instruction and exploited it to predict the nutritional value of meals (Kusmierczyk & Nørvåg, 2016). Similar to the work on recipe ingredient identification, most studies on food image classification utilise techniques such as transfer learning (Lee et al., 2018) and deep convolutional neural networks (Krizhevsky et al., 2017; Min et al., 2017) to learn image features for classifying food images.

2.2. Food-oriented cross-modal retrieval

Our work focuses on the food-oriented cross-modal retrieval task, where relevant recipes from a large number of recipe text resources can be retrieved by a given food image or vice versa. Since this task involves two modalities (recipe text and food image), there exists a semantic gap between the heterogeneous modalities. How to bridge the semantic gap between them to improve retrieval performance is the main challenge (Peng et al., 2017). To address this issue, learning consistent semantic representations of recipes and images and performing effective modality alignment can both be possible resolutions. Since consistent modality semantic representations can better assist modality alignment, we focus on learning consistent semantic representations of recipes and images.

The current research studies can be classified into two categories. The first category is to learn the semantic representations of recipes and images. The methods in this category tend to use different neural networks (such as LSTM, Bi-LSTM) and a popular deep convolutional network (the ResNet-50 model (He et al., 2016)) to learn the semantic representations of two modalities. Finally, the normal metric learning functions (triplet loss function and its variants) are leveraged to perform semantic matching between modality representations for food-oriented cross-modal retrieval learning. The major difference between methods in this category lies in the use of different neural networks to learn the recipe semantic representations. Therefore, we do not introduce the image semantic representation learning and retrieval ranking functions in detail and only focus on recipe representation learning. An earlier study (Salvador et al., 2017) used Bi-LSTM and LSTM to extract textual features from two recipe components (ingredient and cooking instruction) for the corresponding textual embeddings respectively and then directly concatenated them for the final recipe representations. Similar to this work (Salvador et al., 2017), Carvalho et al. (Carvalho et al., 2018) used Bi-LSTM and Hierarchical LSTM to encode the textual information of the ingredient and cooking instruction components to learn their textual embeddings and then generated the final recipe representations by direct embedding concatenation. Since these two studies only considered the textual information of two recipe components (ingredient and cooking instruction) and used LSTM and Bi-LSTM to encode the recipe text in a simple way, the quality of the generated recipe semantic representations needs to be further improved. To make full use of the textual information of different recipe components and explore the deep recipe semantic information, Chen et al. (Chen et al., 2018) leveraged Bi-GRU and attention networks to capture the word-word interactions in the recipe title, ingredient and cooking instruction, as well as the sentence-sentence interactions in cooking instruction to produce textual embeddings of different three recipe components. Then they concatenated them together to generate the final recipe semantic representation. Compared with the earlier work (Carvalho et al., 2018; Salvador et al., 2017), their proposed method not only considered the textual information of the ingredient and cooking instruction components, but also further considered the textual information of the title component. They also introduced an attention mechanism to capture fine-grained information about three recipe components for better recipe semantic representations. However, the ability to capture the long-range dependencies among the textual information in recipes needs to be improved when encoding the recipe components. The transformer technique can address this limitation of Bi-GRU and Bi-LSTM and better capture the long-range dependencies among the textual information in different recipe components. Salvador et al. (Salvador et al., 2021) proposed a hierarchical Transformer framework for food-oriented cross-modal retrieval, where three different hierarchical Transformer encoders (Vaswani et al., 2017) are used to encode word-level and sentence-level information of three recipe components (title, ingredient, and cooking instruction) for their corresponding textual embeddings. Then these textual embeddings are concatenated to generate final recipe representations. In Pham et al.'s work (Pham et al., 2021), they used a Bi-LSTM to encode the title and a tree-structured LSTM as a recipe component text encoder to capture the hierarchical relationships and more meaningful semantic information in two recipe components (ingredient and cooking instruction). Then these learned corresponding textual embeddings are concatenated for the final recipe representations. Overall, these research studies focus on recipe representation learning by learning the textual embeddings of each recipe component based on different neural networks separately and then directly concatenating them to construct the final semantic representations of recipes. However, the direct and simple embedding concatenation for recipe representation learning can only capture the simple interactions rather than complex interactions between the textual information of different recipe components. Furthermore, some information among the textual embeddings of these recipe components may be missing.

The second category mainly focuses on modality alignment. After obtaining the semantic representations of recipes and images, the methods in the second category propose complex models or loss functions to align and refine modality representations for food-oriented cross-modal retrieval. The recipe representation learning of these methods in the second category is similar to that in the first category. Meanwhile, they also utilise the ResNet-50 model as the image encoder to generate image semantic representations. Therefore, we do not introduce the recipe and image semantic representation learning here, but only focus on the modality alignment methods. Wang et al. (Wang et al., 2019) proposed an adversarial network-based modality alignment method to align the learned recipe and image semantic representations. Then they used a cross-modal translation consistency function to assist this alignment process for retrieval learning. This method guided the learning process of the modality semantic representations by using the discriminators and generators of an adversarial network to bridge the gap between them. Similar to this work (Wang et al., 2019), Zhu et al. (Zhu et al., 2019) used an augmented adversarial network to align the semantic representations of recipes and images and introduced a two-level ranking loss function to improve the robustness of the alignment during the alignment process. Fu et al. (Fu et al., 2020) used a cross-modal attention mechanism to learn the interactions of recipe and image representations to generate new semantic representations for them. Finally, the enhanced modality semantic representations are aligned by using a hidden variable model. In Wang et al.'s work (Wang et al., 2022), they introduced the semantic consistency network to regularise and align the recipe and image semantic representations. These methods focus on bridging the semantic distance between the representations of two modalities by using a complex alignment model or auxiliary loss functions.

Overall, the existing food-oriented cross-modal retrieval methods typically use a deep convolutional neural network (the ResNet-50 model (He et al., 2016)) to encode the image semantic information. Different neural networks are utilised to learn textual embeddings of recipe components in a recipe. Then these textual embeddings of different recipe components are directly concatenated together to generate the final recipe representations. However, these methods can only capture simple interactions rather than complex interactions between textual information in different recipe components. Second, the image representations based on the ResNet-50 model only consider the local information of an image and ignore its global information and the interactions between the different extracted image features. Therefore, our work proposes a novel consistent semantic representation learning method for food-oriented cross-modal retrieval.

3. The proposed method

In the food-oriented cross-modal retrieval task, which aims to match the representations of images and recipes. Given a set of recipe-image pairs with size $N$: $\{x_i^R,x_i^I{\}}_{i=1}^N$, where $x_i^R\in R$ denotes a recipe text from the recipe set $R$ and $x_i^I\in I$ denotes an image from the food image set $I$. Each recipe consists of three components: title, ingredient and cooking instruction, which are denoted as $x_i^R=$ ($t^{ttl}$, $g^{ing}$, $s^{ins}$). The existing studies (Fu et al., 2020; Salvador et al., 2017; Salvador et al., 2021; Wang et al., 2019) tended to use popular neural networks to encode the textual information of different recipe components ($t^{ttl}$, $g^{ing}$, $s^{ins}$) into the corresponding textual embeddings (${\mathit{e}}_t^{ttl},{\mathit{e}}_g^{ing},{\mathit{e}}_s^{ins}$), and then concatenate them to generate the final recipe text representation ${\mathit{E}}^R=[{\mathit{e}}_t^{ttl},{\mathit{e}}_g^{ing},{\mathit{e}}_s^{ins}]$. In this work, the goal of our proposed method is to generate the improved image semantic representation ${\mathit{E}}^I$ and the refined recipe representation ${\mathit{E}}^R$. A Transformer architecture is used to encode image features from the ResNet-50 model for improved image semantic representation ${\mathit{E}}^I$. The textual embeddings of different recipe components are leveraged to learn the refined recipe representation ${\mathit{E}}^R$with the latent component weight factors and tensor decomposition. The framework of our proposed method is shown in Figure 1, which includes three main components: 1) image representation learning; 2) recipe representation learning; 3) food-oriented cross-modal retrieval learning. The symbols used in this paper and their corresponding meanings are shown in Table 1.

Figure 1. The overall framework of the proposed method.

Table 1. The symbols and corresponding meanings in the current paper

Symbols	Meanings	Symbols	Meanings
$R$	the cooking recipe collection	${\mathit{x}}_1$,${\mathit{x}}_2$,${\mathit{x}}_3$	the vector space of title, ingredient, cooking instruction components
$I$	the food image collection	$\mathit{X}$	the constructed input tensor
$N$	the number of image-recipe pairs	$d_1$, $d_2$, $d_3$	dimensionality size of ${\mathit{x}}_1$, ${\mathit{x}}_2$, ${\mathit{x}}_3$
$x_i^R$	a recipe	$\mathcal{W}$	the weight tensor of a linear layer
$x_i^I$	a food image	$\mathit{b}$	the bias of a linear layer
$t^{ttl}$	recipe title component	$d_r$	the dimension size of the final recipe embeddings
$g^{ing}$	recipe ingredient component	$k$	the index of $d_r$
$s^{ins}$	cooking instruction component	${\mathit{W}}_k$	Order-3 weight tensor
${\mathit{e}}_t^{ttl}$	the textual embedding of title	$\widetilde{{\mathcal{W}}_k}$	low-rank weight tensor
${\mathit{e}}_g^{ing}$	the textual embedding of ingredient	${{\mathit{w}}_k}_{(j)}^{ttl}$	the weight vector of the title component
${\mathit{e}}_s^{ins}$	the textual embedding of cooking instruction	${{\mathit{w}}_k}_{(j)}^{ing}$	the weight vector of the ingredient component
${\mathit{E}}^R$	representation of a recipe	${{\mathit{w}}_k}_{(j)}^{ins}$	the weight vector of the ingredient component
${\mathit{E}}^I$	representation of a food image	$\mathrm{\Re }$	the minimum rank value
${\psi }_{img}$	the image encoder	$r$	the rank value of the weight tensor
${\mathit{e}}_{res}^I$	feature embedding of image $x_i^I$	${\mathit{w}}_{(j)}^{ttl}$	the low-rank factors for the title component
${\mathit{W}}_{I_q},{\mathit{W}}_{I_k},{\mathit{W}}_{I_v},{\mathit{W}}^{\mathbf{\prime }}$	trainable weight matrices	${\mathit{w}}_{(j)}^{ing}$	the low-rank factors for the ingredient component
$d_I$	the dimension size of a self-attention layer	${\mathit{w}}_{(j)}^{ins}$	the low-rank factors for the instruction component
$wd$	a word token	$\alpha$	the hyper-parameters for image-recipe pair loss
$t$	the number of word tokens in a sentence	$\beta$	the hyper-parameters for recipe component loss
$Sq$	a sentence	$m$	margin value
$\mathit{S}{\mathit{q}}^{ttl}$	the sentence embedding of the title	${\mathit{E}}_p^R$, ${\mathit{E}}_p^I$	positive samples of recipes and images
${\mathit{S}\mathit{q}}_j^{ing}$	the sentence embedding of the $j$th sentence in the ingredient component	${\mathit{E}}_n^R$, ${\mathit{E}}_n^I$	negative samples of recipes and images
${\mathit{S}\mathit{q}}_j^{ins}$	the sentence embedding of the $j$th sentence in the cooking instruction component	${\mathit{E}}_n^I$	negative samples of images
$M$	the number of sentences of ingredient component	$a,b$	the representations of any two recipe components
$P$	the number of sentences of cooking instruction component	${\mathit{a}}_p$, ${\mathit{b}}_p$	positive samples of $a$ and $b$
${\mathit{W}}_{tt{t}_q},{\mathit{W}}_{tt{t}_k},{\mathit{W}}_{tt{l}_v},{\mathit{W}}_c^{\mathrm{\prime }}$	the learned weight matrices	${\mathit{a}}_n$, ${\mathit{b}}_n$	negative samples of $a$ and $b$

3.1. Image representation learning

The existing studies (Fu et al., 2020; Salvador et al., 2021; Wang et al., 2019) usually use a deep convolutional network (the ResNet-50 model (He et al., 2016)) as an image encoder to extract the features of images and regard their corresponding embeddings as image representations. This image encoder can be defined as ${\psi }_{img}$, and the feature embedding of an image can be defined as: (1) $\begin{array}{r}{\mathit{e}}_{res}^I={\psi }_{img}(x_i^I)\end{array}$(1) where ${\mathit{e}}_{res}^I$ represents the corresponding feature embedding of the image $x_i^I$ with the help of the ResNet-50 model. As mentioned in the existing studies (Fu et al., 2020; Salvador et al., 2021; Wang et al., 2019), the image feature embeddings are obtained from the last two layers of the ResNet-50 model (because embeddings from the last layer are combined with a softmax function for prediction task). Therefore, the feature embedding ${\mathit{e}}_{res}^I$ is regarded as the image representation in these studies.

Though feature embeddings as image representations can somewhat represent the image semantic information, the feature extraction based on the ResNet-50 model only considers the local information rather than the global information of images. Furthermore, there may be some correlations between different extracted feature embeddings of the same image, while the interactions between them also are neglected. To alleviate these limitations and learn richer image semantic information, a 2-layer Transformer encoder (Vaswani et al., 2017) is used to re-encode the feature embeddings (${\mathit{e}}_{res}^I$) from the ResNet-50 model to generate the enhanced image semantic representations. In this 2-layer Transformer encoder, there are 4 self-attention heads in each layer and the multi-head self-attention mechanism can jointly focus on important information at different locations to capture the interaction information between extracted local features. The image feature embedding ${\mathit{e}}_{res}^I$ extracted by the ResNet-50 model is treated as queries, keys and values in the self-attention network, and the multi-headed self-attention mechanism of the image Transformer encoder can be expressed as: (2) $\begin{array}{rl}Hea{d}_i({\mathit{e}}_{res}^I)& =softmax\left(\frac{{[{\mathit{W}}_{I_q}{\mathit{e}}_{res}^I]}^T{\mathit{W}}_{I_k}{\mathit{e}}_{res}^I}{\sqrt{d_I}}\right)\cdot [{\mathit{W}}_{I_v}{\mathit{e}}_{res}^I{]}^T\end{array}$(2) (3) $\begin{array}{rl}MultiHead({\mathit{e}}_{res}^I)& =[Hea{d}_1,\cdots ,Hea{d}_4{]}^T{\mathit{W}}^{\mathbf{\prime }}\end{array}$(3) where $Hea{d}_i$ represents the $i$th self-attention head in the image Transformer encoder, ${\mathit{W}}_{I_q},{\mathit{W}}_{I_k},{\mathit{W}}_{I_v},{\mathit{W}}^{\mathbf{\prime }}$ are the trainable weight matrices, $\sqrt{d_I}$ is the scaling factor and it is equal to the dimension of a self-attention layer.

To avoid gradient vanishment and speed up convergence, the multi-head self-attention layer, the feed-forward neural network, and the normalisation layer are combined to generate the final image semantic representation. (4) $\begin{array}{rl}{\mathit{e}}_i^{I^{\mathrm{\prime }}}& =LN({\mathit{e}}_{res}^I+MultiHead({\mathit{e}}_{res}^I))\end{array}$(4) (5) $\begin{array}{rl}{\mathit{E}}^I& =LN(\mathit{e}{i}^{I^{\mathrm{\prime }}}+FFN(\mathit{e}{i}^{I^{\mathrm{\prime }}}))\end{array}$(5) where $MultiHead$ is the multi-head self-attention layer, $FFN$ is the feed-forward neural network, $LN$ is the normalisation layer, and ${\mathit{E}}^I$ is the final enhanced image semantic representation. The enhanced image representation not only can consider the local information of images, but also the global information and interactions between different extracted local image features. Therefore, the image representation ${\mathit{E}}^I$ can contain richer semantic information.

3.2. Recipe representation learning

3.2.1. Textual information encoding for recipe components

Each recipe contains three components: title, ingredient, and cooking instruction. The recipe title component is usually represented as a sentence that highly summarises this recipe, while the recipe ingredient and cooking instruction components both are composed of multiple sentences. To generate strong recipe semantic representations, similar to this work (Salvador et al., 2021), our work also uses three Transformer encoders with the same structure and different parameters to encode the textual information of three recipe components (title, ingredient and cooking instruction) respectively. As we mentioned before, since the recipe ingredient and cooking instruction components usually are made of multiple sentences, the title component only consists of one sentence. A sequence (sentence) includes many word tokens and it denotes as $Sq=(w{d}_0,\cdots ,w{d}_t)$, where $Sq$ represents a sentence and $wd$ means a word token. Then the title can be expressed as ${\mathit{t}}^{ttl}=\mathit{S}{\mathit{q}}^{ttl}$, $\mathit{S}{\mathit{q}}^{ttl}$is the sentence embedding of the title, the ingredient and cooking instruction can be denoted as ${\mathit{g}}^{ing}=({\mathit{S}\mathit{q}}_0^{ing},\cdots {\mathit{S}\mathit{q}}_j^{ing},\cdots ,{\mathit{S}\mathit{q}}_M^{ing})$ and ${\mathit{s}}^{ins}=({\mathit{S}\mathit{q}}_0^{ins},\cdots {\mathit{S}\mathit{q}}_j^{ins},\cdots ,{\mathit{S}\mathit{q}}_P^{ins})$ respectively. These component text encoders make full use of the multi-head self-attention mechanism in the Transformer architecture to capture the long-range dependencies and word-word level interactions of text information in three different recipe components. This can learn the textual embeddings of these components with deep semantic information. The architecture of these component text encoders is similar to that of the image Transformer encoder used in Section 3.1, which is also a 2-layer Transformer encoder (Vaswani et al., 2017) with 4 self-attention heads in each layer. The recipe title ${\mathit{t}}^{ttl}$ is treated as queries, keys, and values in the self-attention network of the Transformer text encoder for the title component, and the multi-head self-attention mechanism in this component text encoder can be defined as: (6) $\begin{array}{rl}Hea{d}_i({\mathit{e}}_t)& =softmax\left(\frac{{[{\mathit{W}}_{tt{t}_q}{\mathit{t}}^{ttl}]}^T{\mathit{W}}_{tt{t}_k}{\mathit{t}}^{ttl}}{\sqrt{d_{TI}}}\right)\cdot [{\mathit{W}}_{tt{l}_v}{\mathit{t}}^{ttl}{]}^T\end{array}$(6) (7) $\begin{array}{rl}MultiHead({\mathit{e}}_t)& =[Hea{d}_1,\cdots ,Hea{d}_4){]}^T{\mathit{W}}_c^{\mathrm{\prime }}\end{array}$(7) where $Hea{d}_i$ denotes the $i$th self-attention head in the Transformer encoder for the title component, ${\mathit{W}}_{tt{t}_q},{\mathit{W}}_{tt{t}_k},{\mathit{W}}_{tt{l}_v},{\mathit{W}}_c^{\mathrm{\prime }}$ are the weight matrices to be learned, $\sqrt{d_{TI}}$ is the scaling factor, and its size is equal to the dimension of the self-attention layer.

Similar to the image encoder, to improve the model performance and avoid gradient vanishment, we also combine the multi-headed self-attention layer, the feed-forward neural network, and the normalisation layer to generate the textual embeddings of the title component. (8) $\begin{array}{r}{\mathit{e}}_t^{{ttl}^{\mathrm{\prime }}}=LN({\mathit{e}}_t+MultiHead({\mathit{e}}_t))\end{array}$(8) (9) $\begin{array}{r}{\mathit{e}}_t^{ttl}=LN({\mathit{e}}_t^{{ttl}^{\mathrm{\prime }}}+FFN({\mathit{e}}_t^{tt{l}^{\mathrm{\prime }}}))\end{array}$(9) where $MultiHead$ is the multi-head self-attention layer, $FFN$ is the feed-forward neural network, $LN$ is the normalisation layer, and ${\mathit{e}}_t^{ttl}$ represents the semantic textual embeddings of the title component. Then, as for the ingredient and cooking instruction components, we also use a similar way to obtain their corresponding textual embeddings ${\mathit{e}}_g^{ing}$and ${\mathit{e}}_s^{ins}$ respectively.

3.2.2. Recipe representation learning based on latent weight factors

Most current research methods generate textual embeddings of different recipe components, then obtain the final recipe semantic representations by directly concatenating these corresponding textual embeddings. However, the simple embedding concatenation for final recipe representations can only model and capture the simple interactions rather than complex interactions between the textual information of different components. Unlike these methods (Fu et al., 2020; Salvador et al., 2017; Salvador et al., 2021; Wang et al., 2019), we propose a novel method based on Latent Component Weight Factors and Global Information (referred to as LCWF-GI) to learn consistent modality representations for food-oriented cross-modal retrieval. This proposed LCWF-GI method first learns textual embeddings of three components (title, ingredient and cooking instruction) via the component Transformer encoders. Then latent component-specific weight factors are used to fuse and integrate these textual embeddings of multiple components into the unified and compact recipe semantic representation. As we mentioned before, a recipe contains three components: title, ingredient, and cooking instruction. The title component summarises the name of this recipe, while ingredient and cooking instruction provide the textual information of the ingredients, quantity, unit information in this recipe and the cooking steps to prepare this dish respectively. These three recipe components are interdependent, and intimately connected to each other. To learn the final robust recipe representations with richer semantic information, we fuse and integrate their corresponding textual embeddings into the unified embedding for representing the final recipe representation. The fusion method in our work is different from that in the studies (Arevalo et al., 2017; Yuan et al., 2021; Yuan et al., 2022a; Yuan et al., 2022b; Yuan et al., 2022c). For example, Yuan et al. (Yuan et al., 2022a) utilised visual features to guide the text representations. Besides, the element-wise product operation was used to compute the cross-modal similarity between visual and text representations. However, different from the study (Yuan et al., 2022a), our work only focuses on leveraging the textual information of three recipe components to learn recipe representations. Furthermore, the element-wise product operation is leveraged to fuse textual embeddings of three recipe components (title, ingredient, and cooking instruction) for acquiring the final recipe representations. Besides, the recipe representation generation only is closely correlated with their recipe components and has no connection with images. Finally, the bi-triplet loss function rather than element-wise product is adopted to calculate the cross-modal similarity between the representations of recipes and images.

As we mentioned before, the three recipe components closely correlate with each other, for example, the cooking steps in the cooking instructions depend on the ingredients and quantity listed in the ingredient component. Tensor decomposition and tensor representation can be utilised to capture the relatedness between these recipe components and make full use of semantic information and features from them for guiding the recipe representations. Therefore, we leverage tensor decomposition to acquire latent weight factors of three different recipe components, because latent weight factors represent the corresponding semantic features of these components. Tensor representation can transform different input representations into a high-dimensional tensor and map this tensor into a low-dimensional vector space (Fukui et al., 2016; Liu et al., 2018; Zadeh et al., 2017). Though we first attempt to use tensor representation to fuse three recipe components for generating the final recipe representations, the tensor fusion network is commonly used in multimodal learning (Fukui et al., 2016; Jin et al., 2020; Liu et al., 2018; Zadeh et al., 2017). Similarly, we use the tensor fusion network to fuse different recipe components rather than multimodal data. In this work, we use the textual embeddings of three recipe components (${\mathit{e}}_t^{ttl}$,${\mathit{e}}_g^{ing}$, ${\mathit{e}}_s^{ins}$) to construct an input tensor $\mathit{X}$ via outer product operation, so we can use each order of the input tensor $\mathit{X}$ to represent the component information. However, to model the interactions between three different recipe components as well as to maintain their own semantic information for each recipe component, a 1 is appended to the textual embeddings of each recipe component before constructing the input tensor $\mathit{X}$ via the outer product operation, i.e. ${\mathit{x}}_1=[{\mathit{e}}_t^{ttl}1]\in {\mathbb{R}}^{{\mathit{d}}_1}$,${\mathit{x}}_2=[{\mathit{e}}_g^{ing}1]\in {\mathbb{R}}^{{\mathit{d}}_2}$,${\mathit{x}}_3=[{\mathit{e}}_s^{ins}1]\in {\mathbb{R}}^{{\mathit{d}}_3}$. The constructed input tensor $\mathit{X}$ can be defined as: (10) $\begin{array}{r}\mathit{X}={\mathit{x}}_1\otimes {\mathit{x}}_2\otimes {\mathit{x}}_3\end{array}$(10) where $\otimes$ denotes the outer tensor product, $\mathit{X}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ is an order-3 tensor,$d_1$, $d_2$, $d_3$denote the dimensionality size of ${\mathit{x}}_1$, ${\mathit{x}}_2$, ${\mathit{x}}_3$ (textual embeddings of the title, ingredient, and cooking instruction components after appending 1) respectively.

Then the tensor $\mathit{X}$ is fed into a linear layer $g(\cdot )$ to generate the recipe semantic representation. The recipe semantic representation can be defined as: (11) $\begin{array}{r}{\mathit{E}}^R=g(\mathit{X};\mathcal{W},\mathit{b})=\mathit{X}\cdot \mathcal{W}+\mathit{b}\end{array}$(11) where $g(\cdot )$ is denoted as a linear layer, $\mathcal{W}$ and $\mathit{b}$ are the weight tensor and bias of the linear layer respectively. Since $\mathit{X}$ is an order-3 tensor (the order of $\mathit{X}$ is equal to the number of recipe components used, our work uses all three recipe components), the weight tensor $\mathcal{W}\in {\mathbb{R}}^{d_1\times d_2\times d_3\times d_r}$ is an order-4 tensor. The extra order ($d_r$) of $\mathcal{W}$ corresponds to the dimension size $d_r$ of the final output recipe representation ${\mathit{E}}^R$. In the dot product operation of $\mathit{X}\cdot \mathcal{W}$, the weight tensor $\mathcal{W}$ can be formed as a stack of $d_r$ order-3 tensors. In other words, the weight tensor $\mathcal{W}$ can be partitioned into $d_r$ order-3 ${\mathit{W}}_k\in {\mathbb{R}}^{d_1\times d_2\times d_3}$, $k=1,\cdots ,d_r$.

To efficiently acquire the final recipe semantic representation, this LCWF-GI method focuses on decomposing the weight tensor $\mathcal{W}$ into a set of latent weight factors corresponding to three different components. This tensor decomposition process not only can project the semantic information of these recipe components into a low-dimensional space to catch their semantic features, but also capture the relevance between these components. As $\mathcal{W}$ is an order-4 tensor, if $\mathcal{W}$ is directly decomposed by using traditional methods, four kinds of elements will be generated. As mentioned earlier, the weight tensor $\mathcal{W}$ is formed by stacking $d_r$ order-3 tensor ${\mathit{W}}_k\in {\mathbb{R}}^{d_1\times d_2\times d_3}$, $k=1,\cdots ,d_r$, so the decomposition of $\mathcal{W}$ is converted into a separate decomposition of each ${\mathit{W}}_k$. For any order-3 tensor ${\mathit{W}}_k$, there will always exist an exact decomposition form that can decompose the tensor into vectors. This decomposition form is defined as: (12) $\begin{array}{r}{\mathit{W}}_k=\sum _{j=1}^{\mathrm{\Re }}({{\mathit{w}}_k}_{(j)}^{ttl}\otimes {{\mathit{w}}_k}_{(j)}^{ing}\otimes {{\mathit{w}}_k}_{(j)}^{ins})\end{array}$(12) where $\otimes$ is the outer product, ${{\mathit{w}}_k}_{(j)}^{ttl}$,${{\mathit{w}}_k}_{(j)}^{ing}$ and ${{\mathit{w}}_k}_{(j)}^{ins}$ are weight vectors of the title, ingredient and cooking instruction components after decomposing the $k$-th order-3 tensor ${\mathit{W}}_k$ respectively. $\mathrm{\Re }$ is the minimum rank value that makes the decomposition form valid. The set of weight vectors $\{{{\mathit{w}}_k}_{(j)}^{ttl},{{\mathit{w}}_k}_{(j)}^{ing},{{\mathit{w}}_k}_{(j)}^{ins}{\}}_j^{\mathrm{\Re }}$ is called as the $\mathrm{\Re }$ latent weight factors of order-3 weight tensor ${\mathit{W}}_k$. These weight vectors (low-order tensors in low-dimensional semantic space) also represent the corresponding semantic features of these three recipe components (title, ingredient, and cooking instruction).

In contrast to this decomposition process, we attempt to recover the original weight tensor via the low-rank approximation. The low-rank approximation can keep the important features and remove the noisy features. Therefore, a fixed rank value $r$ is used. These $r$ latent weight decomposition factors $\{{{\mathit{w}}_k}_{(j)}^{ttl},{{\mathit{w}}_k}_{(j)}^{ing},{{\mathit{w}}_k}_{(j)}^{ins}{\}}_j^r,k=1,\cdots ,d_r$ are leveraged to parametrically represent the recipe component fusion method. In other words, we leverage the $r$decomposition factors $\{{{\mathit{w}}_k}_{(j)}^{ttl},{{\mathit{w}}_k}_{(j)}^{ing},{{\mathit{w}}_k}_{(j)}^{ins}{\}}_j^r$ to reconstruct the low-rank weight tensor ${\mathit{W}}_k$. Each recipe component has its corresponding low-rank latent component weight factors, which are obtained by combining and concatenating these weight vectors into the low-rank latent component-specific weight factors. For example, as for the title component, the corresponding vectors are combined and concatenated to obtain the corresponding low-rank latent weight factors ${\mathit{w}}_{(j)}^{ttl}=[{{\mathit{w}}_1}_{(j)}^{ttl},{{\mathit{w}}_2}_{(j)}^{ttl},\cdots ,{{\mathit{w}}_k}_{(j)}^{ttl},\cdots ,{{\mathit{w}}_{d_r}}_{(j)}^{ttl}]$, $d_r$ is the dimension size of the final recipe representation, then $\{{\mathit{w}}_{(j)}^{ttl}{\}}_j^r$ are the low-rank latent weight factor of the title component. In a similar way, the low-rank latent weight factor $\{{\mathit{w}}_{(j)}^{ing}{\}}_j^r$ of the ingredient component and the low-rank latent weight factor $\{{\mathit{w}}_{(j)}^{ins}{\}}_j^r$ of the cooking instruction component can be obtained. Then the weight tensor $\mathcal{W}$ can be reformulated by the low-rank latent weight factors of three recipe components. Thus, Equation (11)(11) $\begin{array}{r}{\mathit{E}}^R=g(\mathit{X};\mathcal{W},\mathit{b})=\mathit{X}\cdot \mathcal{W}+\mathit{b}\end{array}$(11) can be rewritten as: (13) $\begin{array}{rl}\mathcal{W}& =\sum _{j=1}^r({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins})\end{array}$(13) (14) $\begin{array}{rl}{\mathit{E}}^R& =\mathit{X}\cdot \left(\sum _{j=1}^r({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins})\right)\end{array}$(14) where ${\mathit{w}}_{(j)}^{ttl}\in {\mathbb{R}}^{d_1\times d_r}$, ${\mathit{w}}_{(j)}^{ing}\in {\mathbb{R}}^{d_2\times d_r}$, ${\mathit{w}}_{(j)}^{ins}\in {\mathbb{R}}^{d_3\times d_r}$. It can be seen that they maintain the same second dimension $d_r$. The outer product is performed only in the first dimension, i.e. ${\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\in {\mathbb{R}}^{d_1\times d_2\times d_r},{\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ins}\in {\mathbb{R}}^{d_1\times d_3\times d_r},{\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins}\in {\mathbb{R}}^{d_2\times d_3\times d_r}$.

To efficiently fuse the textual information of different recipe components, the input tensor $\mathit{X}$ can be decomposed into textual embeddings of these three components in parallel when the weight tensor $\mathcal{W}$ is decomposed. According to Equation (10)(10) $\begin{array}{r}\mathit{X}={\mathit{x}}_1\otimes {\mathit{x}}_2\otimes {\mathit{x}}_3\end{array}$(10) and the mixed product property theorem of the tensor, Equation (14)(14) $\begin{array}{rl}{\mathit{E}}^R& =\mathit{X}\cdot \left(\sum _{j=1}^r({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins})\right)\end{array}$(14) can be redefined as: (15) $\begin{array}{rl}{\mathit{E}}^R& =\mathit{X}\cdot \left(\sum _{j=1}^r({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins})\right)=\sum _{j=1}^r(\mathit{X}\cdot ({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins}))\\ & =\sum _{j=1}^r(({\mathit{x}}_1\otimes {\mathit{x}}_2\otimes {\mathit{x}}_3)\cdot ({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins}))\\ & =\left[\sum _{j=1}^r({\mathit{x}}_1\cdot {\mathit{w}}_{(j)}^{ttl})\right]\circ \left[\sum _{j=1}^r({\mathit{x}}_2\cdot {\mathit{w}}_{(j)}^{ing})\right]\circ \left[\sum _{j=1}^r({\mathit{x}}_3\cdot {\mathit{w}}_{(j)}^{ins})\right]\end{array}$(15) where $\otimes$ is the outer product, $\cdot$ is the dot product, and ${}^{\circ }$ is denoted as the element-wise product. After decomposing the weight tensor $\mathcal{W}$ and the input tensor $\mathit{X}$ in parallel, we can directly compute the recipe representations without actually computing the tensor $\mathit{X}$. As seen from Equation (15)(15) $\begin{array}{rl}{\mathit{E}}^R& =\mathit{X}\cdot \left(\sum _{j=1}^r({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins})\right)=\sum _{j=1}^r(\mathit{X}\cdot ({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins}))\\ & =\sum _{j=1}^r(({\mathit{x}}_1\otimes {\mathit{x}}_2\otimes {\mathit{x}}_3)\cdot ({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins}))\\ & =\left[\sum _{j=1}^r({\mathit{x}}_1\cdot {\mathit{w}}_{(j)}^{ttl})\right]\circ \left[\sum _{j=1}^r({\mathit{x}}_2\cdot {\mathit{w}}_{(j)}^{ing})\right]\circ \left[\sum _{j=1}^r({\mathit{x}}_3\cdot {\mathit{w}}_{(j)}^{ins})\right]\end{array}$(15) , the final recipe semantic representations can be expressed as element-wise over the textual embeddings of three recipe components and their corresponding latent weight factors. Equation (15)(15) $\begin{array}{rl}{\mathit{E}}^R& =\mathit{X}\cdot \left(\sum _{j=1}^r({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins})\right)=\sum _{j=1}^r(\mathit{X}\cdot ({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins}))\\ & =\sum _{j=1}^r(({\mathit{x}}_1\otimes {\mathit{x}}_2\otimes {\mathit{x}}_3)\cdot ({\mathit{w}}_{(j)}^{ttl}\otimes {\mathit{w}}_{(j)}^{ing}\otimes {\mathit{w}}_{(j)}^{ins}))\\ & =\left[\sum _{j=1}^r({\mathit{x}}_1\cdot {\mathit{w}}_{(j)}^{ttl})\right]\circ \left[\sum _{j=1}^r({\mathit{x}}_2\cdot {\mathit{w}}_{(j)}^{ing})\right]\circ \left[\sum _{j=1}^r({\mathit{x}}_3\cdot {\mathit{w}}_{(j)}^{ins})\right]\end{array}$(15) corresponds to the module of Fusion of Textual Embedding of Multiple Components based on Latent Weight Factors in Figure 1. Compared with the direct embedding concatenation method, this proposed LCWF-GI method allows the interactions between individual elements in textual embeddings of different recipe components through the element-wise product operation. Therefore, the final recipe representation can capture the complex interactions between the text information of different recipe components rather than simple interactions between them.

3.3. Food-oriented cross-modal retrieval learning

Similar to the work (Salvador et al., 2021), our objective loss function contains two parts. The first part is a bi-directional triplet loss function ${\mathcal{L}}_{pair}$based on recipe-image pairs, and this function is mainly used to learn the similarity between the representations of recipes and images. The second part is a bi-directional triplet loss function ${\mathcal{L}}_{rep}$based on the textual embeddings of different recipe components (title-ingredient, title-cooking instruction, ingredient-cooking instruction, etc.), which is used to ensure that the textual information between different recipe components is as similar as possible. These two loss functions are combined together as our final loss function. (16) $\begin{array}{r}\mathcal{L}=\alpha {\mathcal{L}}_{pair}+\beta {\mathcal{L}}_{rep}\end{array}$(16) where $\alpha$ and $\beta$ are the balance factors of the recipe-image pair loss function and the pair loss function of the textual information of different recipe components, $\alpha =1$ and $\beta =1$ mean that the above two loss functions are considered simultaneously.

Recipe-image pair loss function ${\mathcal{L}}_{\mathit{p}\mathit{a}\mathit{i}\mathit{r}}$. The goal of this loss function is to make the similarity between an anchor and the positive samples higher than its similarity to the negative samples. For example, when a food image sample is used as an anchor, we should ensure that the similarity between an image anchor and the positive recipe samples is higher than its similarity to the negative recipe samples, and vice versa when the recipe sample is used as an anchor point. (17) $\begin{array}{r}{\mathcal{L}}_{pair}=max(0,s({\mathit{E}}^I,{\mathit{E}}_n^R)-s({\mathit{E}}^I,{\mathit{E}}_p^R)+m)+max(0,s({\mathit{E}}_n^I,{\mathit{E}}^R)-s({\mathit{E}}_p^I,{\mathit{E}}^R)+m)\end{array}$(17) where $m$ is the margin value in the bi-directional triplet loss function and $s(\cdot )$ is the cosine similarity measurement. ${\mathit{E}}_p^R$ and ${\mathit{E}}_p^I$ are positive samples of recipe and food images respectively, while ${\mathit{E}}_n^R$ and ${\mathit{E}}_n^I$ are negative samples of recipe and food images in small batches respectively.

Loss function ${\mathcal{L}}_{\mathit{r}\mathit{e}\mathit{p}}$ of different recipe components. We attempt to reduce the dependence of model training on recipe-image pairs and make full use of the relationships between the textual information of different recipe components. A self-supervised learning loss function between different pairs of three recipe components (title-ingredient, title-cooking instruction, ingredient-cooking instruction, etc.) (Salvador et al., 2021) is adopted. This loss function is composed of the average of the bi-directional triplet loss functions of different textual embedding pairs of three recipe components. Its objective is similar to that of using bi-directional triple loss functions for recipe-image pairs, which is also used to ensure that the similarity between an anchor point and positive samples is higher than that between them and negative samples. (18) $\begin{array}{r}{\mathcal{L}}_{rep}=\frac{1}{6}(max(0,s(\mathit{a},{\mathit{b}}_n)-s(\mathit{a},{\mathit{b}}_p)+m)+max(0,s({\mathit{a}}_n,\mathit{b})-s({\mathit{a}}_p,\mathit{b})+m))\end{array}$(18) where $\mathit{a},\mathit{b}\in \{{\mathit{e}}_t^{ttl},{\mathit{e}}_g^{ing},{\mathit{e}}_s^{ins}\}$, $\mathit{a}$ and $\mathit{b}$ denote the textual embeddings of any two recipe components (title, ingredient, and cooking instruction), 1/6 denotes a total of six recipe component pairs, and the loss function is averaged by these six combinational relationships. $m$ is the margin value in the triplet loss function. ${\mathit{a}}_p$ and ${\mathit{b}}_p$ are the positive textual embedding samples of two recipe components, while ${\mathit{a}}_n$ and ${\mathit{b}}_n$ are the negative textual embedding samples of two recipe components in a small batch respectively.

4. Experiments

4.1. Dataset and evaluation metrics

The same as the existing studies (Fu et al., 2020; Salvador et al., 2017; Salvador et al., 2021; Wang et al., 2019), we conduct experiments on the standard Recipe 1M dataset (Salvador et al., 2017) for food-oriented cross-modal retrieval. It is a large publicly available dataset on English recipes and food images. The original dataset consists of 1 million recipes and 900,000 food images. This dataset remains 341,421 recipe-image pairs after preprocessing (Salvador et al., 2017). Then all the recipe-image pairs are divided into the training set (238,999), the validation set (51,119) and the test set (51,303). Since this work considers the relationships between the pairs of textual information of different recipe components, 482,231 samples from the rest of this dataset (recipes only, without their corresponding images) are also used to train our proposed method.

The food-oriented cross-modal retrieval task consists of two tasks, i.e. the image-to-recipe retrieval task (Image-to-Recipe) and the recipe-to-image retrieval task (Recipe-to-Image). Ten subsets of 1,000 (1 K) and 10,000 (10 K) recipe-image pairs are extracted from the test dataset to validate our proposed method. Our proposed method and the representative food-oriented cross-modal retrieval baseline methods are evaluated by using Median Rank (MedR), Recall@1 (R1), Recall@5 (R5), and Recall@10 (R10). The results used in our work are the average results of 10 sets of 1 and 10 K test samples.

4.2. Comparison methods

We compare our proposed method with several representative food-oriented cross-modal retrieval methods, which are described as follows.

JE (CVPR2017) (Salvador et al., 2017): Salvador et al. used LSTM and the ResNet-50 model to learn the semantic representations of recipes and images respectively, then used semantic regularisation function to align these modality representations for retrieval learning.

ATTEN (MM2018) (Chen et al., 2018): Chen et al. introduced hierarchical attention networks to capture word-word and sentence-sentence interactions between the textual information of each recipe component for enhanced recipe representations. Then they learned consistent semantic representations of two modalities based on a bidirectional triplet loss function.

AdaMine (SIGIR2018) (Carvalho et al., 2018): Carvalho et al. used Bi-LSTM and the ResNet-50 model to learn recipe and image representations respectively. The category label information, a bi-directional triplet loss and ranking loss are used for the retrieval purpose.

R2GAN (CVPR2019) (Zhu et al., 2019): Zhu et al. used two discriminators in augmented adversarial networks to generate recipe representations and proposed a new two-level ranking loss function to align the semantic representations of images and recipes.

MCEN (CVPR2020) (Fu et al., 2020): Fu et al. firstly extracted recipe and image features based on Bi-GRU and the ResNet-50 model respectively, then used a cross-modal attention mechanism to learn their consistent representations. Finally, they aligned their semantic representations via a hidden variable model.

ACME (CVPR2019) (Wang et al., 2019): Wang et al. proposed a modality alignment method based on adversarial networks and used an improved triplet loss function and cross-modal translation consistency loss function to assist the modality alignment for food-oriented cross-modal retrieval.

HTL (CVPR2021) (Salvador et al., 2021): Salvador et al. encoded the textual information of three recipe components separately based on a hierarchical Transformer encoder and fused them to obtain the final recipe semantic representations. Then the alignment between different recipe components and between image-recipe pairs based on the triplet loss functions are introduced to perform food-oriented cross-modal retrieval.

JEMA (CIKM2021) (Xie et al., 2021): Xie et al. focused on modality alignment and presented a three-tier modality alignment optimisation method for image-recipe retrieval.

IMHF (ICMR2021) (Li et al., 2021): Li et al. learned image and recipe representations through a unified transformer framework with the inter-and intra-modality fusion rather than the separated encoders for food-oriented cross-modal retrieval.

SCAN (TMM2022) (Wang et al., 2022): Wang et al. combined attention mechanism and Bi-LSTM to generate recipe representations and introduced semantic consistency networks to regularise and align the modality semantic representations.

MSJE (TSC2022) (Xie et al., 2022): Xie et al. incorporated TFIDF semantics, sequence semantics, and category semantics to learn enhanced joint embeddings for food-oriented cross-modal retrieval.

LCWF-GI: This is our proposed method that considers the latent component-specific weight factors and global information to learn consistent modality representations for food-oriented cross-modal retrieval. This method uses latent component-specific weight factors to fuse the textual embeddings of three recipe components to generate the final recipe representations. It also considers the global information of images and interactions between different image features to produce enhanced image representations.

4.3. Experimental settings

In the experimental setup, we use the ResNet-50 model (He et al., 2016) (pre-trained on ImageNet (Dosovitskiy et al., 2020)) with initialised weights as an image encoder to extract features of images, and the dimension of the final enhanced image semantic representation is 1024. The output dimension of textual embeddings of all three recipe components is 512 and the output of the final recipe representation generated after fusion is 1024. The Adam optimiser is used to train our proposed method, and the learning rate parameter is set to 0.0001. The batch size is 128. The rank $r$ of latent component-specific weight factors is set to 4 during the process of generating recipe representations. The margin parameter m of the bi-directional triplet loss function is 0.3. The hyperparameters α and β of the final objective loss function of our proposed method are both set to 1.0.

5. Result discussion

5.1. Main retrieval results

Experimental results of our proposed LCWF-GI method and the representative food-oriented cross-modal retrieval baseline methods for the Image-to-Recipe and Recipe-to-Image retrieval tasks are shown in Table 2. The bold values in this table indicate the best results achieved by our LCWF-GI method. The representative baseline methods are divided into the consistent representation learning-based method (focusing on learning high-quality recipe representations) and the modality alignment-based method (aiming at aligning the semantic representations of both recipes and images). As seen from Table 1, several conclusions can be drawn as follows.

1. Overall, the proposed LCWF-GI method achieves more competitive results than all baseline methods except the HTL approach in both two retrieval tasks.

2. In the 1 K test set, the LCWF-GI method shows a similar performance with the HTL method for two retrieval tasks in terms of MedR, Recall@5 and Recall@10. As for the Recall@1 evaluation metric, the LCWF-GI method performs better than all consistent representation learning-based methods (JE, ATTEN, AdaMine, IMHF, MSJE and HTL methods) on two tasks. For example, as for Recall@1, the LCWF-GI method increases from 58.6 to 59.4 and 59.2 to 60.1 compared with the HTL method for Image-to-Recipe and Recipe-to-Image retrieval tasks respectively. This indicates that the enhanced recipe and image semantic representations can effectively improve retrieval performance. However, as for Recall@5, the performance of the LCWF-GI method is slightly weaker than the HTL method on the Image-to-Recipe and Recipe-to-Image tasks. This is due to the fact that the LCWF-GI method cannot fully exploit the relationships between different components to learn its own useful information, leading to inferior performance in recall@5. Compared with the modality alignment-based methods (R2GAN, MCEN, ACME, SCAN and JEMA methods), the LCWF-GI method has more obvious advantages in terms of recall metrics (Recall@1, Recall@5, and Recall@10). Compared with the best modality alignment-based method (JEMA method), the LCWF-GI method improves from 58.1 to 59.4, 85.8 to 86.8, 92.2 to 92.5 on the Image-to-Recipe task for these three recall metrics (Recall@1, Recall@5 and Recall@10) respectively. On the Recipe-to-Image task, the LCWF-GI method increases from 58.5 to 60.1, 86.2 to 86.7, and 92.3 to 92.7 for the same evaluation metrics respectively. Besides, the LCWF-GI method shows significant advantages compared with the second-best modality alignment-based method (SCAN method) on the Image-to-Recipe task and Recipe-to-Image task for these three recall metrics (Recall@1, Recall@5 and Recall@10) respectively. Although modality alignment of the SCAN and JEMA methods can bridge the semantic gap between the recipe and image representations to some degree, enhanced and stronger semantic representations of recipes and images can be more conducive for the retrieval purpose.

3. On the 10 K test set, as the increase of test set size (from 1 K to 10 K), the retrieval performance of the LCWF-GI method and all baseline methods shows a significant decrease compared with their performance on the 1 K test set. This is because a larger test set will make the retrieval more difficult. However, compared to the HTL method, the LCWF-GI method improves from 27.3 to 27.9, 55.2 to 56.0, and 67.0 to 67.8 on Image-to-Recipe retrieval task in terms of Recall@1, Recall@5, and Recall@10 respectively. For the Recipe-to-Image retrieval task, the values of Recall@1, Recall@5, and Recall@10 of our LCWF-GI method increase from 27.9 to 28.6, 55.3 to 55.8, 66.9 to 67.9 respectively. The LCWF-GI method outperforms the HTL method by a margin improvement on both retrieval tasks. There are two possible reasons: firstly, the LCWF-GI method generates the final recipe semantic representation by fusing the textual embeddings of different recipe components via the latent component weight factor, which can capture the complex interactions and complementary information between the textual information of different components. Secondly, the LCWF-GI method introduces the Transformer encoder to re-encode image feature information from the ResNet-50 model for enhanced image semantic representations. These refined image representations consider global information of images and capture the interactions between different extracted local image features. Compared with the modality alignment-based JEMA and SCAN methods, the proposed LCWF-GI method not only achieves better scores in terms of the MedR evaluation metric, but also still has good performance on three metrics of Recall@1, Recall@5, and Recall@10 for the Image-to-Recipe and Recipe-to-Image tasks. The performance of the LCWF-GI method is significantly better than that of the modal alignment-based methods, which further verifies that the recipe and image representations with high quality can better bridge the semantic gap between the two modalities.

Table 2. The comparison results of our proposed method and popular baseline methods. In this table, bold values represent the best scores.

			Image-to-Recipe				Recipe-to-Image
Size		Method	MedR↓	R1↑	R5↑	R10↑	MedR↓	R1↑	R5↑	R10↑
1K	consistent representation learning-based method	JE	5.2	24.0	51.0	65.0	5.1	25.0	52.0	65.0
		ATTEN	4.6	25.6	53.7	66.9	4.6	25.7	53.9	67.1
		AdaMine	2.0	39.8	69.0	77.4	1.0	40.2	68.1	78.7
		IMHF	1.0	53.2	80.7	87.6	1.0	54.1	82.4	88.2
		MSJE	1.0	56.8	84.7	90.9	1.0	56.2	84.9	91.1
		HTL	1.0	58.6	87.1	92.5	1.0	59.2	86.8	92.5
	modality alignment-based method	R2GAN	2.0	39.1	71.0	81.7	2.0	40.6	72.6	83.3
		MCEN	2.0	48.2	75.8	83.6	1.9	48.4	76.1	83.7
		ACME	1.0	51.8	80.2	87.5	1.0	52.8	80.2	87.6
		SCAN	1.0	54.0	81.7	88.8	1.0	54.9	81.9	89.0
		JEMA	1.0	58.1	85.8	92.2	1.0	58.5	86.2	92.3
		LCWF-GI	1.0	59.4	86.8	92.5	1.0	60.1	86.7	92.7
10K	consistent representation learning-based method	JE	41.9	-	-	-	39.2	-	-	-
		ATTEN	39.8	7.2	19.2	27.6	38.1	7.0	19.4	27.8
		AdaMine	13.2	14.9	35.3	45.2	12.2	14.8	34.6	46.1
		IMHF	6.2	23.4	48.2	58.4	5.8	24..9	48.3	59.4
		MSJE	5.0	25.6	52.1	63.8	5.0	26.2	52.5	64.1
		HTL	4.0	27.3	55.2	67.0	4.0	27.9	55.3	66.9
	modality alignment-based method	R2GAN	13.9	13.5	33.5	44.9	12.6	14.2	35.0	46.8
		MCEN	7.2	20.3	43.3	54.4	6.6	21.4	44.3	55.2
		ACME	6.7	22.9	46.8	57.9	6.0	24.4	47.9	59.0
		SCAN	5.9	23.7	49.3	60.6	5.1	25.3	50.6	61.6
		JEMA	4.2	26.9	54.0	65.6	4.0	27.2	54.4	66.1
		LCWF-GI	4.0	27.9	56.0	67.8	4.0	28.6	55.8	67.5

5.2. Ablation studies

To explore the contributions of textual information of different recipe components (title, ingredient and cooking instruction) in our LCWF-GI proposed method, we conduct ablation studies on 1 and 10 K test sets. Firstly, the textual information of the title and ingredient components is fused by latent weight factors to generate the recipe representation (denoted as TL + IG). The textual information of the title and cooking instruction components (denoted as TL + IS), and the textual information of ingredient and cooking instruction components (denoted as IG + IS), are fused in a similar way to produce the final recipe representations. The proposed LCWF-GI method fuses the textual information of all three recipe components, including title, ingredient and cooking instruction (denoted as TL + IG + IS). Experimental results are reported in Table 3. The following conclusions can be drawn from this table: (1) The recipe semantic representations generated by fusing textual information of all three recipe components (TL + IG + IS) can achieve better retrieval performance than only fusing textual information of two recipe components. (2) Compared with TL + IG and TL + IS, the fusion of two recipe components, IG + IS is more conducive to generating recipe representations with high quality. This is because the textual information of the ingredient and cooking instruction components contains multiple sentences, while the textual information of the title component only contains one sentence. Thus, the textual information of ingredient and cooking instruction components can contain more useful information than the title component with a single sentence.

Table 3. The food-oriented cross-modal retrieval results of this proposed method fusing different recipe components. In this table, bold values represent the best scores.

		Image-to-Recipe				Recipe-to-Image
		MedR↓	R1↑	R5↑	R10↑	MedR↓	R1↑	R5↑	R10↑
1K	TL + IG	1.0	57.1	85.3	91.6	1.0	56.7	84.7	91.2
	TL + IS	2.0	47.8	79.6	87.9	1.9	48.1	79.2	88.0
	IG + IS	1.0	57.5	85.5	91.7	1.0	57.1	84.9	91.6
	TL + IG + IS	1.0	59.4	86.8	92.5	1.0	60.1	86.7	92.7
10K	TL + IG	5.0	25.5	52.4	64.3	5.0	25.5	52.4	64.2
	TL + IS	8.2	18.1	42.1	54.4	8.3	18.7	42.1	54.2
	IG + IS	5.0	25.6	52.8	64.8	5.0	25.9	52.7	64.5
	TL + IG + IS	4.0	27.9	56.0	67.8	4.0	28.6	55.8	67.5

5.3. Parameter settings

We also explore the effect of the latent weight factor with different rank values in the LCWF-GI method. Since the results of Recall@1 are more obvious than those of Recall@5 and Recall@10, only the value of Recall@1 is chosen for the corresponding demonstration. Figure 2 shows the performance of our proposed LCWF-GI method on the 1 and 10 K test sets for two retrieval tasks when the latent weight factor is set to different rank values. From the experimental results, it is clear that our proposed method performs best when the rank value of the latent weight factor is 4.

Figure 2. The retrieval results of latent weight factor with different rank values in our proposed LCWF-GI method on Image-to-Recipe and Recipe-to-Image tasks on 1 and 10 K test set. (a) The Value of Recall@1 of Our LCWF-GI method for Image-to-Recipe and Recipe-to-Image Retrieval Tasks on 1 K Test Set; (b) The Value of Recall@1 of Our LCWF-GI method for Image-to-Recipe and Recipe-to-Image Retrieval Tasks on 10 K Test Set.

5.4. Visualisation example

Figure 3 shows an example of food-oriented cross-modal retrieval results on a 10 K test set using our proposed LCWF-GI method and the representative baseline method (the HTL method). To save space, only the Recipe-to-Image retrieval task is visualised in this paper as an example. In this paper, two different recipes (Honey Lemon Fruit Salad and Blue Ribbon Apple Crumb Pie) are selected as query terms, and we use our proposed method and the HTL method to retrieve the relevant food images. The top five retrieval results (i.e. the top five food images) are returned. Compared to the real images, the visualisation example shows that our proposed method can retrieve 4-5 relevant results for recipes 1 and 2 respectively, while the HTL method can only return 3-4 relevant results. Therefore, our proposed method can retrieve food images related to recipes more effectively than the HTL method. It indicates that our proposed method can better learn the consistent semantic representations of recipes and images and use them to bridge the semantic gap between these two modalities.

Figure 3. The example of Recipe-to-Image retrieval of our proposed method and HTL baseline method on 10 K test set. The first column represents that the query is a piece of recipe text. The second column is the ground truth of the recipe text. The final column is the top 5 retrieval result.

5.5. Complexity analysis

In our work, we use the parameter count and the training time of one epoch to evaluate the model complexity, because recording the whole training process will spend too much time. The information about parameter counts and the training time of each epoch is shown in Table 4. From this table, we can see that this proposed method contains about 100.25M (million) parameters, while the HTL method includes about 93.95M (million) parameters. Compared with the best baseline method (the HTL method), the number of parameters in our proposed method is a bit more than that of the HTL method. However, the training time of each epoch is less than that of the HTL method. Furthermore, in our experiments, we observe that only using fewer epochs can get a similar performance in comparison with the HTL method. The results (to save time and space, we only conduct experiments on 1 K test set) are shown in Figure 4. We can see that our proposed LCWF-GI method can achieve better performance or similar performance in comparison with the HTL method, but our LCWF-GI method only needs much fewer training epochs. It indicates that our LCWF-GI n method can save training time.

Figure 4. The results achieved by our LCWF-GI method and the best baseline method (HTL) on two retrieval tasks on 1 K test set when using the different training epochs. The solid curve represents our LCWF-GI method and the dashed curve means the HTL method. The solid and dashed curves with orange, green and blue colours represent the metrics of Recall@1, Recall@5 and Recall@10 respectively.

Table 4. The parameter counting and the time of each epoch of our LCWF-GI method and the best baseline method (HTL). M and s represent million and second.

Method	Parameter Count (Million)	Time (second)
HTL	93.95M	7,106s
LCWF-GI	100.25M	6,977s

5.6. Attention visualization in transformer

We also visualise the self-attention mechanism in Transformer to further understand it learned in recipe representation learning. As we mentioned before, we use three Transformer encoders with the same structure and different parameters to encode the textual information of three recipe components (title, ingredient and cooking instruction) respectively. Therefore, the principle of attention mechanism for these three recipe components is similar. In our work, we take the attention mechanism in Transformer for the cooking instruction component as an example. The attention visualisation is shown in Figure 5. There are two sentences “cook chicken on grill for about 30 min” and “turning it once brush chicken with sauce” in the cooking instruction component. The blue colour represents positive and the orange colour represents negative. With the deeper blue colour of associated words on the right, the chosen word in the left sentence is more closely associated with it. Instead, the deeper orange colour of associated words on the right, the chosen word in the left sentence is not associated with it. It can observe that our attention mechanism not only can capture the important words in the cooking instruction component, but also catch the long-distance dependency relationships between words of different sentences in the cooking instruction component.

Figure 5. The visualisation of attention mechanism in transformer for the cooking instruction component. (a) The self-attention visualisation of a sentence in the cooking instruction component; (b) The self-attention visualisation of sentence a and the next sentence in the cooking instruction component.

6. Conclusion

In this paper, we propose a novel method based on latent weight factors and global information to learn consistent modality representations for food-oriented cross-modal retrieval. Compared with the existing retrieval methods, this proposed method first extracts the hidden textual information from three recipe components (title, ingredient and cooking instruction) with their corresponding Transformer encoders. Then we fuse the textual information of these multiple components by using the latent weight factors and tensor representation to generate richer recipe semantic representations. The final recipe representation can be represented as the element-wise product over the textual embeddings of three recipe components and their corresponding latent weight factors. Compared with the direct and simple embedding concatenation method (which can only capture simple interactions between different components), our proposed method can allow the interactions between individual elements in the textual embeddings of different components through the element-wise product operation. This can capture the complex interactions between the textual information of these multiple components. In addition, we introduce the Transformer encoder to re-encode the image features from the ResNet-50 model for learning the enhanced image semantic representations. The newly generated image representations take the global information of images into consideration and leverage the self-attention mechanism in the Transformer encoder to capture the interactions between different extracted local image features. Several experiments are conducted on the standard Recipe 1M dataset. Experimental results show that our proposed method can obtain better retrieval results compared with the existing representative baseline methods. In future work, we not only can further consider the hierarchical alignment between textual information of different components, but also consider new retrieval learning objective functions.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was supported in part by the Hunan Provincial Natural Science Foundation of China [grant no 2022JJ30020], the Guangdong Basic and Applied Basic Research Foundation of China [grant no 2023A1515012718], the Philosophy and Social Sciences 14th Five-Year Plan Project of Guangdong Province [grant no GD23CTS03], the Scientific Research Fund of Hunan Provincial Education Department [grant no 21A0319], and the Hunan Provincial Innovation Foundation for Postgraduate [grant no CX20210986].