How COVID-19 impacts telehealth: an empirical study of telehealth services, users and the use of metaverse

Abstract

Since the outbreak of the coronavirus 2019 (COVID-19) pandemic, telehealth services are regarded as a good approach to keep health workers and patients safe while simultaneously managing available resources. In this paper, we discuss the impact that COVID-19 has on telehealth services and on telehealth users' opinion of the service. We collected 245 Android telehealth apps, 144 iOS telehealth apps and 86 telehealth websites, and performed a systematic analysis on this dataset. In this analysis, we conducted a comparison analysis and relevant content analysis of the telehealth apps as well as their security risks. Apart from the mobile platforms, we also inspected the telehealth websites' features, particularly those related to the use of metaverse to improve current telehealth solutions. To further understand people's attitude towards telehealth services, we invited users to participate in a user study aimed at revealing what impact COVID-19 has on users' willingness to adopt telehealth services and revealing the gap between the telehealth service and its users. Our result shows that 27.1% new iOS apps and 27.4% new Android apps were released after the COVID-19 announcement, and a surge of updates were noted within 4 weeks after the COVID-19 announcement. We further found that COVID-19 is frequently mentioned in telehealth app reviews in the second and third quarter of 2020, and the most mentioned aspects related to COVID-19 include family, test result and vaccine. According to our user study, COVID-19 has a significant impact on the selection of telehealth services, especially for female participants, people aged 46–55, and students. The investigation also finds out that the use of metaverse will significantly improves the effectiveness of traditional telehealth solutions.

KEYWORDS:

• Telehealth
• COVID-19
• human–computer interaction

1. Introduction

On 30 January 2020, the World Health Organisation (WHO) triggered its highest global emergency alert by declaring COVID-19 a Public Health Emergency of International Concern. It declared the coronavirus disease (COVID-19) outbreak a pandemic on 12 March 2020. Around the world, governments announced lockdowns aimed at managing and eliminating community transmission of the coronavirus (World Health Organisation, 2021). Although lockdowns allow for people to leave their places of residence to receive or provide medical care, many people opted to remain within their homes to avoid any contact with the public, and therefore minimising their risk to contract the coronavirus. During these periods of lockdown, telehealth services provided a practical solution to overcome safety concerns and physical barriers to provide patients and caregivers with access to appropriate medical care. In our study, we aim to investigate the impact that COVID-19 has had on telehealth service and users' opinion of these services after the outbreak.

Apart from general medical service, telehealth allows patients to receive treatment remotely, a big consideration during times when coronavirus community transmission is a big concern amongst users. We consider the definition for telehealth as cited by the International Organisation for Standardisation as the foundation for our study: “the use of telecommunication techniques for the purpose of providing telemedicine, medical education and health education over a distance”(Telehealth, 2021). As such, these telehealth services also help users to obtain instant information related to COVID-19, including COVID-19 health and safety guidance, vaccination updates and advisory messaging. Telehealth services further provide a reasonable alternative for patients in mandated COVID-19 quarantine or isolation, providing them with appropriate medical care and intervention during a vulnerable time, whilst remaining separated from the rest of the community and limiting possible exposure to and spread of the coronavirus.

The impact of COVID-19 is an emerging topic and evolving at a rapid and unprecedented rate due to varying infection rates and medical management strategies across the world. As such, COVID-19 has become a trend topic in various research fields (Ankenbauer & Lu, 2020; D'Ignazio et al., 2020; Ferrag et al., 2021; Ohata & Bezerra, 2021; Venigalla et al., 2020) and the user perspective related to this, including users' willingness to adopt telehealth services as a remediating service in lieu of in-person medical care, are increasingly being investigated in recent works (Gentry et al., 2020; Lee et al., 2020; J. Li et al., 2014; L. A. Lin et al., 2020; Triana et al., 2020). However, to date very little research has been done to investigate the impact that COVID-19 has on the telehealth services and its users. In addition, limited focus has been placed on identifying whether current telehealth services are able to satisfy patients' needs in the context of COVID-19. To address this gap in the research, we systematically analysed telehealth services provided for use on different platforms (e.g. telehealth websites and telehealth apps), user review inspection and topic modelling analysis, and the comparison analysis of the telehealth app neighbouring versions. We further designed a user study based on the results obtained from the systematic analysis, comprising two questionnaires to identify how COVID-19 has influenced users' opinions towards the telehealth service.

Our research results show that nearly one-third of current Android and iOS telehealth apps were released between the COVID-19 announcement and the first quarter of 2021. In addition, there was a surge of updates released to apps right after the announcement and apps on both platforms are found to be updated more frequently than before the COVID-10 announcement. From our analysis, we find that most telehealth platforms were already equipped with core telehealth functionalities before the announcement of COVID-19. After the COVID-19 announcement, their focus generally shifted more towards delivering services related to COVID-19, enhancing the security and optimising marketing strategies. From our user study, we find that the distribution of telehealth users is diverse in terms of gender, age and background. Most telehealth services still heavily rely on online communication platforms or the traditional phone call.

In this paper, we also investigated the use of Metaverse (2023) into the health sector. In recent years, the metaverse has garnered significant attention due to its capacity to craft immersive virtual realms designed for collaboration, interaction and unique experiences (Wang, 2022a, 2022b). Presently, professionals are proposing that the metaverse might bring about a groundbreaking transformation in the delivery of healthcare, particularly in terms of improving remote care through telehealth. Through the provision of captivating 3D environments and tools that enable healthcare providers and patients to engage virtually by using avatars, the metaverse offers the potential to address numerous shortcomings inherent in existing telehealth services (Medium Report, 2023). We searched for the related information from web and other technical or academic sources. The analysis results suggested that COVID-19 indeed prompts the development of technical innovations such as metaverse on health sector. These innovations are supposed to improve the effectiveness of traditional telehealth solutions. We summarise the major contributions of our study as follows:

• We systematically analyse telehealth service based on different sources. Our analysis includes the inspection of the information provided from different platforms (e.g. telehealth websites and app stores) and a comparison analysis of the telehealth app neighbouring versions.

• We conduct a user study that contains two questionnaires. This study shows how COVID-19 has influenced users' willingness to adopt telehealth service and reveals the gap between the telehealth service and users.

• We also conduct an investigation on metaverse that has been applied to health sector. The results suggested that this new technical innovation will improve the effectiveness of traditional telehealth solutions.

The remainder of this paper is organised as follows. We first present the detailed design of our study in Section 2. We also list research questions in this section. In Section 3, we elaborate and analyse the results. We then investigate metaverse for telehealth in Section 4. In Section 5, we discuss the implication of our study, followed by the related work in Section 6 and the conclusion in Section 7.

2. Methodology

To investigate the changes to telehealth service platforms taking place after the announcement of the COVID-19 pandemic in 2020, we perform data analysis associated with telehealth services and invite participants to take part in our user study. In this section, we introduce the study's research questions and discuss the methodology applied for collecting and processing the data. We further introduce the design of our user study.

2.1. Research questions

Our exploratory research focuses on the impact that the COVID-19 pandemic has had on the attitude of patients in terms of telehealth services. To assess the impact that COVID-19 has had on the telehealth services and the attitude of those making use of those services, we aim to address three main research questions:

• RQ1: How does telehealth service change before and after the outbreak of COVID-19?

  Motivation: COVID-19 has changed many aspects of our world. As the healthcare services are encouraged during the quarantine, we would like to investigate the changes have been made specifically to the telehealth services after the COVID-19 outbreak. Understanding the changes could provide insightful findings to identify and address the challenges in online telehealth platforms.

• RQ2: How does the COVID-19 outbreak change the way in which users review telehealth platforms from the app stores?

  Motivation: Telehealth was born long before the outbreak of COVID-19 when traditional treatments remain easy access. We would like to know if COVID-19 has transformed people's behaviour or view towards telehealth. The analysis of the top trends and issues from users' reviews could inform future app refinement.

• RQ3: Who are the telehealth users, and what are the telehealth users' choices under the context of COVID-19?

  Motivation: Apart from how people have changed their view towards telehealth, we also would like to analyse the telehealth users in depth from other aspects, which includes the demographics information of the telehealth users, the reason why they start to use the service, the platforms they are more likely to receive the service and how do they rate their telehealth experience.

2.2. Telehealth app and websites collection

In this section, we detail how we define the scope of our analysed data and how do we collect the data.

2.2.1. Preliminary study on data collection

Before the data collection, we conducted a preliminary study to define the scope of the data analysis. In our preliminary study, we invite and ask our participants from three aspects: (1) if they have the experience of using a telehealth service, (2) did they start to use the service before or after the outbreak of the COVID-19 and (3) from what platforms they receive their telehealth service. The whole study takes a week to complete. In total, we have received 923 responses. After analysing the result, we are able to obtain a general view of the telehealth users.

Based on the results obtained from our preliminary study, apart from consulting via phone calls and text messages, our participants mainly receive telehealth services from telehealth websites and telehealth apps installed on their mobile devices. Therefore, to analyse the telehealth services, we target telehealth mobile apps and websites. Telehealth apps are collected from the official app stores of two major mobile Operating Systems (i.e. Google Play for Android and App Store for iOS), and the websites that are authorised to provide telehealth services. We only consider telehealth apps and websites that offer remote health consultation or treatment. Consultation and treatment can be completed through any form of remote communication such as text messaging and online call. We manually filter out the apps developed by the medical sector that do not provide consultation and treatment through the platform. For example, the platform allows users to make an appointment online, but the actual consultation and treatment must occur offline.

We introduce the data sources in Figure 1 and detail the collection process for mobile apps and websites in Sections 2.2.2 and 2.2.3, respectively.

Figure 1. Telehealth apps and websites collection.

2.2.2. Telehealth apps

We plan to study the changes in the telehealth apps from two aspects: (1) the evolution of the app's user interface (UI) and functionality and (2) the changes in the app's metadata (e.g. release notes, user reviews). Therefore, we accordingly collect both the app's executable file and metadata.

We used Google Play and App Store as the main sources for our data collection. Google Play is the official Android app Market and a digital distribution service operated and developed by Google. App Store is a digital distribution platform developed and maintained by Apple, providing mobile apps to iOS devices. Android and iOS users can browse and download apps directly from these two official stores, respectively.

We first search the app stores using specific keywords, “telehealth”, “telemedicine”, “e-health ” and “telecare”, to obtain a list of candidate apps for our study. Once we have a complete telehealth app list, we download the apps' executable files as well as retrieve the apps' metadata. Since iOS is a close-source system and there are no off-the-shelf tools for reverse engineering iOS apps, our executable file analysis is only conducted on Android apps. Nevertheless, as the apps on Android and iOS usually provide identical functions, we believe that excluding iOS apps for code analysis may not significantly affect our results. To observe the impact of COVID-19 on app updates, we must also collect the historical versions of the apps' executable files (i.e. Android APK). Unfortunately, the official app store (i.e. Google Play) does not archive apps' historical versions. We hence resort to a third-party app repository, namely APK.Support, ¹ to download the historical APK versions.

As for the metadata provided on both app stores, for each telehealth app, we collect the following information: (1) release note, which maintains the changelog of historical versions since the app's initial release, (2) app description, which introduces and explains the functions and details of the app, and (3) the user reviews the app has received so far. All metadata is collected from the official app stores (i.e. Google Play and App Store), except the release notes for Android versions since Google Play does not maintain the release notes. We therefore obtain the Android app release notes from a third-party mobile app analytics platform App Annie. ² It should also be noted that since Apple's App Store website has anti-crawler protection that forbid us from crawling the user reviews directly from the website, we instead use the App Store API to retrieve the reviews, which do not contain the timestamps of the reviews.

2.2.3. Telehealth website information

Telehealth website is another popular platform where people can receive telehealth services. Therefore, our study also includes websites that provide telehealth services. We collect telehealth websites from two sources. Similar to the selection criteria of telehealth apps, we only include websites that offer remote consultation or treatment. On the one hand, for each telehealth app identified in our previous step, we retrieve the URL of the official website from the app description. We then carefully inspect the websites and remove the ones that do not provide remote consultation or treatment services. Even though we already take the app version into consideration, its telehealth website usually provides additional information and the services provided on websites and apps can be slightly different. On the other hand, we search intensively using the keywords “telehealth”, “telemedcine”, “e-health” and “telecare” from Google and manually record other telehealth websites that can deliver their service through a website.

2.3. User study

Based on preliminary results obtained from the data analysis, we design two separate surveys. The first survey does not have specific requirements for participation, it helps us to gather the information of both telehealth users and non-telehealth users. The second survey is focused on inspecting the experience and opinion shared by the telehealth users. The participants from our second survey are selected from our first survey who have used telehealth service. Both surveys are conducted on Amazon Mechanical Turk (MTurk) (What is Amazon Mechanical Turk?, n.d.). We recruit our participants from Amazon's Mechanical Turk (MTurk), where it has surged in popularity in experimental and survey-based social science research (Berinsky et al., 2012; Hu et al., 2022).

The first survey takes approximately 2 minutes to complete, whilst the second survey takes between 10 and 15 minutes. The first survey is aimed at identifying which participants have previously used a telehealth service. If participants indicate that they have used a telehealth app previously, they are asked to provide relevant evidence such as the name of the telehealth apps they use and the approximate date when they started using the telehealth service. After carefully validating and obtaining reliable results from the first survey, we invite the participants from the first survey who have experience in using telehealth services to complete the second survey. The second survey covers participants' experience of using the telehealth service before and after the outbreak of COVID-19 and some scenario analysis related to telehealth service.

Before answering the survey questions, participants will be given a participant information sheet and agree to participate by clicking the accept button on the consent form page on MTurk. In our study, we did not collect any personal identifiable information, such as their full name and date of birth. We also anonymised the data by removing their worker IDs during the analysis step. Participants who agree to participate will complete one to two surveys, which are related to the topic of the telehealth service in the context of COVID-19.

In the pilot study, we ran one round for the first survey with 923 participants. For the second survey where only people who have used telehealth are invited as participants, we run 3 rounds in total, each with 10 telehealth platform users involved. For every round conduct in our pilot study, we ask our participants to complete our design survey. After we gather participants' feedback after each round, we identify several potential problem areas, evaluate the feasibility, duration and cost. Relevant improvements are made prior to conduct the full-scale research project. Besides, most of the survey questions are first designed as the open questions in our pilot study. After we gather different possible answers from our participants, we change the open questions to multiple choice questions. In this way, our survey is less likely to overlook some possible answers, and participants can also complete the survey in a shorter period of time when our surveys are officially released. After we complete our pilot study, we publish the final versions and invite people to take part in our survey.

2.4. Data analysis

After the data is collected, we perform in-depth semi-systematic data analysis, including APK static analysis and the telehealth platform information analysis. The overview of our data processing process is displayed in Figure 2.

Figure 2. Data processing.

2.4.1. App similarity analysis

We compare the difference between history versions of an app from two perspectives: the permission used and the code implementation. Recall that due to the inaccessibility of the source code of iOS app, the app similarity analysis only includes Android apps.

Android app permission system is a mechanism to protect the privacy of app users (Liang, Li, et al., 2023; Liang, Yang, et al., 2023; S. Zhang et al., 2023). Apps must declare the permissions in the manifest file before accessing sensitive user data (such as personal identity information and health data), as well as certain system features (such as the camera and sensors). Telehealth apps usually offer functionality that requires access to restricted data or restricted actions, such as requesting accessing the camera and microphone when making video calls. Some apps are equipped with more advanced features, such as connecting with external IoT devices through Bluetooth, and accessing health or biometric information. The changes in required permissions reflect the change of app behaviours related to sensitive information. Therefore, we extract the list of permissions required by each app version and compare the permission lists between neighbouring app versions and further discuss the changes being made to the used permissions.

We investigate the similarity of apps in neighbouring versions from the method, component and resource levels. Android components are the core building blocks of an Android app, which have well-defined life cycles. The components include Activity (i.e. a User Interface with back-end class to handle the action performed on User Interface), Service (i.e. a back-end class with no User Interface), Broadcast Receiver (i.e. a component that receives and handles broadcast intents from the Android system or other apps) and Content Provider (i.e. a component stores shared data of an app that other apps can access). Android methods are the code blocks to perform certain behaviours and will be executed when it is invoked. Resources are the non-code assets that can be accessed by the app code, such as images. Comparing the method and component level similarities directly reflect the added/removed functionalities between the two versions, while comparing the resource level similarity can, to some extent, show the changes in the user interface. We leverage SimiDroid (L. Li et al., 2017) to extract the differences of methods, components and resources between every two neighbouring versions. SimiDroid extracts the necessary features from two apps and then generates a similarity analysis report based on the desired levels (i.e. method, component and resource). It can assist APK analysts to mine changes and discover the similarities and dissimilarities between apps.

2.4.2. Telehealth information from platforms

There are three data resources of the information obtained from different platforms: (1) app release notes, (2) app reviews, and (3) content from telehealth website and app stores. The content includes the description of the provided service, COVID-19 related news and safety guidance.

Release notes: We first analyse the release notes of the telehealth apps by extracting the version release dates for each app. The analysis of the release notes focuses on (1) the initial release dates of the telehealth apps, (2) update frequency before and after the announcement of the COVID-19 and (3) the update information in the release note. We use the date that the World Health Organisation (WHO) declared COVID-19 a pandemic (i.e. 11 March 2020) when referring to the situation before and after COVID-19. When calculating the update frequency after the COVID-19 announcement, we use the app's latest release date rather than the current date.

App review: We further analyse the user reviews of the telehealth apps. We extract the date and content of each review. The analysis of the app reviews mainly focuses on (1) the number of reviews left in each quarter, (2) review topic analysis based on quarters, (3) the analysis of the reviews related to COVID-19 and (4) analysis of the reviews related to special groups (e.g. elders, people with disabilities, children, etc.). Topic modelling is a statistical method for discovering the abstract “topics” that appear in a collection of documents. We use Latent Dirichlet Allocation (LDA) (Blei et al., 2003), a state-of-the-art method for topic modelling, to analyse topics of COVID-related reviews and topic trends of the set of app reviews. We implement LDA in Python using gensim library (Řehůřek & Sojka, 2010). We use Coherence measures to evaluate the performance of the generated topic models as it has better human interpretability than other measures, such as perplexity (Röder et al., 2015). We employ the CoherenceModel in the gensim library to generate LDA models along with their topic coherence. To determine optimal number of topics, we compare the coherence of generated models with different values from 1 to 15 in steps of 1. Specifically, we extract COVID-related reviews from the entire set of reviews by using a list of keywords, i.e. “covid”, “pandemic”, “corona”, “coronavirus”, “virus”, “quarantine”. These keywords are selected based on the careful inspections of the app reviews. We apply LDA following the above setup to generate the topics for the COVID-related reviews, where each review is represented as a set of topic probabilities, and each topic is a list of words along with computed weights. To further analyse the topic trends of the overall reviews, we group the reviews by quarters and concatenate the texts as a new document. We run LDA again to identify the topics for the documents and compare the trends of different topics over time.

Telehealth information content: Finally, we analyse the content from the telehealth websites and the app description. These types of information can help us better understand the features of the telehealth service. Unlike release notes and user reviews that provide information with timestamps, telehealth website information and app description provided by app stores in the past can hardly be retrieved, therefore, we mainly focus on content analysis. We create two criteria to evaluate the telehealth websites and apps, respectively. The criteria can help understand the telehealth service and also evaluate the telehealth service based on the aspects related to our study, which includes COVID-19 functionalities, symptoms can be treated through telehealth service, family member management, etc. The detailed criteria can be found in Table 5. The criteria are designed based on a careful inspection of the telehealth platforms. Three of our authors then manually inspect the telehealth websites and the app descriptions provided by both app stores to check if the studied platforms fulfil the criteria.

3. Results

In this section, we demonstrate the results obtained from our data analysis and user study.

3.1. RQ1: How does telehealth service change before and after the outbreak of COVID-19?

In this section, we demonstrate what are the changes made by telehealth platforms after the outbreak of COVID-19. We collect 245 android apps, 144 iOS apps and 86 websites that offer telehealth services. We discuss the changes from the (1) analysis of the app release notes and (2) comparison of app neighbouring versions.

3.1.1. App release notes

Due to the fact that the release notes of some apps are not available, we collect 144 and 219 release notes of the iOS and Android telehealth apps respectively. WHO announces the outbreak a Public Health Emergency of International Concern on 30 January 2020 (World Health Organisation, 2021). We identify the date of outbreak to further investigate the impact of COVID-19 on telehealth platforms from releasing notes. More specifically, we aim to answer the following three questions: (1) How many telehealth apps are born after the COVID-19 outbreak? (2) Do telehealth apps update more frequently after the COVID-19 outbreak? (3) What are the changes mentioned in release notes after the COVID-19 outbreak?

To answer the first question, we find that 39 (27.1%) out of 144 iOS apps and 60 (27.4%) out of 219 Android apps are initially released after the announcement. It means more than 1/4 of the telehealth apps from both platforms are created after the COVID-19 announcement.

As for the second question, based on our results, before the COVID-19 outbreak, iOS telehealth apps release their new versions every 72 days on average. The update frequency slightly decreases after the announcement, which is 73 days on average. On Android platform, the update cycle reduces from 75 days to 70 days on average. Telehealth apps from Android platform update more frequently after the COVID-19 announcement compared to iOS. To analyse the change of the update frequency regarding the number of apps, we extract the telehealth apps that have both release record before and after the announcement, and compare their update frequency. Among these apps, 40 (37.0%) iOS apps release their new app versions more frequently after the outbreak among the 108 iOS apps. 59 (39.6%) Android apps releases new app versions more frequently before the announcement among 149 Android telehealth apps. We count the number of released new app versions in each month, as shown in Figure 3. It shows a significant increase in the number of telehealth apps deployed on both iOS and Android platforms in the year of 2020. Besides, a larger amount of version updates are observed in the following 3 months after the COVID-19 announcement made by WHO.

Figure 3. App release number.

We further look into the content of app release notes and collect the update information for apps from both app stores to answer the third question. Most of the release notes do not have a description or just simply describe the updates as “Bug fixes and performance improvement ”. We select the information that are related to our study, and share some interesting updates which are made to the telehealth apps. Most of the apps have made general improvements and enhancements before and after the outbreak such as the quality of the audio and video calls, instant message, network performance and online appointment arrangement. The most obvious new topic aroused from the release notes after the announcement is COVID-19 and its relevant content. Based on our analysis result, 26 release notes from both Android and iOS mention their updated information and functionalities related to COVID-19, which include:

• New or improved assessment and screening tool for COVID-19

• Advisory messaging for COVID-19

• Personal COVID-19 testing status management and COVID-19 recovery monitoring

• COVID-19 exposure notifications

• COVID-19 content and guidance from Government sources, articles and resource documents

• Hundreds of new, patient-asked, doctor-answered questions about COVID-19

• COVID-19 testing locations update

• COVID-19 Vaccination status management, and new resources and information about COVID-19 Vaccination.

As we can see from the release notes, telehealth apps have taken serious steps to assist user under the certain difficult time. Not only users can obtain relevant COVID-19 information and resource from the apps but also the patients with COVID-19 or they think they might be infected can receive remote assessment and the provision of care.

Apart from the general improvements and enhancements of the telehealth functions, we also explore on some other aspects mentioned in the release notes. Among the rest of the aspects, security enhancement is mentioned most in the release notes both before and after the announcement (Chen et al., 2019). The security improvements include password management, secure video/audio call and message, personal information protection, etc. We also find apps start to utilise biometric authentication (e.g. fingerprint) to protect user's information before the COVID-19 outbreak. Apps also start to care about special groups such as elders and young children. Users are allowed to add their kids or dependants to their telehealth account, invite the trusted persons to the video call and share their medical results directly to the trusted persons. We also note that telehealth apps are more aware of the importance to connect with users' insurance information. More apps allow users to add their insurance information to link their insurance account. As the patients receive the service remotely, doctors usually need to diagnose the symptoms through attached photos or from some documents. More apps have enabled large file transmission through video call or chat and support high quality of the video call.

27% new iOS apps and 27.4% new Android apps are released after the COVID-19 announcement. There is a surge of updated apps released right after the announcement, and the apps on both platforms update more frequently after the COVID-19 outbreak.

3.1.2. App version comparison

Release notes give us a general idea from user's view of how telehealth apps have updated. We can clearly identify the updates being made overtime. However, the release notes hardly provide the details of the update information, making it difficult to obtain reliable statistics as based on a larger amount of app data. Therefore, we analyse the 248 comparison results obtained from the Android APK comparison based on four different aspects: (1) permission, (2) component, (3) method and (4) resource.

Permission: We first separate the apps based on whether the apps are released before or after the announcement and collect their used permissions. Then, for each time period, we check if a certain permission is used by a majority ($>50\mathrm{\%}$) of the apps released during the period. The result is given in Table 1. Apart from the overall permission usage distribution, we also look into the updated permission between the neighbouring app releases. Only some slight changes between neighbouring versions are observed. Top 10 newly added permissions can be found in Table 2, and the list of top 10 added permissions can be categorised into 3 groups: (1) access to location, (2) security enhancement and (3) performance improvement.

Table 1. Most used permissions before and after the announcement of the COVID-19 outbreak.

	Most frequent used permissions
Rank	Before announcement	After announcement
1	android.permission. INTERNET	android.permission. INTERNET
2	android.permission. WRITE_EXTERNAL_STORAGE	android.permission. ACCESS_NETWORK_STATE
3	android.permission. WAKE_LOCK	android.permission. WAKE_LOCK
4	android.permission. CAMERA	com.google.android.finsky.permission. BIND_GET_INSTALL_REFERRER_SERVICE
5	android.permission. ACCESS_NETWORK_STATE	android.permission. MODIFY_AUDIO_SETTINGS
6	android.permission. RECORD_AUDIO	android.permission. CAMERA
7	android.permission. MODIFY_AUDIO_SETTINGS	android.permission. RECORD_AUDIO
8	com.google.android.c2dm.permission. RECEIVE	com.google.android.c2dm.permission. RECEIVE
9	android.permission.BLUETOOTH	android.permission. WRITE_EXTERNAL_STORAGE
10	android.permission. ACCESS_FINE_LOCATION	android.permission. ACCESS_FINE_LOCATION
11	android.permission. READ_EXTERNAL_STORAGE	android.permission. ACCESS_WIFI_STATE
12	android.permission. VIBRATE	android.permission. RECEIVE_BOOT_COMPLETED
13	com.google.android.finsky.permission. BIND_GET_INSTALL_REFERRER_SERVICE	android.permission.USE_FINGERPRINT
14	android.permission. ACCESS_WIFI_STATE	android.permission. VIBRATE
15	android.permission. ACCESS_COARSE_LOCATION	android.permission. READ_EXTERNAL_STORAGE
16	android.permission. RECEIVE_BOOT_COMPLETED	android.permission. ACCESS_COARSE_LOCATION

Note: The permissions used by a majority (>50%) of the apps we collect

Table 2. Mostly added new permissions after the announcement of the COVID-19 outbreak.

Permissions	Rank
android.permission. ACCESS_COARSE_LOCATION	1
android.permission. USE_BIOMETRIC	2
android.permission. BLUETOOTH	3
android.permission. VIBRATE	3
android.permission. FOREGROUND_SERVICE	3
android.permission. ACCESS_FINE_LOCATION	6
android.permission. RECEIVE_BOOT_COMPLETED	7
android.permission. READ_PHONE_STATE	8
android.permission. USE_FINGERPRINT	9
android.permission. RECORD_AUDIO	10

As we can see from Table 1, the distribution of the frequently used permissions is very similar before and after the COVID-19 outbreak, as nearly all telehealth apps require several core permissions to provide their functionalities such as connecting to the internet, making video/audio call and receiving/sending files. We observe that the permission $\mathrm{BIND\_GET\_INSTALL\_REFERRE}$ $R\mathrm{\_SERVICE}$ is more commonly used after the COVID-19 announcement, as it is given in Table 1. This permission is defined by Google, and it can provide some useful information for the telehealth companies or organisations to better understand how user find their apps (Google play referrer api, n.d.). This type of data is especially important when the companies/organisations make decision on their advertising strategy and budget. Besides, most of the telehealth apps start to enhance their app security by introducing new authentications (e.g. fingerprint and face authentication) of how users can login to their telehealth account, as users' medical and health data, including clinical data, clinical trials are all a type of sensitive data. More apps are found to utilise the fingerprint authentication login (rank 13 in Table 1) to protect users' account compared with the telehealth apps released before the announcement (rank: 21). Two mostly updated permissions $\mathrm{USE}\mathrm{\_BIOMETRIC}$ and $\mathrm{USE}\mathrm{\_FINGERPRINT}$ related to security authentication can also be found in Table 2. We also find that lower percentage of the apps utilise Bluetooth (rank: 17) after the COVID-19 announcement. However, Bluetooth is listed as one of the most frequently updated permissions as shown in Table 2. It may be due to the reason that a large number of new telehealth apps released after the outbreak of the COVID-19, and many of them currently do not support Bluetooth function, which causes less percentage of the telehealth apps using Bluetooth. But the number of app releases that support Bluetooth function increases from 82 to 112 after the outbreak. Also, combining the information obtained from the release notes and the app source code analysis, we find the common usage of the Bluetooth is to mainly support Bluetooth headphones currently, but we also note some telehealth apps are taking active steps to involve Bluetooth-integrated IoT (Internet of Things) devices in their telehealth service. Last, we find that more apps update the permissions related to location. Telehealth apps usually use this permission to help user to find the pharmacy and nearby clinics. The location information is also very helpful under the current COVID-19 context. Even though telehealth apps are not the type of app specifically designed for COVID-19 tracing, we still find four telehealth apps that provide the tracing function. $\mathrm{ACCESS}\mathrm{\_COARSE}\mathrm{\_LOCATION}$ allows to provide location accuracy to within a city block, which is an ideal option used for COVID-19 exposure notifications and securing user's privacy.

Component, method and resources: In SimiDroid, the maximum similarity score is 1, which indicates there is no update made between two versions. On the contrary, a smaller score indicates more updates are made. Based on the results, the components of the apps ($\mathrm{similarityscore}=0.95$) are less likely to be changed since the outbreak of the COVID-19, followed by method ($\mathrm{similarityscore}=0.76$) and resource ($\mathrm{similarityscore}=0.70$).

After the announcement, five telehealth apps add separated Android activities especially for COVID-19. We calculate the number of the methods that are related to COVID-19, and find out that 384 of them are removed by the telehealth apps, and 471 of the new methods are added. It indicates that methods that are related to COVID-19 are modified after the announcement, and the number of methods related to COVID-19 is increasing. We further check the methods and inspect the functions offered in each method. Majority of the methods are applied to help users to perform the COVID-19 screening test and COVID-19 information management.

• Most telehealth platforms are already equipped with core telehealth functionalities before the announcement of the COVID-19. After the COVID-19 announcement, their focus is more on delivering service related to COVID-19, enhancing the security and optimising marketing strategies.

• Functionalities related to COVID-19 are mostly developed to help user perform the COVID-19 screening test and COVID-19 information management.

3.2. RQ2: How does the COVID-19 outbreak change the way in which users review telehealth platforms from the app stores?

We collect 1985 reviews related to COVID-19 on App Store and Google Play Store. Based on the results obtained from our topic modelling process, we exclude the keywords that are related to COVID-19 directly after the topic modelling and further analyse what are the other keywords that users also would like to mention together with COVID-19. From both app stores, “family”, “result”, “test”, “vaccine”, “prescription”, “home” and “office” are the noun keywords frequently mentioned. The adjectives we can extract from the keyword list are “good”, “glad”, “helpful”, “easy”, “great”, “amazing”, “convenient”, “love” and “safe”.

Based on the keywords, we inspect the reviews that are related to the keywords listed in the topic of COVID-19. From the “family” perspective, users mention the telehealth is very helpful, and it reduces family's potential exposure to the virus. There are also a lot reviews which mention using telehealth and consulting doctors for the whole family. During the COVID-19, they do not need to go anywhere for their family's sickness because telehealth has doctors all the time. Users also mention they would like to recommend the telehealth services to their family. “Test” and “result” are the other two keywords extracted from the reviews that are related to COVID-19. These groups of reviews mostly share the experience of getting COVID-19 test from the clinic and receiving the test result from the telehealth app. As for the other lab tests that are irrelevant to COVID-19, a lot of users complain the delay in test report due to the COVID-19 situation. Besides, users also discuss about the vaccinations. A large proportion of the users complain the functionalities related to the vaccination as they cannot complete the registration of the vaccination or book the appointment of the vaccination due to unknown problems. Apart from the reviews that comment on the service of vaccination provided by some telehealth apps, some users recommend the telehealth apps to provide the information of the vaccination. We can conclude that the function related to vaccination is still under the development and needs more improvements. “Home” and “office” are the other set of keywords mentioned in the reviews. In the reviews, users comment that instead of waiting in the waiting area for hours especially now with all this COVID-19 going on, it is better to stay home and talk to a doctor through video call and avoid visiting the doctor's office. From the adjective keyword list, we can tell users attitude towards telehealth are generally positive under the situation of the COVID-19.

We also analyse how topics have changed over time. Table 3 shows the topic changing during 10 years from the first season in 2011 to the first season in 2021. We analyse the keywords listed in every season and inspect the relevant content in the original reviews. From the first season in 2012 to the fourth season in 2013, users describe the telehealth apps are helpful, but they also mention errors they are faced with. “Prescription” are more discussed in the first two seasons in 2014. Users mention lab test more from the fourth season in 2018 to the fourth season in 2019. The relevant reviews more focus on the experience of how they order lab test and obtain the result from the apps. The keyword “COVID” appears in the second and the third season in 2020, which shows COVID-19 are widely discussion after the outbreak of the COVID-19.

Table 3. Review topics by season.

Season	Topic ID
11/S1	7
11/S2	5
11/S3–11/S4	7
12/S1–13/S4	6
14/S1–14/S2	2
14/S3–14/S4	7
15/S1	2
15/S2–16/S4	3
17/S1	1
17/S2–17/S4	4
18/S1	1
18/S2–18/S3	4
18/S4–19/S4	1
20/S1	2
20/S2–20/S3	0
20/S4–21/S1	7
Topic ID	Topic keywords
0	easy, video, service, work, visit, covid, consultation, issue, phone, doctor
1	easy, health, information, appointment, medical, result, record, convenient, access, test
2	easy, service, care, prescription, office, visit, wait, call, convenient, hour
3	doctor, helpful, appointment, easy, good, love, book, reply, gud, search
4	service, medicine, order, doctor, appointment, consultation, helpful, customer, therapist, online
5	free, minute, download, perfect, worth, professional, question, add, love, kind
6	answer, question, health, helpful, doctor, information, lot, informative, error, application
7	log, update, appointment, work, result, password, health, access, ve, login

Note: S1: season 1, S2: season 2, S3: season 3, S4: season 4.

• Apart from the keywords that are directly related to COVID-19, telehealth users are also likely to mention “family”, “test”, “result”, “vaccine”, “prescription”, “home” and “office” in the COVID-19 related reviews.

• From the adjective keyword list, we can tell users' attitude towards telehealth are mostly positive under the situation of the COVID-19. They think the telehealth app is “helpful”, “easy”, “convenient” and can help them to stay “safe”.

• COVID-19 is widely discussed in the telehealth apps review section on both app stores after the outbreak of COVID-19.

3.2.1. Telehealth and telehealth users from different groups

Undoubtedly, telehealth service has helped different groups of people over the years. To analyse how users from different groups think of the telehealth received before and after the outbreak of the COVID-19, we retrieve their reviews from app stores and feedback from our user study, focusing particularly on opinions left by the persons with disabilities, elderly people and the reviews related to families.

3.2.1.1 Persons with disabilities

Due to mobility issues, users with disabilities often find it difficult to move round or drive themselves to doctors' office. The overall pattern of findings from the previous studies suggests that telehealth services may be an effective approach to empower persons with disabilities and their family caregivers to “self-manage” their health conditions (Forducey et al., 2012; Long et al., 2023).

We collected 62 reviews and 4 related comments from our user study, which are all left by users with disabilities. From these reviews and feedback, we find that that users with disabilities show a positive attitude towards telehealth and often use the words “convenient” and “cheap” to describe the services. A participant commented that “···  It's also way cheaper than my co-pays and being on a very fixed income that's a blessing! Great job on this service!” Another participant said: “I'm disabled and don't drive. This service saves me time, hassle and money especially during COVID-19! Great resource! I haven't had a bad experience yet”.

In general, the attitude towards telehealth services, as described by people with disabilities, was positive. One participant stated: “This is absolutely wonderful. I'm disabled and high risk for COVID. This allows me to keep up with my health issues”. Another person mentioned that “… This is perfect for disabled people like me who have no transportation and need a doctor”, whilst another participant said “I am grateful for the service because being disabled makes it quite difficult to make it to all appointment if I had to leave my home. Happy to say MVP is up-to-date with technology and these services”.

3.2.1.2 Elders

We collected 42 reviews and some feedback from our user study that discuss the telehealth service from the perspective of older users. Most of the reviews describe telehealth works well for elders. Elders do not feel like going out to visit a doctor's office and due to pre-existing issues, and telehealth makes it very convenient for them to receive the proper treatment remotely.

However, compared with other special groups (e.g. persons with disabilities and persons have multiple family members) with less number of negative reviews. There are also 38.1% of the reviews complained that telehealth services are not user-friendly for elders. These reviews are left by elder users, elders' family members or their caregivers. Elders find the telehealth app is difficult to navigate and difficult to read because of the text colour and font size. Many reviews were very negative in terms of usability: “Poorly designed. helps is awful, circular thought process in the help menu. finding where to reset a password or a pin is too hard. Settings are hidden. Help menu says to click on triangle in upper left” and “Hard to change settings and difficult to read print”. Many users thought that the telehealth services were not designed with those individuals who are not technology focused in mind: “Not crazy about this app. I am elderly and not real computer savvy. At beginning seems like I was going in circles before getting to information I needed. Than when I tried to go into app again to read the information better, couldn't find it”.

Family members or elders' caregivers think it is necessary to have technical support with the telehealth app because it is not user-friendly for the users who have minimal proficiency with technology. They also suggest to add their elderly family members account to their account, in this case, elders do not have to deal with the technology all by themselves. One review stated: “Trying to help my dad with his doctor. This is not a good way for elderly people to get medical attention. They also need technical support with this app and their phones. They have to also have the tech capability in their homes/phones. This is not good for people who have minimal proficiency with tech. Not good for seniors. My parents are in a different state and trying to help them with a phone app is hard to do if I can't be there. Doctors, figure out another way-how about a phone call?!”. Another review stated: “How can I add my elderly mother's account to mine so that I can keep up with her lab work and notifications? She don't have computer nor internet”.

3.2.1.3 Family

3412 of the retrieved reviews mentioned “family”. 409 of these reviews suggest they would like to recommend the telehealth services to their family members or they will help their family to use the service. According to some reviews, users also mention they are with their kids or elder parents during the consultation.

Telehealth users are also of the opinion that telehealth has made schedule easier, especially for the families have multiple kids. Because when parents are sick, or their children are sick, finding time for office visits is impossible, telehealth is so much easier, and faster, than trying to get into the doctor's office or urgent care. Besides, from most parents' view, they also think using a telehealth service instead is a perfect alternative way to avoid their kids being around other sick individuals, and waiting in their home is a lot better than waiting in urgent care or a hospital setting with other sick people. One reviewer stated: “With three kids, finding time for office visits is impossible. We've been a very healthy family for a long time but we recently moved and our kids are in a new school district and our youngest started daycare… We've had a heck of a year, without this app we wouldn't have made it through like we did. Every one of us has used the app. I know it sounds like a cheesy infomercial, but telehealth has really made doctors visits quick, easy and effective”. Some telehealth users also recommend the service to their family and friends after they use the telehealth. As one telehealth user stated: “Absolutely amazing. I'm a nurse in a hospital and I'm so pleased with the care and professionalism here. I recommend this service to anyone I know with concerns or experiencing COVID symptoms. I even sent my own parents, siblings, and bf. Keep up the great work. Exceeded expectations. As far as the app. so easy to communicate with the healthcare team, view results, diagnostic”.

3.3. RQ3: Who are the telehealth users, and what are the telehealth users' choices under the context of COVID-19?

In our first survey, we receive 984 responses from MTurk. We carefully review their responses, answers that are careless and irrelevant to the questions are all removed. Finally, we obtain 888 responses in total after the invalid responses are excluded. Based on our first user study, 630 (70.9%) participants have the experience of using the telehealth service and 258 (29.1%) participants have never used a telehealth service.

We further analyse the demographics of participants including gender, age and background, the result is shown in Figure 4. First, in terms of the gender, females (74.7%) are more likely to use telehealth services than males (68.5%). Second, as for the different age groups, the results suggest for the group of people that age between 46–55 (75.6%), 18–25 (74.4%) and 26–35 (72.3%) have the highest percentage of using the telehealth services. Participants who are older than 65 have the lowest percentage with 52.6% of them used telehealth services. While as it has been mentioned the importance of delivering the telehealth service to the elders, telehealth services still need improvements to adapt to the needs of people from the special groups. We further discuss this problem in Section 5. Third, we also inspect whether our participants have used telehealth services based on their backgrounds. It is not surprise to see participants that work related to health sectors use the telehealth more than the participants having other backgrounds. Moreover, it is also worth to mention that more than half of the participants from each background have the experience of using telehealth services.

Figure 4. Users' selections grouped by different demographics.

From the participants who have used the telehealth service, 67.1% of them claim they start to use the telehealth service after the coronavirus outbreak in their country, and only around one-third of the participants once used the telehealth services before the outbreak of the coronavirus in their country. We further use ${\chi }^2$ proportions tests to make inferences about which groups' (i.t. age, gender and background) selection of telehealth are more likely to be impacted by the outbreak of the COVID-19. A smaller p value indicates more significant impact the COVID-19 has on the group. The result shows that COVID-19 has most significant impact on people that have different genders ($p-\mathrm{value}=2.543\mathrm{e}-06$), followed by background ($p\mathrm{value}=0.0006089$ ). People's selection of the way they get health care from different age groups are less likely to be impacted by COVID-19 ($p\mathrm{value}=0.0133$).

As for the participants do not have the experience of using the telehealth service, we find 17.1% of them are not even aware of the existence of the telehealth services. To further inspect their attitude towards the telehealth service, we also ask the question “Will you use telehealth services during COVID-19 when you feel unwell?” 26.4% of them choose “I will consider telehealth as my first preference ”, 58.5% of them choose “It depends on the symptoms. If the symptoms are not too serious, I will use a telehealth service ”, and only 15.1% of them still prefer seeing doctor in person regardless of the symptom seriousness. We can find that even though they have never used a telehealth service, 84.9% of them will still consider telehealth as an alternative option if they feel unwell during the COVID-19.

3.3.1. What telehealth platforms do users use?

In our study, we also ask the platforms they receive the telehealth services from, which is shown in Figure 5. Most of our participants access the telehealth services by using the chat or video call through the online communication platforms such as WhatsApp and ZOOM (41.6%), followed by making a direct phone call (36.4%), using telehealth apps installed in their mobile devices (27.8%), and telehealth websites (15.3%). From this result, we can conclude that even though 70.9% participants from our study have the experience of using the telehealth services, most of them did not use typical telehealth websites or apps. Most telehealth services are still heavily rely on the online communication platforms or the traditional phone call. We also ask the question “Will you use telehealth services after COVID-19 when you feel unwell?” 34.1% of them choose “I will consider telehealth as my first preference ”, 55.9% of them choose “It depends on the symptoms. If the symptoms are not too serious, I will use a telehealth service ”, and only 10.0% of them still prefer seeing doctor in person regardless of the symptom seriousness. 90.0% of them will still consider telehealth as an alternative option if they feel unwell even after the COVID-19.

Figure 5. Users' selections on platform.

• Female are more likely to use telehealth service than male.

• Elders are less likely to use telehealth services.

• People have the background related to health sectors use telehealth service more.

• Most telehealth services still heavily rely on the online communication platforms or the traditional phone call.

• 84.9% of our participants who have never used the telehealth services will still consider telehealth as an alternative option if they feel unwell during the COVID-19. 90.0% of our participants who have used the telehealth will still consider telehealth as an alternative option if they feel unwell even after the COVID-19.

3.3.2. Why do users choose telehealth?

We ask our participants who have the experience of using telehealth service how and why do they start to use the telehealth service. The most common reason mentioned by our participants is the impact of the COVID-19 (50.7%), they explain it is the only or best choice for non-urgent health issues under the COVID-19 restrictions. Besides, apart from the reason of avoiding the transmission of the coronavirus, users also comment telehealth services are helpful and convenient (28.7%). It's simply more convenient to make a telehealth appointment at their convenience instead of trying to rearrange their personal schedule, because some participants are busy with their work or usually work at night, and telehealth platforms can provide 24/7 service. Participants also mention telehealth is actually quicker to reduce waiting time than trying to visit the after hours clinic or emergency room and they can get care from their home, and doctors' appointments are much harder to schedule than in the past during COVID-19. The complete list of reasons is shown in Table 4. We also perform ${\chi }^2$ test and further infer what are the main reasons user choose to use telehealth after COVID-19 announcement. The result shows pandemic, and COVID-19 ($p-\mathrm{value}=1.28\mathrm{e}-23$) is the main reason followed by telehealth's usefulness and convenience ($p-\mathrm{value}=2.75e-20$).

Table 4. Reasons of why user choose telehealth before and after the announcement of COVID-19.

Reasons	B ($N=169$)	A ($N=401$)
Useful, convenient	62	88
Pandemic, COVID-19	0	265
Efficient, reduce waiting time	13	33
Cheaper	6	11
Authorise referral (doctor, telehealth, insurance service)	3	35
Other referral (family, friend)	4	8
Avoid the embarrassment	0	4
Difficult to make appointment with doctor in person	3	11
Unable to go out due to personal reasons	2	5
No appropriate clinician near me	1	3

Note: B: Number of reasons before the announcement of COVID-19.

A:Number of reasons after the announcement of COVID-19.

3.3.3. How do users rate their telehealth?

To further explore telehealth users' experience, we ask several questions such as how do they rate their experience of using telehealth, how do they think their doctors online and share their experience of using telehealth if it is possible. Based on the result, most of the users are satisfied with their telehealth experience. 18.9% users are very satisfied with the telehealth service they have received, 60.9% are satisfied, 16.6% users choose neither agree or disagree and only 4.1% of the users are not satisfied with the telehealth service.

We also ask if their doctors online are familiar with the telehealth process. 19.5% and 59.0% users think their doctors are very familiar or familiar with the telehealth process. 16.2% of them choose neither agree or disagree, and 4.9% of the telehealth users think their doctor online are not familiar with the process.

• 79.8% telehealth users are satisfied or very satisfied with the telehealth service they have received.

• 21.5% telehealth users do not think their online doctors are familiar with the telehealth process.

• Users' selection on the types of health care received is largely impacted by the outbreak of the COVID-19.

4. The use of metaverse

We searched for the related information from web and other technical or academic sources. Based on our investigation, we create a few research questions below:

• RQ1: What are the limitations of current telehealth?

• RQ2: What is the role of metaverse for telehealth?

• RQ3: What are the benefits for patients and healthcare providers?

• RQ4: What are the technical barriers?

4.1. Limitations of current telehealth

Telehealth has experienced substantial growth, largely accelerated by the onset of the COVID-19 pandemic. Nevertheless, the widespread adoption of virtual healthcare is still impeded by several critical constraints inherent in current telehealth implementations. First, healthcare providers are unable to perform direct physical examinations or utilise diagnostic devices during virtual consultations. This limitation adds complexity to making diagnoses and treatment decisions. Second, some patients may lack access to essential monitoring devices required for specific chronic conditions or specialised areas of care. Moreover, establishing the same level of patient–provider relationships virtually can be challenging compared to in-person interactions. Some patients encounter difficulties in using the technology necessary for telehealth appointments. In addition, telehealth platforms often do not ensure consistent interactions with the same healthcare provider. While telehealth enhances accessibility and convenience in numerous situations, these limitations signify that virtual care cannot entirely supplant traditional in-person healthcare for the majority of patients at present. This is where the metaverse holds the potential to be truly transformative.

4.2. Role of metaverse for telehealth

The pandemic has unmistakably demonstrated that telehealth is a permanent fixture in healthcare, and virtual reality is poised to further transform this realm. One significant domain where virtual reality will assume a pivotal role is patient recovery. With real-time data inputs and patient consent, healthcare providers will have the capacity to recommend immediate and tailored solutions as a result of enhanced patient monitoring. This capability will prove especially beneficial in elderly care scenarios, where continuous support is essential. Telehealth has democratised access to high-quality healthcare. Virtual reality will elevate the quality of remote consultations, as previously virtual sessions and video calls become more lifelike. Healthcare professionals will interact with patients as three-dimensional avatars, enabling more precise examinations and treatments. Moreover, this advancement promises an enhanced experience for both patients and healthcare providers, enriching the human element throughout the entire process. The metaverse presents a fresh approach to delivering convenient virtual care by addressing prevalent telehealth obstacles such as restricted physical examinations and detached consultations.

4.3. Benefits for patients and healthcare providers

The envisioned telehealth model, integrated with the metaverse, promises numerous advantages for patients. It introduces a more immersive and engaging experience during consultations, surpassing the limitations of conventional video visits. By leveraging the metaverse, patients may see a significant reduction in the need for time-consuming travels to clinics, as healthcare services become more accessible in the comfort of their homes. Additionally, maintaining consistent access to a dedicated care team enhances the quality of long-term patient–provider relationships. The utilisation of 3D models and simulations in the metaverse empowers healthcare providers to offer personalised guidance, visually illustrating health risks and interventions tailored to each patient's unique needs. Moreover, the metaverse's user-friendly features have the potential to alleviate accessibility challenges, particularly for populations like the elderly. In sum, this patient-centric approach to virtual care, enriched by the metaverse, upholds the essential qualities of in-person healthcare delivery.

The metaverse also holds significant promise for healthcare professionals, including doctors, nurses, physicians assistants and others. It offers a range of benefits, such as enhancing diagnostic capabilities through digital twin representations of patients, enabling non-invasive symptom evaluation within metaverse environments. Additionally, it streamlines remote collaboration by providing a platform for easy consultations with specialists and fellow care team members. The metaverse also broadens treatment options by facilitating the demonstration of complex medical procedures through 3D simulations and models. Furthermore, healthcare providers can foster deeper patient engagement and education by actively interacting with patients through avatars. Another advantage is the potential reduction in workload, as the metaverse may merge virtual and in-person care, optimising healthcare practice. Overall, the metaverse equips healthcare professionals with intuitive tools to enhance the effectiveness and efficiency of telehealth services.

4.4. Technical barriers to use metaverse

To fully harness the metaverse's potential in reshaping telehealth, a series of formidable challenges must be confronted. These include addressing technical limitations such as the development of haptic feedback for touch interactions, which currently lags and hampers certain remote physical examination capabilities. Seamless integration of patient data from wearables and electronic health records (EHRs) is imperative for the creation of digital twins. The metaverse's entry into healthcare will necessitate the implementation of new data protection measures and identity verification protocols to mitigate cybersecurity risks (Feng et al., 2022; G. Lin et al., 2020; J. Zhang et al., 2021; Zhu et al., 2022). Furthermore, integrating metaverse telehealth into existing care delivery workflows poses interoperability concerns that must be resolved. Lastly, the emergence of novel telehealth modalities like the metaverse demands the overhaul of regulations and reimbursement models, leading to regulatory uncertainty. Through the establishment of technical standards, policy adaptations and the seamless integration of metaverse technology with existing healthcare systems, the metaverse holds the potential to ultimately redefine the telehealth landscape.

In a nutshell, the metaverse presents a fresh approach to telehealth, potentially replicating and even elevating the quality of traditional in-person care within a virtual realm. With its immersive and interactive features, the metaverse has the potential to address numerous existing limitations that have hindered the widespread adoption of telehealth. Should the challenges related to regulation, interoperability and technology be effectively surmounted, the metaverse has the potential to profoundly transform the patient experience and the delivery of remote healthcare by providers. While the COVID-19 pandemic accelerated telehealth out of necessity, the metaverse now emerges as a deliberate leap forward, offering a new era of healthcare possibilities, driven by choice and innovation.

5. Discussion

In this section, we particularly look at the alignment of the collected telehealth website content. We further look into more detail of when users prefer to make use of telehealth services over going into the health clinic. Finally, we discuss the limitation of our work.

5.1. Website content

When we manually inspect the content of the website, we categorise them into three categories: (1) Telehealth ($N=39$ ): A website that mainly or only delivers telehealth service, (2) Hospital/Clinic ($N=11$ ): An official website of a hospital or clinic that also offers telehealth option and (3) Product ($N=28$ ): Companies that promote their telehealth platform. For each telehealth website, we investigate the listed features below:

• Func: Provide basic telehealth functionalities including scheduling an appointment, searching a doctor, making video/audio call, filling a prescription and delivering the medication directly to users

• Fam: Option to manage telehealth account for family members

• Symp: Clearly specify the symptoms that can be treated through telehealth service

• COVID: Provide COVID-19 related information (except the websites in the Product category)

• HC: For the websites in the Hospital/Clinic category, telehealth option can be found in the home page or near the “make an appointment ” section.

The features of the telehealth websites are shown in Table 5. Regarding the core functions that are commonly supported by telehealth platforms, the remote consulting is available on every telehealth website. Users can either choose to contact online doctor by making a video/audio call or texting message. As for making an appointment, most of the websites are appointment-based, but a few websites do not support appointment scheduling, as they are designed for on-demand or urgent cares. Users can browse doctor profiles and select the doctor they want to see. If no one is available, users will be placed in a virtual waiting room and notified by a text message once a doctor is available. The function that is relatively less mentioned by the telehealth websites is filling a prescription and delivering the medication directly to the users. After users receive the prescription, they will need to pick up the medication from local pharmacies. Besides, there are only around half of the websites mention that users can manage the telehealth information for their family members under their telehealth account. Apart from the websites from the Product category, COVID-19 related information can be found from the websites in the Telehealth and Hospital/Clinic category. Especially for the websites from the Hospital/Clinic category, they pay more attention to the services and information related to COVID-19 compared with the websites in the Telehealth category. Hospitals and clinics start to provide telehealth solution for their patients, which is a very important step under the COVID-19 situation. However, based on our inspection, users need to explore the website a bit more to find the telehealth service. Because, most of the hospitals and clinics do not put the telehealth service in an obvious section on their websites.

Table 5. Features of the telehealth websites.

	Func
Category	A	D	C	P	Fam	Symp	HC	COVID
Telehealth ($N=40$)	38	39	40	33	23	22	N/A	20
Hospital/Clinic ($N=10$)	10	10	10	5	5	3	4	9
Product ($N=28$)	22	20	28	14	15	N/A	N/A	N/A

Note: A: Make an appointment

D: Find a doctor

C: Remote consulting

P: Fill a prescription.

Table 6. Users choice based on symptoms.

Types	Symptoms	See a doctor in person (%)	See a doctor remotely (%)
A	Itchy and red skin (Skin Infections)	36.2	63.8
	Vomiting and Diarrhoea	28.5	71.5
	Insect bites and stings	26.3	73.7
	Joint Aches and Pain	23.7	76.3
	Ear Infection	51.5	48.5
	Bronchitis	55.8	44.2
	Minor Burns	19.5	80.5
	Fever	27.5	72.5
	Pressure from Work/Study	10.6	89.4
	Depression	19.7	80.3
B	Shortness of Breath	74.6	25.4
	Chest Pain	82.3	17.7
	Seizure	71.0	29.0
	Passing Out	73.6	26.4
	Stroke	87.8	12.2

5.2. Telehealth or clinic? How will users choose based on the symptoms

Based on above analysis, we inspect different telehealth websites and apps, analyse their provided information and features. Some websites and apps clearly specify the symptoms they can treat through online and also list the symptoms can only be treated in the clinic or hospital, but most of the telehealth services do not provide such information. While we read the reviews collected from the app stores, there are some users aware that telehealth is ideal for minor issues. However, we also find some reviews state that they are told their symptoms cannot be treated online after they are charged. It is critical to define the scope the telehealth can deal with, and also warn the users to seek direct help for hospital or clinic if some severe symptoms are found. In our user study, to inspect if the telehealth users have the prior knowledge of what kind of symptoms can be treated online, we design a matrix where lists different kinds of symptoms, users are put in a COVID-19 scenario, and are required to choose “see a doctor in person ” or “see a doctor remotely ”. In Table 6, category A is a list of symptoms can be treated online, and category B is a list of symptoms can only be treated in the hospital or clinic. Compared with the symptoms in category B, symptoms in the category A are commonly seen in daily life and less acute. Most users are willing to choose see a doctor remotely under the situation of COVID-19. In category B, more than 70% of users are aware that these types of symptoms cannot be treated online, but there are still a large amount of users choose to see a doctor remotely. Even though our participants all have the experience of using the telehealth service, the capabilities of the telehealth service are still not clear for the telehealth users. Choosing the wrong approach might miss the best treatment time. The rows highlighted in grey are the symptoms related to COVID-19. We can find out under COVID-19, telehealth users still think it is better to get diagnosed at home even if they get fever. As it is suggested by most of the clinics and hospital, if symptoms for COVID-19 are observed, such as fever, cough, shortness of breath, body aches or sore throat, patients are advised to be treated by a doctor remotely, via phone or video.

5.3. Limitations

In this section, we will discuss the factors that could have influenced the results.

In our user study, we design two questionnaires to further inspect the people's opinion towards the telehealth service. We invite 989 participants in our first survey and then invite 532 participants from our first survey to complete our second survey. This amount of participants may not sufficiently reflect the exact situation in real practice. Therefore, it may face the threat of the representatives of the subjects. However, for each participant we have carefully checked their responses and eliminate the responses that are answered carelessly. In the future, we plan to invite more people to participate in our user study.

In our experiment, we collect several different telehealth websites, apps and the reviews left on the app stores. However, there are still a lot of other telehealth websites we miss, as we try to cover different telehealth websites belong to different categories (e.g. hospital based telehealth), and there are some categories have a large number of instances. Therefore, for the websites that only focus on providing telehealth services, we randomly select examples and manually analyse their features. As for the telehealth apps, we exhaust the telehealth apps provided in both official app stores.

6. Related work

6.1. COVID-19 and telehealth

Smith et al. (2020) suggest that telehealth needs to be integrated into the health service appropriately. In their work, they outline important requirements to ensure the value of telehealth is realised, not only in emergencies (such as pandemics) but also in everyday practice. Caetano et al. (2020, june) suggest that telehealth offers for remote screening test, health care and treatment. It also helps COVID-19 symptom identification, monitoring, prevention and mitigation. Their results show that the initiatives triggered in current process can reshape the future space of telemedicine in the health sector. Koonin et al. (2020) suggest 154% increase in telehealth visits during the last week of March 2020. It has been proved that the changes are related to policy changes and public health guidance during the pandemic. Their work also emphasises the benefits of telehealth services during the pandemic, which include expanding access to care, reducing disease exposure for staff and patients, preserving scarce supplies of personal protective equipment and reducing patient demand on facilities. Their results show positive attitude towards the telehealth during and after the pandemic. Novara et al. (2020) state the diffusion of COVID-19. The infections have recently increased the crowd's interest in telehealth. Their result shows that the pandemic gives a significant boost to the use of telemedicine, and more robust data on long-term efficacy, safety, and costs are necessary. To optimise the health care system to be ready for the next threat to the health, Maese et al. point out the importance of maintaining the positive role telehealth take in the post-COVID era (Maese et al., 2020). In their work, they mention it is important for physicians to embrace technology, to transform and innovate the current practice of medicine. Davis et al. (2021) highlights the process that can determine the obstetrics and who are the candidates for telehealth. In an academic setting, their guidelines are vital in providing structure amid a sudden transition and can ensure the safety of patients and providers. Liu et al. (2020) advocate for more innovative designs in telehealth to be considered. With reduced opportunities for traditional clinic visits during pandemic, patients with chronic diseases are adopting telehealth services from different platforms. It is difficult to satisfy the needs of the patients with chronic diseases during the pandemic. In their work, they emphasise the importance of enhancing patients' feelings of co-presence with their health care providers during the pandemic. In response to the pandemic, Hirko et al. (2020) provide specific cases of telehealth services implemented in a large rural healthcare system. It further inspects how the shift from traditional health care to telehealth and social distancing impacts the rural health. Anthony (2021) reveals some findings on how health practitioners, patients and the medical-centres employ telehealth and other related digital services. Also, this study recommends further systematic review on telehealth and digital care solutions, as these are powerful tools to provide safe medical-care while patients under self-quarantine. One recent work (Parisien et al., 2020) assess the current utilisation of telehealth capabilities at academic orthopaedic departments in the United States and to determine how practice patterns have been directly influenced by the coronavirus disease 19 (COVID-19) pandemic. In Anthony (2020), it provides implications to inform medical staffs on the potential of digital technologies that are provided to support during and after the COVID-19 pandemic. Touson et al. (2020) conduct a case study, which creates an approach by using Harrison's open-systems model (Harrison, 2005) for rapid adoption of existing telehealth technologies in a large-scale academic medical centre. Hawley et al. (2020) identify patient-perceived barriers to home telehealth visits and classify patients into four phenotypes based on the barriers. Doraiswamy et al. (2020) suggest telehealth may have a significant effect on advancing health care in the future. However, in resource-limited settings, such as some low and middle-income countries, the feasibility and application of telehealth must be established to assist reveal its potential and transform health care for the world's population.

6.2. Telehealth adoption willingness

Lee et al. (2020) mention that the Centres for Medicare and Medicaid Services guidance has removed the restrictions that have roadblocked telehealth adoption for decades. It starts to expand the telehealth coverage during the pandemic. Gentry et al. (2020) demonstrate the willingness of the mental health clinicians. Their results indicate even though without prior telehealth experience, when telehealth is supported by the institution, licensing requirements have been addressed, and with telehealth training and IT supported to patients and clinicians, mental health clinicians are willing to embrace new technology and provide health care during the COVID-19 pandemic. L. A. Lin et al. (2020) suggest more work is needed to support large-scale telehealth dissemination and adoption. In their work, they also point out, lifting of restrictions during COVID-19 is helpful, but many of these guidelines only stress the importance the under the current public health emergency, it will also be critical to understand the pandemic effects on remote treatment and the quality of care delivered in the future. Triana et al. (2020) state that although many older adults obtain benefits from technology, common barriers still exist including self-efficacy, cost and privacy concerns. Prior research has shown that technology adoption can be improved through education and increasing perceived self-efficacy. J. Li et al. (2014) aim to find out how the telehealth service is integrated into existing models of healthcare at each site and how it impacts on practices and care processes within each particular setting.

7. Conclusion

In this paper, we inspect the impact that COVID-19 has on telehealth services including the telehealth information provided from different platforms and the comparison analysis of the telehealth app neighbouring versions. We also conduct a user study to analyse how COVID-19 has influenced users' adoption willingness towards the telehealth service and reveal the gap between the telehealth service and its users. The result shows 27.1% new iOS apps and 27.4% new iOS apps are released after the COVID-19 announcement, and surge of updates are noted right after the COVID-19 announcement. COVID are frequently discussed in the app reviews in the second and third seasons in 2020, and the most mentioned aspects that are related to COVID-19 are family, test result and vaccine. Based on our user study, COVID-19 has significant impact on the selection of the telehealth service, especially for the female group, people aged 46–55, and students. In the future, we need to invite more participants and discuss the gap between telehealth service and users. We also need to observe the influence COVID-19 has on telehealth sectors after COVID-19.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://apk.support

2 https://www.appannie.com