Collaborative Intelligence: A Scoping Review Of Current Applications

ABSTRACT

This review provides a novel examination of the emerging field of collaborative intelligence and demonstrates the value that human-AI teams can deliver. Humans and artificial intelligence (AI) systems have complementary strengths. This complementarity creates the potential to achieve a step-change in performance by combining inputs from human and AI on a common task. We introduce the construct of “collaborative intelligence” and develop a set of criteria, for evaluating whether an AI system enables collaborative intelligence. Applications utilizing collaborative intelligence had to have (1) complementarity (i.e. the collaboration draws upon complementary human and AI capability to improve outcomes), (2) a shared objective and outcome, and (3) sustained, two-way task-related interaction between human and AI. A systematic review of 1,250 AI applications published between 2012 and 2021 was carried out to investigate whether real-world examples of “collaborative intelligence” could be identified. The review yielded 16 AI systems which met the criteria, demonstrating that collaboration between humans and AI systems is possible and that these systems offer a wide range of performance benefits including efficiency, quality, creativity, safety, and human enjoyment.

Introduction

Industry 4.0 is underway with organizations adopting automated systems, building their internet of things and using big data, smart systems, and cyber-physical systems to expand their capabilities and improve productivity (Mason, Ayre, and Burns 2022; Schuh et al. 2020; Szász et al. 2021; Veile et al. 2020). The next wave of innovation, known as Industry 5.0, focuses on human-centric technology development and adoption (De et al. 2021; Maddikunta et al. 2022). Rather than simply automating tasks that were previously performed by humans, there is greater focus on using collaboration between humans and smart machines to augment the capability of humans (Sindhwani et al. 2022).

The term collaborative intelligence refers to collaborative human-AI systems that leverage the different attributes and strengths of each agent to achieve further improvements in work outcomes (Billman et al. 2006; Daugherty and Wilson 2018; Jarrahi 2018; Seeber et al. 2020). Organizations investing in AI-human collaboration are expected to boost revenues by 38% within five years (Shook and Knickrehm 2018). While the literature provides a strong rationale for utilizing collaborative intelligence (Dellermann et al. 2019; Madni and Madni 2018; Traumer, Oeste-Reiß, and Leimeister 2017), the construct has not been tested through systematic empirical research. A systematic review of AI applications is needed to determine whether any collaborative intelligence applications have been developed and, if so, how they are being used and what benefits they provide. In this paper, we develop a set of criteria for identifying applications that utilize collaborative intelligence and use them to systematically analyze potential collaborative intelligence applications reported in academic and gray literature. Our goal is to test whether collaborative intelligence exists as more than a theoretical construct and, if so, to describe the types of collaborative intelligence that are currently technologically and economically feasible.

Why Collaborative Intelligence?

Examples of more collaborative human-AI interactions (Epstein 2015; Kolbeinsson, Lagerstedt, and Lindblom 2019; Poser and Bittner 2020; Seeber et al. 2020) are described using a range of terms including human-robot interaction (Cesta, Orlandini, and Umbrico 2018), human-robot teams (Wolf and Stock-Homburg 2023), human/machine in the loop (Ostheimer, Chowdhury, and Iqbal 2021), hybrid intelligence (Akata et al. 2020; Ostheimer, Chowdhury, and Iqbal 2021) and collective intelligence (Dellermann et al. 2019). There is currently little consensus across the literature regarding how to differentiate the range of ways in which humans and AI can collaborate (Wolf and Stock-Homburg 2023). In this study we use descriptions of collaborative intelligence suggested by other researchers to draw out three defining characteristics of collaborative intelligence. First, the collaboration involves a sequence of shared actions between human and AI agents toward a shared objective (Cienki 2015; Kolbeinsson, Lagerstedt, and Lindblom 2019; Wang et al. 2020). Second, to enable this level of interaction the AI agents must have the ability to share and respond to information about the task and adapt to changes in the state of the human agent and the task (Kolbeinsson, Lagerstedt, and Lindblom 2019; Wang et al. 2020). Finally, the collaboration between the human and AI agents improves the performance, novelty, productivity, or quality of work above what could be done individually (Dellermann et al. 2019; Madni and Madni 2018). Together, these descriptions provide three criteria for identifying AI systems that enable collaborative intelligence:

• Complementarity:The goal of the interaction between human and AI agents is to leverage their unique strengths to achieve improved outcomes. This excludes human-AI interactions that use the human to teach the AI so that in the long run the AI can perform the task independently. It also excludes applications that are designed to probe or test the dynamics of collaboration or teamwork rather than to complete a task (Yan, Wang, & Gerada 2021).

• Shared objective: The human and AI agents are focused on the same objective and the activities of the human and AI agents are integrated and indivisible in the final output that is produced (Dellermann et al. 2019; Dubey et al. 2020; Johnson et al. 2014). The workflow must go beyond a simple division of labor or a transactional relationship.

• Sustained interaction: Interaction between the human and AI agents must extend beyond a static interaction such as a single question/answer dynamic (Traumer, Oeste-Reiß, and Leimeister 2017). Reciprocal communication which enables each agent to understand changes in the state of the objective or the other agent and respond adaptively is critical for all collaborations and is a key feature of collaborative intelligence (Madni and Madni 2018; McDermott et al. 2018).

The motivation for developing collaborative intelligence applications (as opposed to AI applications with collaborative capability) has two sources: the potential for improved task performance and improved work satisfaction. Whilst the capability of AI has been improving rapidly, there are still many tasks that AI cannot perform despite these being simple tasks for a human. The strength of AI lies in its computational power and its ability to process very large amounts of data, recognize patterns and evaluate alternative decision options (Ajay Agrawal, Gans, and Goldfarb 2019; Jarrahi 2018). However, AI struggles to understand common-sense situations (Jarrahi 2018), make intuitive decisions or judgments based on indescribable factors (Ajay Agrawal, Gans, and Goldfarb 2019; Goldfarb and Lindsay 2020; Jarrahi 2018) and respond to novel situations (Jarrahi 2018) – tasks that a human can perform well on. These complementary strengths suggest that there are likely to be many fields in which performance can be optimized by using a combination of human intelligence and AI (De Luca 2021).

The potential of collaborative intelligence is demonstrated in the evolving use of human-computer chess teams that combine human intuition and computational power (Kasparov 2010). When IBM’s Deep Blue program defeated world chess champion Gary Kasparov in 1997, the ability to process hundreds of millions of moves per second outperformed human creativity and imagination (Kasparov 2010). Less than a decade later, in 2005, in a chess tournament of human-computer teams, two amateur players won against teams of grandmasters and the most powerful computer programs. The amateurs developed a superior process to leverage the most value from their computers demonstrating “weak human + machine + better process was superior to a strong computer alone and … superior to a strong human + machine + inferior process” (Kasparov 2010; Thompson 2013).

The second argument for utilizing collaborative intelligence lies in the potential to improve the quality and scope of work for humans. Some types of work are inherently more motivating than others (Hackman and Oldham 1975). By automating functions that are less motivating for humans but allowing the human to add value by performing more rewarding tasks we can improve the quality of work. Typically, AI systems are implemented and designed to increase productivity and accuracy of tasks (Birhane et al. 2022). However the sociotechnical nature of collaborative AI allows us to extend this capability and consider improving worker satisfaction (Sartori and Theodorou 2022). While it is not within the scope of this paper to discuss the optimal design of collaborative work involving both humans and AI, this important topic is already being explored (Parker & Grote). Current research suggests that the experience of using AI can affect a human worker’s experience of predictability, controllability, meaningfulness, and fairness (Battina 2018; Langer and Landers 2021; Oh et al. 2018; Parker & Grote). In addition, by allowing human capability to be augmented by AI capability, there is potential to address skills gaps within the existing workforce or increase the pool of workers who can perform a given role.

Research Objectives and Contributions

Our goal through this research is to evaluate the proposition that complementary facets of human and artificial intelligence can be combined to achieve a step-change improvement in performance. To this end, we developed a set of criteria for evaluating whether or not “collaborative intelligence” is embodied in an AI system. We then carried out a systematic review of AI applications published between x and y to determine whether these criteria were met. Having identified examples of AI systems that met our criteria, we then provided a description of these first examples of collaborative intelligence, describing (1) what types of tasks they perform, (2) the roles of the human and the AI in the collaboration, (3) the mechanisms through which the human and the AI interact and (4) the improved outcomes achieved from the collaboration. The review illustrates that collaboration between humans and artificial intelligence on a shared goal is not only possible but that it has the potential to support positive outcomes across a wide range of measures, from creativity, to safety, to productivity.

Methodology

A systematic methodology was adopted for the literature search, using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (Arksey and O’Malley 2005; Liberati et al. 2009). Scopus, ProQuest, Web of Science, and IEEE Xplore databases were used for the literature search because they contain a mix of academic and gray literature sources across a broad range of topics. Additional directed internet searches were carried out using the Google search engine in private browser mode to avoid the impact of cookies and previous searches on the returned results. In addition, where secondary sources of potential applications were revealed during the review of a full-text article, the secondary reference was also reviewed.

Keywords were identified and selected through an iterative approach including pilot searches and reviewing key documents. These keywords were “human” and “artificial intelligence collaboration;” “hybrid intelligence” and “artificial intelligence;” “collective intelligence” and “artificial intelligence;” “human computer collaboration” and “artificial intelligence;” “hybrid teamwork” and “artificial intelligence;” “cobot” and “artificial intelligence;” “human machine collaboration” and “artificial intelligence;” “work” and “artificial intelligence.” Proximity and wildcard syntax were used where databases allowed, along with commonly used acronyms and synonyms for artificial intelligence (e.g., AI, machine learning, ML). The results of the searches were limited to English language documents published between January 1 2012 and 31 December 2021.

The keywords were simplified for the Google search protocol to accommodate the broader range of potentially relevant documents and to avoid being overly restrictive or returning results that replicated those done in the database search. The following key terms were used: human machine collaboration artificial intelligence machine learning; hybrid intelligence; collective intelligence artificial intelligence machine learning; human computer collaboration; hybrid teamwork artificial intelligence machine learning; cobot artificial intelligence machine learning; human artificial intelligence machine learning collaboration. Restrictive limiters such as proximity searches and quotation marks were also removed. The Google relevancy ranking was relied on to identify the most relevant documents, and the first 30 items returned of each of the seven targeted google searches were analyzed for applications of collaborative intelligence.

Criteria

Literature Review

The database search returned 1,250 documents. The titles and executive summaries/abstracts (where available) of these documents were initially screened to identify those that were eligible for inclusion for full-text review. To be included there needed to be an indication or reference to an AI application that involved tasks completed by both AI and human. Of the 1,250 documents, 335 were eliminated as duplicate documents and a further 445 were excluded after the initial screening process. The remaining 470 documents were assessed for inclusion in the systematic review through a detailed full-text review. Each document was assessed against the following eligibility requirements:

• Complementarity:The collaboration between the human and AI agents improves the performance, job satisfaction, novelty or productivity, of work. The outcome of the collaborative task in a collaborative intelligence application achieves a better result than either the human or the AI would alone.

• Shared objective and output: The human and AI agents are focused on the same objective and the final output represents an integration of their individual contributions.

• Sustained period of interaction: Interaction between the human and artificial intelligence must occur over time, rather than via a singular or static interaction.

From an analysis of the full texts, additional records were excluded because either (1) the full-text article could not be sourced after extensive searching (n = 3), or (2) the document did not meet one or more of the inclusion criteria (n = 448). A total of 451 documents were eliminated from further analysis based on these full-text exclusion criteria. The remaining 19 documents detailing 16 collaborative intelligence applications (3 collaborative intelligence applications were detailed in two unique documents) were included in the systematic review for further analysis (see Figure 1).

Figure 1. Results of the literature search.

Results

An extensive search (encompassing 1,250 documents from the academic and gray literature) revealed 16 examples of AI applications that met the criteria for collaborative intelligence (see Table 1). Analysis of these applications revealed that they were designed to perform various types of work. We identified five types of collaborative intelligence applications: creative agents, industrial agents, healthcare agents, emergency services agents and knowledge work agents.

Table 1. Collaborative intelligence applications identified from the review.

Type	Description	Collaborative process	Details
Creative agents	Evolver (Feldman 2017) is designed to collaboratively produce generative graphic design artifacts based on constraints controlled by a human graphic designer in an iterative design process. Evolver enhances human creativity during the production of artworks and graphic designs.	The human agent provides Evolver with a design brief. Based on the seed material and design constraints, Evolver begins generating design outputs. The design outputs can be modified and edited by the human agent and fed back to Evolver to create alternative designs based on updated and specified features.	Interface: GUI Stage of Development: Prototype Country of Development: Canada Sector: Academia/research
	Flow Machines Professional (Avdeeff 2019) is an AI-assisted music composing system. The goal of Flow Machines is to enhance the creativity of humans in the production and development of music.	The human agent provides Flow-Machines with a set of scores that they want to take inspiration from. In response, Flow machines suggests melodies, accompaniments, and instrumentation. The process is iterative, where human agents can provide feedback to Flow Machines and provide additional desired parameters until they reach a satisfactory result.	Interface: GUI Stage of Development: In commercial use Country of Development: Japan Sector: Private organization (Sony)
	StoryDrawer (Zhang et al. 2021) is an intelligent agent that supports children aged 5–10 years old in creating oral stories through collaborative drawing. This application aims to enhance the creativity of the fictional stories and provide an educational experience for children learning to develop creative writing skills.	StoryDrawer produces a random story element in the form of a doodle on the screen, which the child can add. Based on the doodle, the child begins orally telling a story. StoryDrawer analyses the oral description and extracts keywords to generate additional illustrations in real-time. If new material is desired by the child during storytelling, they can create a drawing and StoryDrawer will generate a sketch based on their drawings, or the child can generate a random doodle for inspiration. The child can provide feedback to StoryDrawer; if the feedback is negative, StoryDrawer will provide another “idea” in the form of a doodle.	Interface: GUI Stage of Development: Prototype Country of Development: China Sector: Academia/research
	An unnamed robot dancer application performs classical dancing patterns in harmony with the artistic moods expressed by a human pianist (Saffiotti et al. 2020; Thörn, Knudsen, and Saffiotti 2020). The purpose of the collaboration between the robot and human dancers is to spark inspiration and creativity in the human co-performers and enhance the audience’s experience.	The robotic AI system analyses the artistic expression of the human dancer and dynamically responds with dancing/gestures to match this expression. The AI system may be creative and proactive in setting parameters. In response to how the robot performs, the human dancer may adjust their moves, resulting in a harmonious joint performance between the human and the robotic agent.	Interface: Cyber-physical system Stage of Development: Prototype Country of Development: Sweden Sector: Academia/research
	Shelley (O’Brien 2017; Yanardag, Cebrian, and Rahwan 2021) is an AI bot that collaboratively creates short horror stories with human users on Twitter. The application aims to spark creativity and introduce novel ideas for fictional horror story writing. Shelley also provides a form of entertainment for the humans interacting with it on Twitter.	Shelley generates and posts new stories on Twitter and invites followers to participate in collaborative story generation. Shelley tracks the responses sent by the human users and automatically generates and posts new continuations. Shelley also responds to stories initiated by Twitter users.	Interface: GUI Stage of Development: In use Country of Development: Turkey and Germany Sector: Academia/research
	Calliope (Urban Davis et al. 2021) is a collaborative virtual reality system that enables designers to explore and manipulate generative design solutions to a given design problem. The objective of Calliope is to provide innovative and novel designs to enhance human creativity.	Calliope facilitates an iterative design process, while encouraging a collaborative dialogue between the machine and designer. The human user generates or selects an object/form they would like to create from a range of available options. The AI then generates various designs based on specifications and edits that the human makes to the object. The AI can also integrate elements of design from various samples.	Interface: Virtual reality headset Stage of Development: Prototype Country of Development: USA, Canada Sector: Academia/research and private organization (Autodesk)
	The Cobbie (Lin et al. 2020) robot is designed to support the human design process by providing inspirational stimuli at the ideation stage. The application sketches on top of the human’s sketches when invited by the human designer. The objective of the collaboration is to inject new inspiration and creativity into the in early-stage design of consumer goods in a human-led manner.	Cobbie and the human have shared pens for sketching. The human chooses which pen (stroke type) to use at each stage. The human can move Cobbie to select what section of the sketch it uses for inspiration. Cobbie has buttons that allow the human to direct it draw or pause sketching. The human can provide feedback to the Cobbie on the quality of its design.	Interface: Cyber-physical system Stage of Development: Prototype Country of Development: China Sector: Academia/research
	Wordcraft (Coenen et al. 2021) is an AI-assisted editor for story writing in which a human writer and AI system collaborate to write a story. The collaborative writing assistant supports the writing process from planning through to writing and editing.	Wordcraft comprises a traditional text editor and an AI assistant. The human writer can ask the AI to complete specific tasks, like expanding upon a plot or story that the human agent has started, elaborating on specific elements of a story and filling in elements of the story to spark new ideas. A side-by-side word-processor and dashboard are used to enable communication between the human and the AI agent.	Interface: GUI Stage of Development: Prototype Country of Development: USA Sector: Private organization (Google)
Industrial agents	The COMPLEMANT system (COllaborative robot aMPLifying and Extending huMAN capabiliTies) (Bettoni et al. 2020) comprises a cobot and smart decision-maker for industrial and manual processes. The collaboration is designed to increase productivity and reduce the fatigue and stress of human industrial workers.	Wearable devices collect the human agent’s mental and physical workload and physiological responses during industrial tasks. A vision system monitors the environment and work process. The COMPLEMANT system uses inputs from these sensors to compute and suggest optimal working configurations for human workers and cobots to complete the industrial process. The human worker can ask for reconfigurations or choose to reject the suggestions made by COMPLEMANT.	Interface: Cyber-physical system Stage of Development: Prototype Country of Development: Switzerland Sector: Academia/research
	ARMAR-6 (Asfour et al. 2018) is a humanoid robot able to assist industrial workers. It can carry out various tasks, including grasping, mobile manipulation, integrated perception and natural language understanding. It is equipped with a suite of sensors and the cognitive abilities to facilitate natural, safe collaboration with humans. The objective of the collaboration is to provide support so that humans can concentrate on the “skilled” part of a job whilst the robot takes on heavy lifting and support roles.	ARMAR-6 can safely and intuitively collaborate with humans in various industrial and maintenance tasks by combining advanced sensorimotor skills with learning and reasoning abilities. ARMAR-6 can infer when a human needs help, and proactively offer the most appropriate assistance. ARMAR-6 can recognize human activities and intentions, reason about situations, and interact with humans in a natural way. ARMAR-6 can also grasp and manipulate objects bimanually to accurately and safely use tools such as power drills and hammers.	Interface: Cyber-physical system Stage of Development: Prototype Country of Development: Germany Sector: Academia/research
	Bionicworkplace (Festo 2018) is an intelligent collaborative workstation that works with humans in customized, short manufacturing runs. The objective of the system is to support workers to carry out manufacturing tasks more efficiently, relieving them of tiring or hazardous tasks.	The Bionicworkplace workstation is equipped with a bionic arm, various assistance systems, peripheral devices, and tools. It uses sensors and camera systems to register the positions of the human operator, components, and tools. It uses this information to derive the optimal program sequence of operations to be carried out between the human and the robot to manufacture the target item. The system continuously learns from each action. Humans can intuitively control the Bionicworkplace by means of gestures, touch, or speech.	Interface: Cyber-physical system Stage of Development: In commercial use Country of Development: Germany Sector: Private organization (Festo)
	SHERLOCK solution (Dimitropoulos et al. 2020; Nikos Dimitropoulos, Togias, Zacharaki, & Michalos, 2021) provides robotic assistance for human workers on a variety of manufacturing and industrial tasks by identifying what actions need to be completed and then supporting the human to complete the task by moving and holding objects based on the human operator’s preferences and ergonomic factors. The system is designed to improve productivity, quality and safety and enable customization of manufacturing and assembly processes.	The SHERLOCK solution system is a general-purpose robotic system that can be adopted for use across a range of manufacturing and industrial tasks. It is comprised of three modules that can capture the human operator, environment, and process status of a task. It can identify the tasks that are being executed by the human operator using vision-based machine learning and provide customized support to reduce physical strain and promote ergonomically correct posture. It also responds to inputs from the human operator around their preferences for how they would like to carry out the industrial task.	Interface: Cyber-physical system Stage of Development: Prototype Country of Development: Greece Sector: Academia/research
Healthcare agents	HALS (Human-Augmenting Labelling System) (Diao, Chen, and Kvedar 2021; van der Wal et al. 2021) assists humans with cellular annotation on tissue images to identify and label pathological cell types. It is designed to decrease the human annotator’s workload and increase the effectiveness of the annotated data, enabling annotation of data sets that were previously cost prohibitive.	A human annotator labels cells within a small region of the tissue slide. An untrained classifier AI system is trained based on the human’s annotations and then begins making suggestions to the human, which the human can accept or reject. HALS learns from these corrections and progressively improves its predictions progressively over time. The AI also identifies patches of the slide to annotate next, guiding the annotator around the image. The machine has three deep learning models – to (1) learn the labels provided by an annotator (2) provide recommendations to that annotator designed to increase their speed, and (3) determine the next best data to label to increase the overall quality of annotations while minimizing total labeling burden.	Interface: GUI Stage of Development: Prototype Country of Development: USA, South Korea Sector: Academia/research and private organization (Salesforce)
	The SAGE patient management system (Goldberg, Belyaev, and Sluchak 2021) supports accurate diagnosis and treatment of pathologic syndromes. The AI agent does not propose a diagnosis, but it analyses the health provider’s diagnosis and treatment plan in relation to data collected from the patient to identify where there may be logical inconsistencies and contradictions in the diagnosis and ongoing management of the patient.	A physician determines input parameters (vital signs, labs, imaging) for a patient and makes a preliminary diagnosis. SAGE checks the data for possible errors and develops a diagnosis. If SAGE’s prognosis matches the physician’s opinion, SAGE does not manifest itself in any way. If they do not match, SAGE displays the parameters which do not match the physician’s diagnosis and prompts the physician with an additional question. SAGE determines the most useful question that would challenge the physician’s opinion. If an answer to this question matches SAGE hypothesis, but the physician’s original assessment of the situation remains unaltered, the SAGE asks a new question (maximum two additional questions). SAGE displays the trends of the overall severity of the patient’s condition, the severity of leading syndromes relative to the dynamics of type and intensity of treatment. SAGE offers trends of parameters that need attention when defining treatment. The system detects and analyses contradictions in the provider’s decision-making process, as it compares the provider’s prognosis with the outcomes which are reflected in improvement or deterioration of the patient’s condition. It brings to attention various contradictions in the management of the patient.	Interface: GUI Stage of Development: Prototype Country of Development: USA Sector: Academia/research
Emergency Response agents	DroneResponse (Agrawal, Cleland-Huang, and Steghöfer 2020) is a cyber-physical system (CPS) involving the collaboration of humans and smart, semi-autonomous unmanned aerial vehicles (UAVs) to carry out rescue operations and emergency response missions. The purpose of the collaboration is to support faster, more successful, and safer emergency responses.	Teams of humans and semi-autonomous UAVs are deployed to find people in need of rescue and assist in their safe recovery. Directed by the human personnel, the UAVs scans an area to locate the person in need of rescue and request feedback (e.g., to confirm a potential rescuee sighting). Once the rescuee is located, the AI agent can request additional support from other UAVs (e.g., one equipped with a flotation device that can be dropped to the rescuee) while streaming imagery of the rescuee until human rescuers arrive. Humans can leverage this information to make more informed decisions and directly intervene in the UAV’s behavior (e.g., directing the UAV to collect imagery from a different angle or assigning additional UAVs to provide support to the rescue effort).	Interface: Cyber-physical Stage of Development: Prototype Country of Development: USA Sector: Academia/research
Knowledge work agents	ForSense (Rachatasumrit et al. 2021) is a web browser extension that supports and improves online research work. The AI agent accelerates and extends the human’s online research activity by identifying additional material and suggesting ways of classifying the material.	The human begins identifying relevant material from webpages and organizing it into groups. The AI uses that material to identify other website material relevant to the research and suggest groups the new material may belong to. The human can accept or reject the new material and refine the original groupings based on their own sensemaking or suggestions provided by the AI. If the human decides to regroup material the AI recomputes the word embeddings so that it can begin to identify website material relevant to the new groupings.	Interface: GUI Stage of Development: Prototype Country of Development: USA Sector: Academia/research and private organization (Microsoft)

Characteristics of Collaborative Intelligence Applications

The second objective of this review was to profile the current state of collaborative intelligence applications by describing their characteristics.

Stage of Development

Although our analysis covered documents published between 2012 and 2021, all of the 16 collaborative intelligence applications included in the analysis were documented after 2017, indicating the emerging nature of this field. Three of the collaborative intelligence applications had been released for use publicly, namely, Flow Machines Professional (Avdeeff 2019), Shelley (O’Brien 2017; Yanardag, Cebrian, and Rahwan 2021) and Bionicworkplace (Festo 2018; Kärcher et al. 2017) (see Table 1). Bionicworkplace was the only identified collaborative intelligence application that was commercially available. Bionicworkplace, developed by Festo, is a complex and adaptable cyber-physical system comprised of various sensors, tools, and abilities to provide a flexible, collaborative workstation. The adaptability of the cobot is especially beneficial for increasing productivity in the development of short-runs of customized items (Kärcher et al. 2017).

The other two publicly available collaborative intelligence applications were offered as free digital products. Flow Machines, developed by Sony (Pachet, Roy, and Carré 2021), enables collaboration between humans and AI to enhance and inspire creativity during music production (Pachet, Roy, and Carré 2021). Shelley, a TwitterBot developed by a group of researchers to probe the success of human-AI collaborative fictional horror story development, has been deployed for public use on the Twitter platform, resulting in over 500 collaborative narratives. Shelley was designed to enhance the emotional impact of human stories, introducing novel and surprising directions to the narrative (Yanardag, Cebrian, and Rahwan 2021). The collaborative stories between Shelley and humans were found to induce greater negative effect and state anxiety than those created by Shelley alone, indicating the success of collaborative creation (Yanardag, Cebrian, and Rahwan 2021). There is no indication that it has been used to generate any financial profit for the developers (Yanardag, Cebrian, and Rahwan 2021). The remaining 13 collaborative intelligence applications were defined as prototypes and proof of concept designs, developed in universities and research organizations. Some of these applications appeared to be undergoing further development toward bringing the application to market for public use, for example DroneResponse, AMAR-6 and Forsense (see Table 1).

Collaboration Channels

The applications that we identified were evenly divided in terms of whether the collaboration between the human and AI occurred in a virtual or physical environment. The virtual collaborations occurred through a graphical user interface (GUI) (n = 7) and a virtual reality headset (n = 1). Forsense (see Table 1) is an example of collaborative intelligence that occurs in a virtual environment. It was developed for people carrying out exploratory research online (e.g. collating, organizing and making sense of information) (Rachatasumrit et al. 2021). The system provides collaborative support to human users allowing them to accelerate, improve and coordinate their search tasks. The collaboration is enabled through a web browser extension. The GUIs used in the identified collaborative applications have an interface that enables the human and the AI to provide feedback and respond to one another, although the human is the arbiter in decision-making (with the exception of Shelley (O’Brien 2017; Yanardag, Cebrian, and Rahwan 2021)).

The collaborations involving cyber-physical systems used robots (or cobots) or drones. The human and the cobot collaborated through a virtual communication channel using a GUI, sensors or wearable devices. DroneResponse (Agrawal, Cleland-Huang, and Steghöfer 2020) is an example of a cyber-physical system where communication occurs through a GUI. The DroneResponse prototype was developed in 2020 by a group of academics from the United States using semi-autonomous UAVs that collaborate with human agents through GUIs to provide faster, more successful and safer emergency responses than human or UAVs could alone (Agrawal, Cleland-Huang, and Steghöfer 2020). In this application, the GUI enabled bi-directional communication around mission plans and situational changes between the human and UAV rescue teams. ARMAR-6 (Asfour et al. 2018) was another example of a GUI-based cyber-physical cobot, which collaborated with human technicians on maintenance tasks. Communication with the human agent occurred through gestures, voice commands and various sensors used by the cobot to detect changes in the state of a task or their human collaborator.

In terms of the way that information is shared between the human and AI agents during collaboration, the SAGE application (Goldberg, Belyaev, and Sluchak 2021) presents a unique design choice. It is a patient management system designed to collaborate with human medical practitioners to improve diagnostics and patient treatment and care. The designers of SAGE sought to develop a system that communicates with human agents in an explainable way to build trust and ultimately better collaborative outcomes. Non-collaborative patient management systems will simply provide a diagnosis, but SAGE collaborates with the practitioner by probing data collected from the patient during the course of treatment to identify and communicate any indicators that are not consistent with the practitioners’ original diagnosis or prognosis. SAGE uses a high-level and intuitive visual interface to communicate any issues of concern but the practitioners can interrogate SAGE to understand the patient indicators that underpin the issues that SAGE identifies. In this way, SAGE can reduce the impact of the human practitioners’ biases or limited attention and improve decision confidence (Goldberg, Belyaev, and Sluchak 2021). The explainability and transparency of the system’s decision making was designed to address socio-technological barriers such as trust between humans and machines that affect the quality of collaborative decision making (Goldberg, Belyaev, and Sluchak 2021).

Types of Benefits/Outcomes Sought from the Collaboration

Enhanced productivity or accuracy was a common objective for the collaborative intelligence applications that we identified. Improving human worker satisfaction and safety was another motivation for collaborative intelligence applications. The third type of outcome sought was creativity.

An example of an application that was designed to increase productivity is HALS (Diao, Chen, and Kvedar 2021; van der Wal et al. 2021). HALS is a collaborative human-AI labeling workflow that assists human pathologists with cellular annotation of pathological cell types in tissue samples. The system is designed to enable accurate annotation of large data sets that were previously cost prohibitive. The collaborative use of HALS by a group of seven expert pathologists found a manual work reduction of 91% along with a boost in data quality of 4.34% (van der Wal et al. 2021).

There were several collaborative intelligence applications that were designed to improve human safety, either for a human client or for a human worker. Bionicworkspace (a cobot used in collaborative manufacturing and production tasks) is designed to reduce physical strain on human workers as well as improving their productivity (Kärcher et al. 2017). On the other hand, SAGE (Goldberg, Belyaev, and Sluchak 2021) and DroneResponse (Agrawal, Cleland-Huang, and Steghöfer 2020) improve safety by supporting accurate patient diagnosis and management (SAGE) and providing faster search and response for people in need of rescue (DroneResponse).

Evolver (Feldman 2017) is example of a collaborative intelligence application designed to enhance human creativity during the production of artworks and graphic designs. Evolver collaboratively produces generative graphic design artifacts based on constraints controlled by a human graphic designer in an iterative design process enabled through a software program. A group of 10 designers went on to test the application and how it impacted their outputs and the creative process. It was found that the collaboration provided accessibility to alternative design solutions, helping designers to step out of their current frame of reference to assist with the ideation process (Feldman 2017). Although the designers largely reported positive experiences associated with collaborating with Evolver, the issue of authorship was raised by several participants. This is likely to be an issue that arises with a number of co-creative collaborative agents and could be a challenge to the adoption of these technologies into professional creative practices.

Discussion

Our systematic review of the literature yielded 16 examples of collaborative intelligence applications in which human and AI agents work collaboratively to achieve a shared outcome that achieves more than either agent could achieve on their own. Of these examples, almost all were in early/prototype stages of development rather than commercially available products. There was a mix of embodied cyber-physical and virtual software systems and the channels and format of the communication between the human and the AI also took several forms, ranging from sensors and computer vision to natural language processing and GUI key commands. The applications were designed for a variety of fields including healthcare, manufacturing, graphic design, emergency services and creative writing.

Technological Feasibility of Collaborative Intelligence

The range of applications that we identified illustrate that collaboration between humans and AI is technologically feasible across a range of domains and toward multiple ends. Furthermore, the applications delivered a range of benefits. Working with creative collaborative intelligence applications improved both efficiency and creativity. Manufacturing and assembly collaborative intelligence applications improved efficiency and health and safety. Knowledge work applications improved the accuracy and coverage of the decisions and classifications that were produced. We infer that it is technologically feasible to combine a variety of human and AI capabilities and thereby achieve benefits in terms of efficiency, quality, creativity, safety, and human enjoyment.

Some researchers argue that because AI is designed to support humans and should give humans control in the interaction, AI is more appropriately described as a tool or knowledge artifact than as a collaborator or agent (Cabitza, Campagner, and Simone 2021; Shneiderman 2020). In our view, the essential aspect of collaboration is not the equality between the actors in the collaboration but rather, the fact that they each contribute to a shared output. For example, the first author of a research paper might define the scope of the paper and make decisions about how to incorporate the input of each author in the final manuscript. Nevertheless, the authors of the paper are considered to be collaborators by virtue of their contributions to that joint output. Furthermore, it is important to be able to differentiate between AI systems that are designed to perform a discrete task (serving as a tool for that task) and AI systems that can contribute to a task over time, while responding to and sharing information about that task with a human actor. So while the applications identified in this review do not demonstrate agency in the form of awareness or independent initiative they are designed to contribute to an output using collaborative processes. The socio-technical implications of AI tools, AI knowledge producers and AI collaborators are likely to be very different and they should be differentiated through our terminology.

Addressing Large Language Models

Since conducting our systematic review, Large Language Models (LLMs) such as ChatGPT and DALL-E have entered the market and achieved an unprecedented rate of adoption (Hart et al. 2023). While a review of the growing range of LLMs available in the market is not within scope, we would be remiss not to consider how these AI tools align with our criteria for collaborative intelligence. The LLMs meet the first criterion for collaborative intelligence because they complement human workers through their superior computational power, which enables them to draw upon and integrate an enormous body of data. Experimental research has already demonstrated that this complementarity improves outcomes ranging from productivity, quality and customer satisfaction (Brynjolfsson, Li, and Raymond 2023; Dell’acqua et al. 2023; Noy and Zhang 2023). Furthermore, LLMS are capable of understanding and supporting humans at multiple stages in the workflow (from brainstorming, content creation, editing and adapting) (Irons et al. 2023). LLMs are known for their highly naturalistic conversational capability. When prompted to, they are able to draw upon the history of their interactions with the human user when formulating responses, enabling sustained interaction related to the objective defined by the human user. In addition, they can adapt their output in response to feedback or prompts provided by the human. In this respect, the final output can (even if it does not always) represent a fairly indivisible integration of human and AI contributions. However, due to the large and varying number of LLM models that are available now, classifying them as a single application against the three criteria used to define collaborative intelligence applications in the current study, does not account for the complexity and variation between them. Therefore, we conclude that while LLMs largely embody the defining characteristics of collaborative intelligence, further examination in future research is warranted. The massive interest in LLMs is testimony to the value of AI systems that can complement a human with their greater computational power, understand and communicate with a human about a shared objective and contribute to a shared output. These tools also illustrate the value of communication channels that allow the human to provide contextual information and identify refinements or improvements to the AI’s output that are needed to solve the task at hand.

Current Constraints and Future Directions for Collaborative Intelligence

The collaborative intelligence applications identified in this review were restricted in terms of the range of environments in which they could operate and the variety of tasks that they could perform. Whereas human to human collaborators often make frequent switches between collaborating in the real world (e.g., to scope requirements in an initial meeting) and collaborating virtually (when they write a document together or share information and advice via online channels), the current collaborative intelligence applications do not have this agility. Collaborative intelligence applications can work with humans in a virtual environment (e.g., generating digital content) or in the real world where they take a cyber-physical form. However, while the cyber-physical forms of collaborative intelligence often communicate with their human collaborator via online channels, they are generally not capable of collaborating with them in this environment. Whereas human workers will often collaborate over multiple stages of a project or task, the collaborative intelligence applications were limited to discrete tasks and stages of production or decision-making. This uniquely human ability, the capacity to transfer capability in one domain to a related domain, will be a key challenge in the further development of future collaborative intelligence applications.

The review also revealed hundreds of applications that met some but not all the criteria for collaborative intelligence. These findings suggests that future applications of collaborative intelligence will emerge in finance, investment, and insurance; defense and security operations; and within scientific research. We expect developments in the field of collaborative intelligence to accelerate rapidly as a result of the availability of LLMs. The pace of LLM adoption across tasks and fields is increasing the visibility of the potential benefits of AI and our willingness to engage in collaborative work processes with AI. Finally, the interdependent and interactive nature of collaborative AI applications presents an imperative to examine the ethics aspects of responsible implementation and design of such applications (Abeywickrama and Ramchurn 2024; Seeber et al. 2020)

Limitations

One limitation of this study is that it is based on published academic and gray literature written in English. There are likely to be more collaborative intelligence applications under development not captured in the literature because they are not yet ready for commercialization, and represent valuable intellectual property. Patent databases may offer another fruitful dataset for identifying additional examples of collaborative intelligence.

A second issue that affected the review was the lack of detailed information on all the potential collaborative intelligence applications that were identified. Several AI applications potentially met the criteria for collaborative intelligence but were not described in sufficient detail to be assessed fully. Thus, while this paper provides the important foundations for documenting current applications of collaborative intelligence, it may not capture the full range of applications that currently exist.

Conclusion

The dominance (up to now) of AI applications that automate work can be attributed to the fact that collaborative AI requires advanced capabilities such as the ability to model the human view of the world and engage in dialogue with a human collaborator. The latest wave of AI is developing these capabilities (Stowers et al. 2021). The complementarities of human intelligence and AI and the range of benefits already associated with collaborative intelligence applications suggests that collaborations between human intelligence and AI can catalyze a new wave of innovation that enables more efficient, safer, sustainable and enjoyable work and lives (Nahavandi 2019).

Acknowledgments

We acknowledge the support of the CSIRO Collaborative Intelligence (CINTEL) Future Science Platform.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was funded by the Commonwealth Scientific and Industrial Research Organisation’s Collaborative Intelligence Future Science Platform.