A careful approach to artificial intelligence: the struggles with epistemic responsibility of healthcare professionals

ABSTRACT

Machine learning approaches are being developed to contribute to the treatment of patients and the organisation of care. These new approaches are created in complex environments that include data and computational models as well as new practices, roles and competencies. In such settings, individualised conceptions of agents bearing responsibility need rethinking. In response, we elaborate on the concept of epistemic responsibility based on De la Bellacasa’s work on care (Bellacasa, M.P. de la. (2017). Matters of care: Speculative ethics in more than human worlds. University of Minnesota Press). To better understand these complex environments and the dynamics of responsibility, we use an ethnographic approach and followed Dutch healthcare professionals who learned the basics of (supervised) machine learning, while they pursued a project in their organisations during a four-month-long course. The professionals struggled with different interdependencies and this brought responsibility-in-the-making into relief. Rather than seeing (growing) relations and impure entanglements as standing in the way of responsibility, we show how connections are worthy of inquiry. We argue that connections are essential to knowledge and that producing epistemic responsibility means considering these embedded relations. In contrast to calls for control and clarification of machine learning techniques, and warnings that they create irresponsible black boxes, our care approach shows how responsibility-in-the-making reveals opportunities for ethical reflection and action. Our approach attends to how humans and non-humans are engaged in caring, reveals patterns around kinds of responsibility, and points to opportunities for avoiding neglect and irresponsibility.

KEYWORDS:

• Epistemic responsibility
• machine learning
• mental health
• care
• The Netherlands

Introduction

The teacher stands in front of the fourteen healthcare professionals in a small conference room during the first day of the machine learning course. He explains a dilemma that the professionals could face with their new machine learning prediction models: ‘What if you have calculated the chance that a patient will recover, but you know that there is only a small chance? How do you present the outcomes of your machine learning model to other healthcare professionals? Are you actually giving them a new problem instead of helping them? We face many challenges with these new techniques that are different from the challenges encountered when doing more traditional effectiveness research’. (Observation day 1)

The introduction of machine learning approaches in healthcare is not without dilemmas and uncertainties: what are its many consequences and for whom? What about the new set of approaches makes us feel like it is a game-changer in terms of responsibility? These are the larger questions we seek to address via an ethnographic exploration of projects that seek to incorporate machine learning in mental healthcare.

Artificial intelligence (AI) can be defined as the ability of a computer system to perform tasks and solve complex problems that normally require human intelligence. Machine learning is an application of AI that focuses on developing powerful systems that perform prediction, recognition or classification based on existing datasets. These techniques are increasingly used in healthcare to handle patient data, treat chronic diseases and develop medical procedures (see for example, Dwyer et al., 2018; Jiang et al., 2017; Stevens et al., 2020).

These new technological possibilities have been taken up in various policy agendas, spanning innovation, efficiency and labour shortage, in ways that spark debate about what might be appropriate ways of developing and using AI in healthcare and beyond. This has led various academic institutes, private-sector corporations and public-sector organisations to formulate principles and guidelines for ethical, accountable and responsible AI (Floridi et al., 2018; The Institute for Ethical AI & Machine Learning, 2023; OECD, 2019). Such publications offer valuable lessons, but substantial debate remains about what constitutes ‘ethical AI’ and which ethical requirements, technical standards and best practices are needed for its realisation (Jobin et al., 2019).

One of the central concepts in many publications about the ethics of AI is ‘responsibility’ (Conradie et al., 2022; Jobin et al., 2019). Documents address questions such as: who or what should be responsible for the outcomes produced by AI systems? Under which conditions is it responsible to act based on the outcome of an AI system? The publications subsequently try to clarify the attribution of responsibility and legal liability – ideally something to be done upfront in contracts, otherwise by focusing on remedy. Many different actors are named as being responsible and accountable for AI technology, including AI developers, designers, institutions or industry. Further disagreement has emerged on whether humans should always be the sole actors responsible for technological artefacts (see Jobin et al., 2019 for a detailed analysis).

We notice parallels with some discussions in the field of Philosophy of Technology, where scholars rethink conditions for responsibility (Kroes & Verbeek, 2014). Traditionally, responsibility can only be attributed to human beings, as responsibility depends on moral agency – which in turn depends on intentionality and freedom. Given AI developments in the past decade or two, some philosophers now explore whether technology (such as AI) itself, or networks of humans and non-humans including technology, can have some form of moral agency, accountability or responsibility (Floridi & Sanders, 2004; Introna, 2014; Kroes & Verbeek, 2014; Verbeek, 2014).

We believe that both principles and guidelines, as well as this body of work in the Philosophy of Technology do excellent work in rethinking conditions for being responsible and pinpointing problematic definitions and situations. At the same time, we see shortcomings that arise from assigning roles and responsibilities, and from the tendency to take the autonomous individual as a starting point. These approaches keep us in a very rational, static and structured view of responsibility. They tend to over-value autonomy and agency, thereby devaluing other relations (such as human-technology or symbolic-material) and reinforcing asymmetries between humans and non-humans (see Callon & Law, 1997; Latour, 1987; Verbeek, 2014 for similar critiques inspired by Science and Technology Studies).

In this article, we explore a different way of thinking about responsibility in AI trajectories. Rather than trying to shut down the uncertainty tied to an emerging technology and to address the dilemmas that arise with AI in healthcare through a solid set of principles, guidelines, and fundamental responsibilities, we take concrete practices as a starting point. This focus on what is unfolding in practice enables us to focus on the many ways in which responsibility develops in everyday activities, and on the way, responsibility is shared in networks and adapted to specific circumstances. Rather than peeling back the layers to a frozen set of clearly defined objects, essential roles, hard criteria and structural responsibilities, we set out to embrace the complexity of these relationships as a starting point.

We are not the first to prefer a more grounded approach with regard to ethics, and we draw inspiration for this article from two bodies of scholarly work. First, during the late 1970s and early 1980s, feminist care ethics developed as an alternative to principle-based medical ethics. At the time, many ethicists felt the obligation to define and describe the essence of ‘good’ care, indicating ‘the criteria that need to be met in order to call an activity, relation or practice care and hence good’ (Pols, 2015, p. 82, italics in original). As an alternative, feminist care ethicists introduced a more complex account of ethics. The key was to avoid prescribing a priori what good care is, and instead, to focus on the practice of care, analysing the ‘different and sometimes conflicting notions of what good care within care practices’ is (Pols, 2015, p. 82), and by emphasising relational interdependency. This perspective required not only a broader interpretation of the nature of ethics, but also a more complete account of the nature of care. In a similar way, we set out to move away from criteria and principle-based discussions about responsibility in search of a more layered account of the nature of responsibility.

Second, our exploration builds on recent STS scholarship that focuses on how particular configurations of subjects and objects, and the accompanying ethical roles assigned to each, are outcomes rather than starting points. By following practices of care and how they are reconfigured by machine learning techniques, we remain open to where responsibility might land, and stay attuned to the difficulty of letting go or of embracing responsibility, as these configurations change.

In what follows, we elaborate on the theoretical background for this research. We then introduce our case and methodology, namely an ethnographic approach to embrace all the nuances, efforts and workarounds in practice. This approach enables us to develop a detailed empirical understanding of how responsibility emerges as a concern and a practice in machine learning initiatives. The result of this reframing is to show how responsibility is produced over time and through interactions between actors. Such a shift makes it possible to address the pleas for responsible research and innovation discussed above in novel ways.

A focus on epistemic responsibility

As discussed in the introduction, there is much attention for responsibility and AI. In particular, discussions of epistemic responsibility abound since they describe duties in the process of knowing, namely giving and accepting reasons to acknowledge or believe (or not) what is being stated (Simon, 2015, p. 157). It is often argued that recent developments in AI have made the process of knowing and taking responsibility more complex, due to the growing interconnections of humans and technological systems and the intensification in the mediation of sources of information (Mittelstadt et al., 2016; Simon, 2015). This leads to fear, uncertainty and reduced feelings of personal responsibility (Mittelstadt et al., 2016).

One of the ways in which AI troubles responsibility can be traced to the development of more complicated networks consisting of humans and non-human agents, such as databases, algorithms, infrastructures and organisations. In these environments, technologies, knowledge, and knowers are engaged in complex relationships, see their power shift, and find themselves interdependent rather than hierarchically organised (cf. Callon & Law, 1997; Latour, 1987). In such complex networks, the traditional, individualised conceptions of epistemic responsibility are increasingly difficult to uphold and need rethinking (Mittelstadt et al., 2016; Simon, 2015).

A second and related point is that there is a growing mediation of sources of information (Beaulieu, 2004; Mittelstadt et al., 2016; Simon, 2015). Our ways of knowing have always been enmeshed with material and often technological objects (Rijcke & Beaulieu, 2014; Verbeek, 2015). In this context, it is important to see machine learning not so much as a single technology but as suites of systems that feed into each other, connecting data gathering, processing, computing and communication. As a result, different ‘black boxes’ are created and add to the feeling that we do not know where information comes from nor how it is produced.

Towards epistemic responsibility-in-the-making

Building on these insights, we take concrete practices as a starting point and focus on the many ways in which epistemic responsibility develops in everyday activities, is shared in networks and adapted to specific circumstances. We are inspired by feminist scholarship that insists on a commitment to ongoing relationships between humans and non-humans in knowing, with the implication that responsibility is an accomplishment rather than a fixed position (Suchman, 2009). This body of work further pushes ideas about moral agency by linking epistemology and ethics to ‘care’ and by focusing on the complexities, interdependencies and tensions involved. This combination allows us to foreground the labour, affect and ethical-political dimensions of taking responsibility in networks (Bellacasa, 2017, p. 5), enabling more attention to political and power struggles.

A care lens highlights the doing and the active making of the world while being connected to others. De la Bellacasa (2017) gives an example of how worms take care of soil. They provide care that is essential for our survival, but worms do not consciously engage in it. However, there is a lot to be gained in studying how worms care for us and what they need to keep doing this caring. This draws attention to how we need to care for the worms ourselves, for example, by not making the ground too dry or empty of nutrients or not compacting it by using heavy machinery.

In this example, knowledge of what constitutes good soil is gained by understanding the connections in a web of relations that make up soil. Taking responsibility means acknowledging the care work needed and how it engages different entities – humans, worms and soil, but also machinery and fertilisers. Similarly, in our case, medical professionals, (data)scientists, machine learning algorithms, and data registries work together in machine learning trajectories. We study these entanglements and work with the care perspective to highlight what human and non-human actors undertake to care for the data, methods, results and how they take responsibility through this caring (Bellacasa, 2017).

In our eyes, this care approach to epistemic responsibility has several advantages. What can we learn when we imagine non-human others (such as machine learning algorithms and computers) to be entangled with care? First, a care approach attunes us to the care-like work done by non-human others in sustaining the everyday, while prompting us to pay closer attention to the particulars of their existence. Second, care is neither necessarily positive nor benevolent. ‘Bad’ technologies, such as bombs, polluting cars and nerve gas also need care to keep functioning. The care approach is thus also about mundane maintenance: if the caring stops, things fall apart. The same goes for data; if it is not cared for, it does not get used and does not endure (Beaulieu & Leonelli, 2021; Leonelli, 2014). In addition, it provides an important corrective to the focus on innovation by bringing to the fore the work of maintenance as a form of care. Finally, thinking with care is also a way to address power and politics. As we will see, there are struggles; to decide what to care about, uncertainty about how to care for algorithms and patients (apart or, more interestingly, in conjunction), and frustrations because the algorithm may or may not care about certain things.

To summarise, we could contrast the value of our approach informed by care as the difference between asking ‘what is the situation, such that it has responsibility as an effect’, rather than ‘who is the responsible actor’. We foreground the development of relations so that entanglements are no longer seen as problematic and standing in the way of responsibility, and so that the connections become both worthy of inquiry and generative of responsibility.

The case and methodology

I walk up the stairs to the conference room for the second day of the machine learning course. A couple of the healthcare professionals are already sitting at the long table, and I take a seat. Once again, I notice a shortage of extension cords and electric sockets. Everybody wants to plug in their laptops to make notes and prepare for the R (the programming language) practicum later today. Our teacher, Z, brings in the janitor and who is so kind as to start looking for a few extra extension cords. I decide to use this opportunity to grab a cup of tea in the hallway. Once back, there is some somewhat strange, dark music playing. I ask healthcare professional B what is going on. B explains that this particular song is based on a machine learning algorithm and says that ‘Z told us that this is a quiz; we have to guess which algorithm it is!’. This confuses me, do we really have to guess this? I have no idea where to start! But then I realise that it is only a joke. (Observations day 2)

This vignette contains several figures that are important for our analysis. First, there is care work going on. There is a need to arrange connections; laptops will be essential to the work to be done, and humans in various roles (professional, researcher, teacher, janitor) ensure that the technologies are properly cared for so that these will later enable the humans to engage with other technologies, such as the R programming language and the algorithms that will run on the laptops to make predictions based on their data. A second figure is the reconfiguration of relationships to algorithms: the music provides a sensory invitation to engage with novel objects. The joke foregrounds whether it is even possible to guess the identity of an algorithm from its musical performance. Humour contains a call to consider what kind of epistemic work can be done and plays with the displacement of the boundary between the serious and the fanciful, between what can and cannot be known. While clearly an exaggeration of the possibilities, this joke articulates the invitation to the participants of the course to open themselves to knowing differently, to the logic of correlation and pattern-making that characterises algorithmic knowing.

* * *

At the centre of our ethnographic exploration is a site where machine learning approaches became entangled with healthcare practices: a machine learning course to train healthcare professionals in the many possibilities of machine learning algorithms. It was initiated in 2018 by Z, an expert at a Dutch non-profit institute that specialises in research and takes an active role in distributing (innovative) research and technologies in the Dutch mental healthcare field. Frustrated by the empty promises, complexity and obscurity that surrounded machine learning, Z aimed to train professionals in the basics of supervised machine learning, while supporting them in performing a machine learning research project that could be relevant to their organisations.

The small-scale course lasted four months and was organised around four full training days (March–June 2019). The course was advertised in online newsletters in the Netherlands and many healthcare professionals and organisations soon showed interest. They recognised the importance of the new statistical techniques for their organisations but often lacked expertise. The course was seen as a way to quickly and cheaply obtain this expertise.

Fourteen healthcare professionals (A-N) were selected for the course by Z, while also being mandated by their institutions to pursue (alone or in pairs) a project that could be significant for their organisation. Z made sure that the projects were feasible with regard to the skills of the professionals, and in terms of the amount of time that they could dedicate to the training. He also ensured that a diversity of professionals, institutions and types of projects were included. All healthcare professionals – except one employed by a municipality – worked for large organisations that specialise in the treatment of patients with diverse mental illnesses or addictions. The participants had different roles and career trajectories. While about half identified mainly as medical professionals and the other half as data analysts/desk researchers, all professionals had better patient care as a main concern. While the focus of the organisations varied (addiction, psychiatric conditions or prevention and public mental health), all are part of the publicly funded and highly regulated mental health care system of the Netherlands.

The professionals used the projects to address persistent problems within their organisations. Some projects were aimed at improving the healthcare organisation itself, for example, by predicting treatment costs and time. These predictions could be helpful for organisations in the negotiations of care budgets with their municipalities (a necessary procedure in the Dutch healthcare system) and with staff allocation. Other projects were more directly aimed at improving the treatment of patients. A better estimation of relapses, incidents, referrals and drop-outs could be used to place patients sooner in the ‘right’ treatment trajectory, or give better support and prevent crises. The professionals hoped that prediction models could speed up and improve the recovery of their patients and help alleviate the extensive waiting lists for mental healthcare.

We were interested in this particular course as there are several features that make this healthcare setting especially relevant to studying epistemic responsibility. First, the healthcare field traditionally has a strong epistemic culture that favours high-quality evidence to guide treatment decisions and organise the field, carefully weighing new (epistemic) practices (Stevens, 2021). Second, the mental healthcare field is characterised by considerable uncertainty relating to disease ontology and treatment effects – it is a field that seeks more explanations. Third, it relies primarily on narratives of patients and qualitative questionnaires to make sense of the patient’s conditions and to guide treatment decisions. This means that epistemic cultures of the mental health care field relate to their object in a way that emphasises context and particularity of cases, in strong contrast to abstraction and the search for formal patterns typical of machine learning applications. In this specific sense, the struggles of actors in the mental healthcare field to assign meaning to algorithms enrich our understanding of machine learning because they contrast with fields such as genetics research (Lee & Hegelsson, 2020), where automation and datafication have a much longer history.

We reached out to Z and were allowed to observe the four training days. The first author produced field notes to capture the setting, interactions and actions during the course and paid attention to questions the professionals asked, and moments of uncertainty during plenary and bi-lateral discussions. During the training days, Z gave lectures on the basic principles and underlying theories of machine learning. In addition to the lectures, there were R assignments and short presentations about the various projects. This allowed professionals to exchange experiences. Informal conversations, questionnaires and interviews with the professionals established further relations with them, as did a presentation to the course participants about our fieldwork.

The field site was full of situations where we could join the professionals as they asked, in different ways, ‘how to care’ for information and knowledge in the healthcare field. In our analysis, we focused in particular on moments where responsibility was being put into question or was being put to work, and interactions where responsibility was reeled in or delegated. As learners are taught what is important and relevant and what to care about, various relations are foregrounded and backgrounded. This process brings responsibility-in-the-making into relief.

In the following sections, we describe how new relationships were formed between the course participants and the machine learning approaches they learned during the course. We chose to structure the results in accordance with the process of learning and stay close to the experience of the participants. We thus describe key moments that show how the participants slowly form different attachments and learn to care for the new techniques to analyse data. This allows for a generative mode of critique (Verran, 2001) that takes the daily practices and experiences of professionals as a starting point and opens up possibilities for change in thought and practice as the taken-for-granted ways of doing research are questioned.

A growing sense of attachment

The healthcare professionals entered the course with curiosity about (un)supervised machine learning and the many promises surrounding it. Most of them had an existing affective relation to quantitative research, as data analysis was already part of their healthcare practice. Yet, what the new techniques were and how they could be used was unknown to the professionals, as also explained by professional G:

I have been interested in new techniques for data analysis for quite some time now. … . There is a lot of data within my organisation, but I have the feeling that we are just scratching the surface with the traditional techniques. There is more potential in it, but my knowledge is insufficient right now. (Professional G – Questionnaire)

G expressed that she wants to gain a deeper understanding of the new techniques and the possibilities of re-using data within her organisation. Many professionals used similar arguments in articulating their desire to learn more about machine learning approaches. The professionals often referred to the hype surrounding machine learning and described how ‘others’ started a ‘shallow’ relationship with machine learning approaches, whereas they realised that fulfilling the potential of machine learning required deeper and more targeted engagements.

In the eyes of the professionals, machine learning approaches could help gain new insights and improve healthcare practice. It could intensify relationships with data that is not used enough, but that is valuable in the eyes of the professionals. For example, the professionals felt that ‘Routine Outcome Measurement’ data should be used more often to monitor their patients’ progress. In addition, it could produce new insights as it helped to articulate how a consultation, as a single instance, can be seen in the context of a pattern emerging from data. By entangling individual consultations with other data sources, the professionals could come to know more about and improve healthcare practice.

The formation of new relationships between the professionals and machine learning techniques was not straightforward. In the beginning, this new approach to data and the programming language R used in the course, seemed to be more effortful than other practices more familiar to the professionals (such as effectiveness research and SPSS). The professionals got to know the new machine learning approaches as needy (needing a lot of data) and difficult (giving all these errors). Data also seemed hard to access, requiring the professionals to establish closer relationships with new people and IT departments within their organisations.

As the course progressed, the professionals became more attached to machine learning techniques and some of its logic. They invested personal time and energy, learned to speak a new language (in R) and explored the boundaries of the new techniques as to what it could and could not do. As they watched machine learning algorithms create new entanglements in their data, patients, therapies, and organisations, they became increasingly invested in the possibilities of machine learning. This led to mutual adjustments of interrelated questions, tools and data.

By the end of the course, most projects were still in progress. In contrast with the popular imaginaries of machine learning projects, there was no instantaneous mechanistic processing of pre-existing data through the deployment of ‘an algorithm’. What we witnessed was the nurturing of connections in complex environments, while responsibilities for knowing were slowly being redistributed and reshaped.

Struggles and reflexivity

The professionals were dependent on machine learning techniques, not only because of the course and of their organisational projects, but also because of the expectations of their organisations, their own work and overarching imaginaries about the future of the mental healthcare field. They wanted to care for machine learning algorithms, to get new insights and improve healthcare practice. Yet this led to struggles and moments of reflexivity. The professionals felt uncertain about how to care for the new techniques, its algorithms and the data, and they often did not know what they should and should not do.

Designing a machine learning model meant making assumptions about what matters and should be included. The need to make such decisions did not always sit comfortably with the professionals, because it rendered explicit their responsibility in establishing a relation to the object and in shaping analysis. Taking responsibility for this kind of novel situation made the professionals insecure: they often compared machine learning to approaches they were already familiar with, where what matters felt obvious. During the second training day, feature engineering was discussed as part of the machine learning process. This meant that the professionals needed to throw irrelevant variables out of their datasets and, if necessary, reformulate certain variables. This process is a way of understanding, through iteration, which features machine learning algorithms care about. However, the participants got confused and kept asking for general rules that they could apply:

Professional F: ‘Is it better to be strict or more flexible with regards to feature engineering?’ Teacher Z: ‘I find it difficult to advise in general terms. I would say that if there is little coherence, you can probably throw the variable out, but it is a balance.’ F: ‘But do I have to select the appropriate level for each of the variables? I find that very difficult to decide as a person, that is what entire studies are about!’ Z: ‘Yes, that is possible’. (Observations day 2)

The professionals have been used to other approaches where they routinely make such decisions using familiar (scientific) standards or conventions. However, in this situation, the standards and conventions have not yet been backgrounded into a kind of everyday common sense, so that they feel like personal decisions. The changes in practices and the friction between business-as-usual and novel machine learning approaches highlighted that ethically loaded roles and responsibilities were everywhere. This breach of convention made the professionals realise that they were non-innocent participants and made them start looking for the ‘general rules’ and ‘principles’ to re-externalise these decisions. However, these could often not be given in abstraction from the particularity of a case. In other words, ‘what should I do?’ was not a useful question. In its place, professionals were led to understand that they had to do the best they could with the question ‘What can I do in this context, with this data and choosing this algorithm and given what I'd like to know?’. Yet, this was often a struggle and required further engagement on the part of the professionals.

Situating and including

As the course proceeded, the professionals did engage, despite the departure from familiar rules and conventions and they became increasingly invested in the new techniques. This meant, for instance, that the professionals gained an understanding of the decision process – which variables to keep, to complete the feature engineering step. They increasingly understood how to situate their analysis and include other professionals, thereby enabling machine learning to care for their organisational problems.

Overall, the professionals learned three critical lessons:

First, the professionals experienced that this new approach to knowing required making decisions and connecting to machine learning approaches for particular situations and questions. This meant that they had to decide which method, data, algorithm to select in every situation and create an iterative learning process. Teacher Z shaped the professionals’ responsibility as he continually tried to make relations as precise as possible. He explained, for instance, that machine learning techniques are excellent for finding predictive relations, while randomised controlled trials are better for finding causal relationships between variables.

Second, the professionals learned what machine learning algorithms needed and how they operated. They learned how the algorithms could be trustworthy and dependable as they rigidly followed commands given. In some specific instances, the professionals even came to trust machine learning algorithms. They learned, for example, that Lasso – a machine learning algorithm ‘wrapper’ that can be used in R to support the analysis – is also useful for reducing their variables as it has a built-in feature selection property (Observation day 3). This means that Lasso examines which features are predictive enough to include in the model and is designed so that if features do not contribute sufficiently, they will be kept out of the model. These insights led to a renewed appreciation for machine learning techniques on the part of the professionals.

Third, the professionals also learned that this new approach to knowing required ongoing engagement and the inclusion of people from their organisations. This meant that they needed to rely on others, ask for help from ‘domain experts’ and together reach consensus about goals, definitions and trade-offs to be made. Involving ‘domain experts’ meant that the professionals established sounding boards, started chat groups with psychiatrists and made sure they were invited to relevant meetings within their organisations or set these up themselves. The inclusion of all these experts was especially important as the professionals realised that they needed the support of their colleagues to be able to introduce their prediction models to healthcare practice eventually. This led to a further extension of the range of positions where responsibility lies, and of how decisions in machine learning trajectories were made.

Representing change

We witnessed that responsibilities for knowing were slowly being redistributed (to other actors and to the technique itself) and reshaped (away from existing standards and customs and becoming more situated). As machine learning techniques and the professionals became increasingly invested, a mutual adjustment of knowledge, technology and ethics took place. This became visible during the second training day where attention was paid to the trade-off between bias and variance. The professionals had to make this trade-off so that their final models would perform better and not follow training datasets too closely or, conversely, be too indifferent to them, in order to avoid what is called under – or overfitting. This was new to the professionals, as teacher Z explained:

Teacher Z: ‘When I was trained [in ‘traditional’ statistics], I learned that I should absolutely not accept any bias. Now, I must allow a little bit of bias if it reduces variance. That is different about the types of problems tackled by machine learning.’ (Observations day 2)

The language here is ethical and highlights the responsibilities of the professionals. In this case, the professionals should accept more bias as this results in better machine learning models. The message is that it might be possible to have slightly different configurations of knowledge, technology and ethics, depending on the situation. By accepting bias, the professionals allow machine learning algorithms to care for the data in a better way that reduces variance.

Importantly, the mutual adjustments led to new kinds of freedoms, and new possibilities to produce knowledge opened up. Machine learning techniques enabled professionals to become more creative and explore new relationships with data, since the professionals could test the predictive capacities of their models. This meant that they could try out different (combinations of) variables and data, and see whether the predictive value improved. Or in the words of professional G:

Professional G: ‘What I like about this new technique is that it makes me more creative than the traditional way of doing science. Now I can think much more creatively: what if I now connect this with this? It unleashes many thought processes. (…) That gave me a lot of energy.’ (Observation day 4)

This means that even before the projects were completed, the professionals started to care about different data and consider aspects of the patients’ lives and organisations that they had not connected to so far. These new lessons led to new questions about representation. There were many discussions about how to introduce the machine learning models to their colleagues and to mental healthcare practice. These discussions showed how deeply the distribution of epistemic responsibility is reconfigured when engaging with the new technique. Often, the professionals felt that the details of their entanglements with the machine learning models were lost within ‘simple’ percentages, and graphs. How could they explain all the nuances and intricacies of their models? And if they could not, how could they act responsibly?

During the final training day, professionals D and F led a plenary discussion about presenting their findings. Both professionals struggled; they had nearly completed the course and their organisational projects, but felt that they needed to know more and do more before they were comfortable incorporating their models into healthcare practice:

Professional F: ‘The danger with this machine learning is that we, again, are going to do many things automatically, and then I will again miss knowledge of what I am doing. You have to know the program so well!’ Professional D: ‘I run a script and I get an outcome, but I often do not have an idea about what I am comparing. I find that complicated. I have no idea that I am doing what is described in theory.’ (…) Professional D: ‘This is also scary, that you can do it so wrong!’ (Observation day 4)

The professionals’ decisions were tied to certain situations, outcomes and, as the example above illustrates, the professionals found it ‘scary’ to present a black box to their organisations, which hides the responsibilities, interdependencies and vulnerabilities. They hoped that the course would bring them better, easier knowledge but that turned out not to be the case: by engaging in new relations, they experienced anew the political and ethical aspects of caring about and for data, processes and outcomes.

In these discussions, a particular configuration of responsibility was often put forth. Teacher Z continually suggested further engagement – rather than detachment – to make these decisions. He recommended engaging with the domain experts and investigating with them what it is they care about, and to use those insights to shape machine learning approaches towards better knowledge.

Responsibility, as discussed above, means being accountable for the reason for one's knowledge. During the course, the overall tendency was to be responsible through engagement with other human actors, rather than by assigning more responsibility to machine learning algorithms based on (minimally) better performance. Rather than only looking at the criterion of prediction in evaluating a model, a more explainable, more accountable model was preferred over a model that made slightly better predictions, but was more opaque in its functioning and less open to engagement.

These many instances show how being responsible in a machine learning trajectory is about finding a careful balance, building new networks and including others. There was a constant need to interest others in the predictions of the machine learning techniques, and form new relations between colleagues and the machine learning algorithms.

Conclusions

In contrast to many popular accounts of machine learning and AI, we find that such technologies do not reveal radically new knowledge, but do include significant adjustments in knowledge, technology and ethics. Machine learning trajectories in healthcare rely on the formation of new relations and involve intense commitments to taking and delegating responsibility. Regardless of the positive or more negative valence of entanglements, they do not pose an obstacle to responsibility – on the contrary, they helped to situate ethics and to foreground kinds of accountability in knowing.

We decided to focus on responsibility-in-the-making and on how professionals of a machine learning course slowly learn to trust this particular approach to knowing. The moments when machine learning techniques feel like a ‘breach’ in the normal script of responsibility reveal ‘responsibility-as-usual’ to the course practitioners and signal differences in ways of taking responsibility across epistemic cultures (Knorr-Cetina, 1999; Stevens et al., 2020). Such frictions are important learning moments that should be cultivated (Haraway, 2016), rather than dismissed as an awkward phase on the way to ‘machine learning wisdom’. Such reflexive insights are crucial to epistemic responsibility, since they make knowers aware of how they go about giving and accepting reasons for their belief in the knowledge provided through machine learning techniques.

Rather than call for control and clarification, and deplore that machine learning approaches create even more irresponsible black boxes, our care approach shows how responsibility-in-the-making can emphasise ethical reflection and action. We argue that such an approach should complement more theoretical discussions about responsibility, agency and intentionality in, for example, the field of Philosophy of Technology since it offers important lessons for responsible research and innovation. Our view is that attention to how humans and non-humans are engaged in caring both reveal patterns about kinds of responsibility and point us to opportunities for avoiding neglect and irresponsibility.

The care approach makes it possible to find out how professionals responsibly shape machine learning models. We described three mechanisms professionals used during the course: first, they learn to connect to machine learning techniques for particular questions and situations. Second, the healthcare professionals gain in-depth knowledge about the technology and understand when they can trust it and when they cannot, and finally, they involve domain experts, who helped make the situated decisions needed to create working technology. This shows that responsible artificial intelligence is not simply a matter of explainability or accuracy, but also of creating circumstances in which various actors can learn together, ask questions about the functioning of the technologies, make situated decisions and form tight and fitting networks in which the technology can be used responsibly.

With regard to the specifics of the healthcare sector, the strong figure of the patient shapes the kind of accountability that our healthcare professionals sought. The aim to provide better care for a concrete beneficiary was a strong ethical landmark. This took the concrete form of the explicit involvement of clinicians (psychiatrists and/or psychologists). Their input was repeatedly called upon to negotiate definitions and cut-off points and to learn about how data was created, registered and handled. In contrast, concerns more closely related to informatics (computational efficiency, innovativeness, comprehensiveness, value creation or elegance of models) were much less prominent. Legal and ethical concerns encapsulated by the General Data Protection Regulation also shaped the context of work, since the healthcare professionals often struggled to get access to data, obtained only partial data sets, or had to first effect changes in the organisations’ procedures in order to obtain data in GDRP-compliant ways.

Another important aspect of the mental health care context is the fact that much of the data that course participants considered especially valuable for their projects was generated by psychiatrists and psychologists in the course of their clinical work. Participants were all too aware of the limitations and potential of this data, given the time pressures on professionals, the variation in comprehensiveness and completeness of data entry, the difficulty of putting patient narratives into data formats (especially when it came to outcome monitoring data), as well as negotiation with regard to the use of terms (Stevens et al., 2020). Participants were therefore eager to demonstrate the value of machine learning techniques based on this data, to show it relevance to clinical decision making and to stimulate clinicians to fill out these data fields better. Finally, in spite of the multiplication of relations we witnessed in our site, the figure of the patient, while important and often invoked by actors, largely remained a figurative one. Neither actual patients nor representatives were involved in these machine learning projects. The lay-expert divide in health care was therefore not challenged in the remaking of responsibility.

A sharp contrast therefore emerges between our account and the promises of prediction and upscaling that are important drivers of the adoption of machine learning in mental healthcare. Clearly, standardisation of data and analyses, and the application of generalised rules would pave the way for a faster rollout of machine learning approaches. Yet, these come at the cost of the situated entanglements that make machine learning work in what our participants considered a responsible way. When our participants accounted for the knowledge they hold, they stressed the need to convey uncertainty, trade-offs and balance, and interdependence with a range of other actors. As scholars of knowledge production, we therefore plead for attention to this kind of epistemic responsibility, in the face of the effects of scaling, the consequences of the concentration of power, and the negative upshots of non-accountable knowledge production. By focusing on responsibility as the result of entanglements and stressing the importance of ongoing engagements and iterative learning processes, we show the potential for a different path towards understanding how responsibility arises and can be sustained in machine learning trajectories.

Ethics declaration

An ethical waiver was obtained for this study on 15-04-2019 by the Medical Ethics Review Committee of the Erasmus MC, Rotterdam, the Netherlands (MEC-2017-540).

Acknowledgements

We would like to thank all mental healthcare practitioners and teacher Z, who allowed us to join their course. We are also grateful to the members of the Health Care Governance group at the Erasmus School of Health Policy & Management as well as Marjolijn Heerings and two anonymous reviewers for valuable insights and suggestions.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes on contributors

Marthe Stevens

Marthe Stevens studies the ethical and societal consequences of new technologies, primarily in health and medicine, and education. She is currently a postdoctoral researcher at the Department of Political Philosophy and Ethics (Radboud University, The Netherlands) and Radboud University’s Interdisciplinary Hub on Digitalization and Society (iHub). Her research lies at the intersection of Philosophy of Technology, Science and Technology Studies and critical data studies. She holds a PhD from Erasmus University Rotterdam, the Netherlands (2021).

Anne Beaulieu

Anne Beaulieu holds the Aletta Jacobs Chair of Knowledge Infrastructures at the University of Groningen, The Netherlands. Her work focuses on complexity and transition in knowledge infrastructures. Together with her colleagues in the Knowledge Infrastructures Department, she conducts multi- and transdisciplinary research to study, develop and innovate knowledge infrastructures, with a focus on the areas of climate change and ecology. Beaulieu is co-author of Data and Society: A Critical Introduction (Sage), of Smart Grids from a Global Perspective (Springer), and of Virtual Knowledge: Experimenting in the Humanities and Social Sciences (MIT Press) and of numerous articles that address the use of digital technologies for research, collaboration and intervention. Her recent and current work on Knowledge Infrastructures for Liveable Futures will be published at Bristol University Press.