‘We have opened a can of worms’: using collaborative ethnography to advance responsible artificial intelligence innovation

ABSTRACT

With the recent rapid developments in artificial intelligence (AI), social scientists and computational scientists have approached overlapping questions about ethics, responsibility, and fairness. Joined-up efforts between these disciplines have nonetheless been scarce due to, among other factors, unfavourable institutional arrangements, unclear publication avenues, and sometimes incompatible normative, epistemological and methodological commitments. In this paper, we offer collaborative ethnography as one concrete methodology to address some of these challenges. We report on an interdisciplinary collaboration between science and technology studies scholars and data scientists developing an AI system to detect online misinformation. The study combined description, interpretation, and (self-)critique throughout the design and development of the AI system. We draw three methodological lessons to move from critique to action for interdisciplinary teams pursuing responsible AI innovation: (1) Collective self-critique as a tool to resist techno-centrism and relativism, (2) Moving from strategic vagueness to co-production, and (3) Using co-authorship as a method.

KEYWORDS:

• Responsible AI
• collaborative ethnography
• science and technology studies
• machine learning
• interdisciplinary research

Introduction

One of the enduring cornerstones of the programmes of responsible innovation (RI) and responsible research and innovation (RRI) has been a focus on promoting inclusive knowledge ‘co-production’ through inter/multidisciplinary collaboration,¹ between the natural sciences and engineering on the one hand and the social sciences and humanities (SSH) on the other (Felt et al. 2016; Owen and Pansera 2019; Pansera et al. 2020). To this end, there have been diverse attempts at integrating social and ethical considerations into processes of research and innovation, sometimes encouraged by research funders and more often spearheaded by social scientists (Owen et al. 2021). Of note are collaborative approaches such as constructive and real-time technology assessment (Guston and Sarewitz 2002; Rip, Misa, and Schot 1995; Schot and Rip 1997), socio-technical integration (Fisher et al. 2015), midstream modulation of technology (Fisher, Mahajan, and Mitcham 2006), collaborative experimentation (Balmer et al. 2016; Delgado and Åm 2018), situated interventions (Katell et al. 2020) and ‘Future Labs’ as spaces for anticipatory and reflexive pedagogy (Conley, Tabas, and York 2022).

Our study was conducted within a project funded by the UK Engineering and Physical Sciences Research Council (EPSRC) in a research-intensive university in the UK. RI is now a requirement in many EPSRC grant applications and is often established as a crosscutting area of research within its research centres and centres for PhD training (EPSRC 2018; Owen 2014; Owen et al. 2021; Stilgoe, Owen, and Macnaghten 2013). As a result of commitments of this sort, social scientists are increasingly invited into spaces led by researchers in engineering and computer science e.g. as so-called ‘champions of RI’, to conduct training and pedagogy in RI or to deliver RI-related research within work packages in science-led research projects. These arrangements are often new to many researchers (of all disciplines) and funders like EPSRC take a deliberately flexible and non-prescriptive approach when it comes to how to incorporate RI into everyday academic practices.

While calls and pledges for ‘genuine’ and ‘engaged’ collaborations between different academic communities abound (Lyall et al. 2013), a persistent open question is how these aspirations can be realised in practice (Bromham, Dinnage, and Hua 2016; Lindvig and Hillersdal 2019; Pansera et al. 2020; Schuijff and Dijkstra 2020). Interdisciplinary ventures can be fraught with challenges; they often demand collaborators get out of their comfort zones and spend time understanding and negotiating diverse and unfamiliar jargon, vocabularies, concepts, methods, styles, and procedures (Felt et al. 2016). Collaborators are faced both with the day-to-day frictions between unfamiliar academic traditions and practical difficulties when deciding who does what and how to disseminate research outcomes (e.g. where to publish). All too often, interdisciplinary projects are organised in a siloed and transient fashion, with discipline-specific work packages and deliverables that present limited opportunities for lasting joint collaborations and contributions (Felt et al. 2016). Social scientists are also aware of the potential for ontological capture, instrumentalisation, and being cast in various – sometimes gendered – roles (e.g. the ‘trophy wife’) (Balmer et al. 2015). Whether and how different disciplinary orientations, ontologies and epistemologies seeking to coalesce can effectively do so and pursue more socially beneficial outcomes remains an empirical question worth pursuing.

We address this question while focusing on contemporary debates surrounding the social and ethical implications of artificial intelligence (AI) and the urgent need to advance responsible innovation in the field. In recent years there has been growing concern amongst different academic communities over troublesome issues of discrimination, bias, oppression, exploitation and other harms associated with the growing prevalence of algorithmic systems and data-driven innovation in society. Harmful and unjust outcomes resulting from the use of algorithmic systems have been evidenced in myriad locations such as welfare distribution, insurance valuation, recruitment, migration and mobility, policing and crime recidivism, and the invisible, and often precarious labour that sits behind the technology e.g. for labelling and cleaning datasets (Birhane 2021; Brayne 2017; Eubanks 2017; Noble 2018; Rosenblat and Stark 2016; Williams, Miceli, and Gebru 2022).

Critical scholars in the social sciences and humanities have approached ethical questions about digital technology from various fronts including, among others, moral philosophy, surveillance studies, science and technology studies and critical data/algorithm studies (see e.g. Domínguez Hernández and Galanos 2022; Manokha 2018; Miceli and Posada 2022; Stark, Greene, and Hoffmann 2021; Zuboff 2019). This growing body of scholarship has contributed significantly to advancing ethical and moral debate in a fast-paced and largely opaque arena of data-intensive innovation.

At the same time, different strands of computer and data sciences have engaged in a vigorous programme of research and scholarship related to issues of fairness, accountability and transparency in algorithmic systems. Much of this work has focused on mitigating the failures and unwanted outcomes of algorithmic systems, usually taking a pragmatic, solution-oriented approach where the desired properties are engineered or designed into a system (e.g. privacy enhancing technologies) or where failures are flagged to practitioners and decision makers (Selbst et al. 2019). Examples of the latter include approaches to debiasing ML models and operationalising ethics and fairness in the design of systems through mathematical formalisms or risk scores² (Binns et al. 2017; Feldman et al. 2015; Geisslinger et al. 2021; Kleinberg et al. 2018; Ryan et al. 2021).

Driven by common ethical concerns between social science and humanities, and data and computational sciences, there has been more interlinking and attempts to collaborate in recent years. Reflective of this is the emphasis on socio-technical framings and interdisciplinarity in computer-science-led conferences such as Fairness, Accountability and Transparency (FAccT) and Artificial Intelligence, Ethics, and Society (AIES).³ While this is a positive sign, some scholars insist that there remains a wide ‘divide’ or ‘gulf’ separating the computational sciences and critical work in the social sciences, notably science and technology studies (STS), critical data/algorithm studies and surveillance studies, among others (Moats and Seaver 2019; Sloane and Moss 2019).

Very often researchers venturing into these sorts of interdisciplinary collaborations are faced with tensions and the need to negotiate different, and sometimes conflicting, ontological, epistemological, methodological and institutional norms and commitments (Domínguez Hernández and Galanos 2022; Owen et al. 2021). Not only that, the way in which critique is approached and mobilised within such interdisciplinary spaces is contested and can have significant political, interpersonal and professional implications with which collaborators need to grapple (Griffin, Hamberg, and Lundgren 2013; Wiklund 2013). While critique and critical reflection have been implied by frameworks for responsible innovation and AI ethics, the practical challenges and consequences of doing critical work in this arena have been under-researched.

We wish to contribute to the special issue’s theme of ‘Critique in, of, and for Responsible Innovation’, by drawing lessons from a project-centred and experimental responsible AI collaboration at the nexus between critical social sciences and computational sciences. To do so, we propose, and reflect on, collaborative ethnography as a methodology to advance productive modes of critique in such interdisciplinary spaces. We begin by discussing some of the tensions arising from interventionist and critical endeavours within STS and allied fields and how these sensibilities can be mobilised to engender responsible innovation in the field of AI. Next, we describe the setting up of the RI collaboration and the institutional conditions within which it took place. We then discuss the rationale behind the choice of collaborative ethnography as a methodology and how this was configured and deployed within the AI project. We end by drawing lessons for interdisciplinary teams on how collaborative ethnography can offer a generative and productive space for critique, self-critique/reflexivity and action.

The destabilising role of critiquing and intervening in innovation

Critical inquiry on technology and innovation – and particularly the field of STS – has had a long-standing concern with productively influencing the sites it enters, be it by mobilising insights about technoscience and its social entanglements to new audiences, advising science policy, or reconfiguring (intendedly or not) the objects and practices being studied. Over the years, STS and adjacent social science scholars have been invited to collaborate more closely in the making of science and technology, often as members of interdisciplinary projects, research consortia, responsible innovation programmes, and socio-technical interventions, with varying degrees of success (Balmer et al. 2015; Fisher et al. 2015; Jensen 2012; Owen et al. 2021). It is not rare today to find concepts and methods from STS being productively deployed in allied academic and practice-oriented communities across STEM, design and human–computer interaction (Costanza-Chock 2020; Irani et al. 2010; Lyle 2021; Ratto 2011; York 2018).

In the last decade, there have been increasing efforts within STS to branch out from pure theorisation, thick descriptions and arms-length critique into more action-oriented and politically engaged academic work which seeks to be mutually enriching both to STS researchers and their collaborators. A number of performative, artful and participatory endeavours have emerged under the rubrics of ‘STS in action’, ‘interventionist STS’, ‘STS making and doing’ and other forms of action-oriented research (Downey and Zuiderent-Jerak 2016; Jensen 2012; Lippert and Mewes 2021; Reinsborough 2020). Such an overt interventionist impetus is often characterised by methodological inventiveness and a desire to have a more influential role in the fields of study through using or performing STS (Lippert 2020).

Experiments and experimental practices have long been a paradigmatic site for studying scientific knowledge production and its controversies, and it is perhaps stemming from this familiarity that experiments have become powerful new ways for researchers to mobilise and use this more action-oriented STS (Law 2016; Lezaun, Marres, and Tironi 2017). Experimentation with different ways of doing and communicating STS research beyond its own traditional academic audiences has been a recurring approach in this regard (Downey and Zuiderent-Jerak 2021; Evans, Leese, and Rychnovská 2021). We can see this specifically in the context of data science, where social scientists are invited to be part of interdisciplinary teams and integrated within rather than alongside STEM projects. As part of embedded working arrangements these offer opportunities for more co-productive and deliberative collaboration between data scientists and critical scholars (see e.g. Neff et al. 2017).

In his essay ‘In, with and of STS’, Ingmar Lippert (2020) proposes three different modes of using STS: ‘to contribute in STS, to contribute with STS to other fields, or to conduct a study of STS itself’ (4). As we will show, the collaborative encounter we discuss here is more closely attuned to the second use in Lippert’s typology. We sought to test how critical, ethnographic and action-oriented sensibilities can combine to inform practices in data science, with the explicit aim of engendering reflexivity and responsible innovation. While this exercise was not envisioned as a conventional case study aimed at extending existing STS scholarship, it offered an opportunity to flag possible misconceptions and pitfalls of contributing with STS in practice.

Critiquing and intervening in innovation within interdisciplinary projects are not without their challenges and drawbacks. At their core is a tension between, on the one hand, destabilisation and on the other reaching consensus on finding a constructive way forward, in a mutually respectful manner. STEM projects can have clearly defined goals, envisaged outcomes and impacts which the introduction of critique can destabilise. This can create situations of ambivalence, antagonism, disagreement and friction, which could arise from competing values, ontologies, concerns, logics, norms, professional goals, motivations or from conflict between vested interests and socio-ethical considerations. Scholars have recurrently highlighted the ambivalent and sometimes risky nature of taking the role of critics in the context of such interdisciplinary collaborations.

Balancing and reconciling critique with the desire for a consensus-driven, constructive way forward is equally challenging. The positive connotation of this sort of integrative work is sometimes made explicit through stated or intended project or programme outcomes (i.e. ‘ethical AI’, ‘responsible AI’, ‘data for good’) which set the tone from the outset for constructive collaboration (Floridi et al. 2018). In our collaboration, as we will go on to describe, this was visible through a stated and visible programmatic commitment to ‘responsible, ethical and inclusive innovation’.⁴Yet while consensus might be desirable, it does not always guarantee morally good outcomes and could in fact conceal the power asymmetries between collaborators (Crooks and Currie 2021), a concealment that critique aims to make visible. And indeed, consensus is ‘not the inevitable goal of RI’ (Owen and Pansera 2019, 34).

The balancing of critique and ‘being constructive’ can have significant political, (inter)personal and professional implications, with a risk that collaborating researchers find themselves in precarious, alienating, co-opted and ambiguous situations. Social scientists in particular have been self-conscious about either being overly complacent and unable to be critical enough; or on the other extreme, enacting an unproductive ‘ethics of suspicion’ (Balmer et al. 2015) where social scientists situate themselves on the moral high ground, strategically detached from the practices in question. Research on the effects, affects and politics of collaboration has shed some light on researchers’ experiences of insecurity, anxiety, shame and ambivalence connected with interdisciplinary work (Jönsson and Rådström 2013; Wiklund 2013). In attending to these issues, agonistic approaches and the confronting of risks and discomforts of collaboration, have been proposed as possible avenues for dialogue and generative critique which do not necessarily rely on consensus (Crooks and Currie 2021; Hillersdal et al. 2020; Moats 2021; Smolka, Fisher, and Hausstein 2021).

The contextual specifics of any collaboration are also an important consideration. Our study is set in the context of a data science project aimed at developing AI/ML solutions for misinformation detection and management. This is a technical response to the current ‘post-truth’ environment, which creates a particularly interesting dimension to critique. While the destabilising nature of critique has been an abiding issue in STS (particularly in debates around scientific knowledge production) it has witnessed a renewal in the era of post-truth. In this era, as Michael Lynch (2020, 50) puts it, ‘opposition [to science] is often expressed through the rhetoric of science, voiced by credentialed experts who present counter-narratives and “alternative facts”’. In his well-known essay ‘Why has critique run out of steam?’, Bruno Latour (2004) laments that the recent relativism sparked in public discourse can, at least in part, show similarities with the social construction of facts project advanced by STS. He writes: ‘While we spent years trying to detect the real prejudices hidden behind the appearance of objective statements, do we now have to reveal the real objective and incontrovertible facts hidden behind the illusion of prejudices?’ (2004, 227). The social constructivist argument, Latour asserted, has been misused and has of late been wielded by conspiracy theorists, anti-vaxxers and climate change deniers. Latour suggests critique should shift its analytical lens from matters of fact to matters of concern; a turn to a new realism that aims to redress the potential unintended misconception of social constructivist critiques.⁵

This discussion is not only prescient of ongoing public discourse and science legitimacy crises but directly speaks to the particular focus of our RI collaboration, which aimed to use AI/ML to distinguish facts from misinformation. In our collaboration with data scientists, we sought to contend with and overcome this ambivalence. We explored how the critique of the algorithmic construction of facts could be mobilised productively to help explore and address an urgent and significant matter of concern – in this case misinformation – while avoiding unproductive relativism and falling into the trap of endless debate.

Interrogating the algorithmic construction of facts through collaborative ethnography

In this section, we draw on our experience of initiating an RI-inflected study that integrates social studies of science and qualitative methods within a data science project developing a multimodal,⁶ ML-enabled tool to assist humans in the moderation of misinformation (for example so-called ‘fake news’) on social media. We discuss the project findings in full elsewhere (Domínguez Hernández et al. 2023) here focusing on methodological observations. The study took place under the auspices of a National Research Centre with an interdisciplinary research agenda broadly defined around ‘how to keep people safe online while allowing them to fully participate in digital technologies’.⁷

The authors led a cross-cutting ‘responsible and ethical innovation pillar’ for the Centre. The idea for the collaboration grew out of a series of meetings with principal investigators of several funded projects within the Centre which were identified as having an innovation or technology design and development component. After several exploratory discussions, we, along with our two collaborators from the data science project (hereafter the research team), identified a shared matter of concern: how to conceptualise misinformation and minimise its harm. This shared interest presented us with an opportunity to explore the potential to work with STS to attempt to co-develop responsible practices in ML and address the social issue.

We incorporated two linked STS sensibilities into the data science project, firstly a social constructivist understanding of knowledge production and ‘truth’ and secondly, a situated qualitative approach to open black-boxed technical artefacts and practices. The study was designed as a ‘collaborative ethnography’, whereby interpretation, description, critique and self-critique were deployed collectively to evaluate the development of data-driven ML models. This approach to collaboration is not new. It has been used in interdisciplinary projects as a tool to support collaborators to mobilise joint agendas. Bieler et al. (2021, 81) propose that collaborative ethnography ‘contributes to the assemblage of reflexivity as a practice that is distributed across a set of places, people, and encounters’.

In our study, we apply collaborative ethnography in an experimental fashion, with the explicit aim of defining RI in situ as a form of praxis and phronesis (Owen and Pansera 2019) rather than importing RI as a pre-given normative framework. The UK research funding council that funded the Centre advocates an approach to responsible innovation based on the so-called ‘AREA’⁸ framework, but emphasises the need for a flexible and non-prescriptive approach and that ‘different approaches might be required for different research areas’. We experimented with collaborative ethnography as an approach to RI in a setting that presented a site to test how STS methods and concepts can be brought into conversation with those of data science with the aim of problematising the algorithmic construction of facts and concurrently working out responsible practices in ML and AI.

Our study was necessarily conducted as an open-ended intervention. This was initiated towards the beginning of the technical project – i.e. after this had been funded – and ran in tandem with it, but independently of the deadline and deliverables of the technical project itself. RI was not built into the technical project by design, but the Centre as a whole had a stated and visible commitment to it. Rather, and supported by this broader mandate, we invited the data scientists to enter into a collaboration to expand the initial scope of their project to ask what the potential adverse outcomes of automated misinformation detection are (or could be)? and how we could explore and address these together?

We incorporated regular touch points for discussion and reflection (for more details, see Domínguez Hernández et al. 2023). Regular meetings were held over a period of approximately 8 months (the timeframe for developing the ML tool), during which the team discussed the algorithmic construction of misinformation and the ethical, political and epistemic issues arising during the development of ML models and when they are deployed by social media platforms. We asked our data science collaborators to explain their work to us and to be actively involved in collective mind mapping (an example of which is given in Figure 1) and writing. This constituted a purposeful strategy that we employed which acknowledges the significance of inscription within academic knowledge production and throughout the ethnographic endeavour (Bieler et al. 2021; Lassiter 2005; Latour and Woolgar 1986).

Figure 1. An early mind map of algorithmic contingencies.

A key initial objective for us as social scientists was to develop a rich and shared understanding of the technical process of constructing the machine learning model, in order to collectively interrogate the ontological and epistemological assumptions associated with this. We began by querying how the data scientists defined what counts as misinformation and what counts as truthful information for them. These definitions emerged as important framing constructs for the building of the ‘misinformation classifier model’ and the curation of its referential training and testing dataset – a process we described, following Jaton (2021), as ‘ground truthing’. In sum, this allowed us to interpret the data scientists’ strategy as being the application of machine learning techniques to automate and assist the practice of fact-checking.⁹

This set the context for subsequent discussions in which we collectively problematised the decision to take at face value the verdicts (i.e. the facticity) of claims circulating on social media made by fact-checking organisations. These verdicts can have normative implications, i.e. they can inform decisions about tagging, downranking or censoring claims identified by them as being misinformation. Drawing insights from feminist epistemology and social studies of science we underlined that facts and knowledge claims are context-dependent, partial, contingent on power relations, and socially constructed and therefore that claims to neutrality and objectivity in knowledge production and facticity need to be taken cautiously (Haraway 2013; Harding 1995; Latour and Woolgar 1986). Similarly, in more recent debates on the use of big data in scientific research, critical studies have cautioned against approaches that view data as a reflection of ‘things out there’, suggesting instead that data are always contextual and relational (boyd and Crawford 2012; Leonelli 2015).

The data scientists admitted that one of the problems they had with the ground truth dataset which they used was that fact-checkers do not always agree on their assessment of the same claims and that this introduces uncertainties in terms of labelling and classifying inputs within the ML model. We asked the data scientists to spell out how these ambiguities translated into the design of the classification algorithm and how they impacted the model’s metrics of performance. They enumerated the hypotheses, assumptions and previous studies that they used in order to determine the presence of misinformation, and which help to classify pieces of content as such using ML. Some of these assumptions included: ‘misinformation spreads faster on social media than factual claims’; ‘users who discuss misinformation on social media are different from those discussing factual information judging by their followers’; ‘the images used when discussing misinformation are different to those used when discussing factual information’.

Our collaborators explained to us the importance of metrics of performance – accuracy in particular – and how these were employed to help evaluate incremental progress within the ML subfield of misinformation detection. These metrics serve as benchmarks against which new models are assessed and are usually reported in academic publications independently of models being put into production. The data scientists remarked that there is currently no systematic or standardised approach to reporting progress in the misinformation detection subfield, and that different techniques, genres and topical domains of misinformation are commonly evaluated against the same benchmark. For example, the performance of a model designed to detect medical misinformation could be compared against that of a model detecting fake news during elections. They used the analogy of a ‘wild west’ to illustrate the lack of consensus around evaluation benchmarks in the misinformation research community.

We interpreted such use of performance metrics as a form of internal validation within the subfield rather than as expressions of algorithms’ actual utility in the world. At the same time, we argued that these metrics can be influential and performative. This is the case when they are invoked by actors such as social media platforms to vouch for the validity of an algorithm and normatively inform the need for human intervention and what decisions can be triggered by that algorithm.

Stemming from these deliberations, the research team co-produced a combination of interpretative and descriptive texts, mind maps and schematics of the steps involved in the construction of an ML classification model. We found this process of visualisation and inscription to be particularly helpful in promoting both (technical) understanding and reflection (Figure 2). As our collective discussions continued, the team were able to build on this to surface a series of what we termed ‘algorithmic contingencies’ and cautions. These include for example, that false positives could potentially lead to unfair censorship of content or have the backfiring effect of the reinforcing of entrenched beliefs; and in turn, that false negatives could allow for misinformation to spread further, which is aggravated when there is an overreliance on algorithmic moderation (see Domínguez Hernández et al. 2023). Algorithmic contingencies served as signposts to moments in the development of the model where different choices, assumptions or random events could lead to different model outcomes including negative ones.

Figure 2. Schematic of the process of an ML misinformation detection model. Reproduced from (Domínguez Hernández et al. 2023).

Mobilising the second-order reflexivity dimension of RI (Owen, Bessant, and Heintz 2013), we also situated content moderation algorithms in the contemporary political economy of data. We discussed the potential conflicts of social media companies between the need to address misinformation (and other problematic content online) and profiting from its wide circulation and consumption. We argued that the role of misinformation detection algorithms cannot be considered in isolation from the targeted advertising business model of social media companies (Zuboff 2019) given that these algorithmic tools play a key role in fulfilling the moral and legal responsibilities of such companies, and their incentives to minimise human involvement in content moderation (CDEI 2021).

The overarching critical assessment that resulted from the collaborative ethnographic process was that ML misinformation classifiers and the functional claims made about them have significant ethical, political and epistemic implications which are not commonly discussed in technical papers. The social impact of these ML models is contingent on a series of underlying (epistemic) assumptions, definitions, technical constraints, performance evaluations, and data curation choices made by developers, and, indirectly also on the vested interests of implicated information gatekeepers. These in turn allowed us to develop a set of collective recommendations aimed at responsible innovation within the subfield e.g. relating to transparency and reporting. This was enacted in the project through the making available of the data collection system that was on GitHub (Nielsen and McConville 2022) ‘so others can see and execute the exact code used to build the dataset, and thus all decisions that were made’ (Domínguez Hernández et al. 2023, 18)

Dealing with the AI can of worms: moving from critique to action

Opening the ‘black box’ of technical systems has been an emblematic metaphor for ethnographic explorations of laboratories and highly specialised technical practices. However, considerably less attention has been given to addressing/resolving the dilemmas or new problematics revealed by such acts of opening, interpreting, and translating, and relatedly, to exploring whether and how critical inquiry could at the same time be generative. In the following, we discuss how the research team collectively confronted the critiques raised in the first stages of the intervention that are discussed above. We then offer three methodological and practical lessons for interdisciplinary teams to move from critique to action while collectively pursuing responsible innovation. These can be viewed as non-prescriptive strategies to navigate moments of friction, conflict, and some of the dilemmas arising from doing critical work.

Collective self-critique as a tool to resist techno-centrism and relativism

Although the data scientists recognised the ethical and social issues surfaced by the study, they deemed them to be out of the remit of the data-science project as initially formulated. Being a technical and solution-oriented project, and bound to a relatively short timeframe, our inquiry necessarily prompted an extension of scope and concern. This is not to say that our interlocutors eluded the limitations of their methods. In fact, the opposite was the case: they were both predisposed and open to reflection on these. However, they were working within a rationale that the merit of ML detection models could be evaluated in terms of accuracy and performance and that better (i.e. more balanced) and larger ground truth data could help to offset the tool’s limitations in the future. By contrast, as social scientists, we argued that these issues could not be addressed merely through larger datasets and that the functional claims about the model’s practical utility should be nuanced and transparent in terms of their assumptions, limitations, and risks. We were able to collectively explore how we can reconcile these epistemological differences, in part due to the mandate for responsible innovation (and resources) provided for a programme of responsible innovation within the Centre, but largely because of the openness and willingness of the team to collaborate together: this collaboration was viewed by all as a clear and beneficial opportunity.

In response to the critiques laid out earlier, one of our collaborators remarked: ‘We have opened a can of worms, what do we do about this?’. This statement offers a perhaps more apt metaphor to the classic ‘opening the black box’ and is an earnest reminder of the need for making visible and interrogating not only the practices of data scientists, but of STS inquiry itself and the aim to ‘contribute with STS to other fields’. As Lynch (2000, 26–27) cautions, the practice of reflexivity risks being used as ‘a source of superior insight’ held by social scientists in relation to their ‘unreflective’ counterparts. As compared to the conventional setting of the ethnographer and informant, collaborative ethnography, we suggest, can pave a path for all participants to be reflective of their own epistemological stances, prejudices and potential biases in ways that are both critical and productive, rather than antagonising. As such, the exercise can be setup towards working out co-responsibilities relating to how and whether and how to act upon any critical findings.

For example, as part of our ongoing discussions, we asked within the research team what are the implications of flagging the shortcomings of classification algorithms? To what extent is opening the ‘can of worms’ of ML and calling out its limitations a generative (enough) process? How do we avoid critique becoming either unproductive relativism or instrumentalised as mere ethics washing associated with techno-centric solutions? Beyond listing the limitations of algorithms, can critical researchers take some degree of co-responsibility for the solutions being proposed to tackle the social problem? Can in total we contribute pragmatically to reducing the harm of misinformation as an overarching key ‘matter of concern’?

In this regard, the tensions raised during the collaboration were far from being fully resolved, and we argue did not need to be. However, in collectively working through these questions and negotiating the next steps, we were able to look for mutually enriching modes of integrating the pragmatic objectivism of data science and the reflexive approach of STS. While we acknowledged the potential benefits of the technology to tackle a social issue – harmful misinformation – we also pressed that to minimise potential negative effects of content classification algorithms, developers need to be reflexive and critical about their choices of training data, the assumptions behind these choices and the potential failures of their creations.

We agreed on a tentative way forward by suggesting practical actions aimed at expanding the existing scope of algorithm evaluation and working towards meaningful transparency about the limitations of the ML tool. For instance, we concurred that the publication of ML classification models should be accompanied by transparency reports containing not only the rationale behind data curation and problem definition, but statements about potential limitations stemming from researchers’ values, epistemic assumptions, political stances and institutional commitments. We proposed that an interdisciplinary research team would be better equipped to decide on what specifically to include in such transparency reports.

Ultimately, allocating co-responsibilities is crucial, and it does not mean collaborating researchers must address all the social and ethical issues being surfaced. Instead, engaging in collective acts of reflexivity offers a concrete way to situate the technical solution in its wider context, in this case in relation to the political economy of data, fact-checking, social media platforms and surveillance capitalism.

From strategic vagueness to co-production

At the start of the study, all researchers expressed a lack of familiarly with the others’ discipline-specific methods, language and practices, and it was unclear what the collaboration would entail. In our initial discussions, we focused on addressing questions and common matters of concern for both fields and experimenting with how to better address these instead of settling on specific methods and goals. Despite such a lack of preliminary definitions the data scientists welcomed our invitation and agreed to commit time to the study.

Questions about who would lead this effort and how it would be disseminated were raised early. We agreed that the social scientists, who proposed the collaboration and identified the gaps in the data science project in the first place, would be better placed to lead the collaboration. This agreement meant that the social scientists were responsible for the research design, organising data collection and analysis, facilitating discussions, and steering the overall direction of the study. In turn, the data scientists took the role of reflexive ML experts, consulting on technical methods, practices, and the state of the art in their field, but also providing feedback to help enrich the ethnographic exploration and taking an active part in scoping and formulating recommendations through joint deliberation.

This open-ended, experimental arrangement offered a flexible approach to collaboration, but it also posed challenges in terms of deadlines and impact. In our role as instigators of the study, we sought not only to highlight what was problematic about automation but to explore realistic opportunities for intervention and publication of results, and to contribute meaningfully to addressing the matter of concern we had collectively identified. As such, we often probed our collaborators for tangible ways in which the study could have an impact, be that in the form of reformulated practices and assumptions, or an explicit recognition of AI’s limitations and potential harms in technical papers.

Our collaboration allowed for the elucidation of key contingencies and recommendations for addressing these. While some of the key contingencies identified through the inquiry were well recognised beforehand and addressed by the data scientists, it was not clear how and when these concerns would be integrated into academic articles and datasets. Indeed, the initial outputs of the technical project were published following the original expectations and timeframe of the technical project and as such did not include an explicit reference to the RI intervention or insights from this, which were published later. Mismatched temporalities between the time-constrained technical project and the more open-ended nature of the RI intervention were an issue in this regard and remain we suggest an organisational issue for future interdisciplinary ventures of this kind. Given the technical project’s own deadlines and pre-defined publication goals, some of the findings and recommendations from the study, which were produced at later stages, could only be acknowledged retrospectively.

Ethnography is indeed known to be a slow and evolving methodology, reliant on co-location and continued commitment to the field, with some of the most emblematic studies spanning several years and even decades (Beaulieu 2010; Palmer, Pocock, and Burton 2018). Our approach to collaborative ethnography was helpful in reducing the time needed for data collection and analysis, and it also provided for all members of the research team a generative space for reflection, dialogue and mutual learning, albeit transient (Felt et al. 2016). Yet as with any exploratory and time-constrained project, the outputs of intervention were necessarily limited to partial and tentative recommendations. We laid out an inventory of remaining research gaps for the field of content moderation as well as pointers for further research and debate on how to deal with misinformation and harmful content.

Translation and inscription: using co-authoring as method

From the outset, co-authoring an academic publication served as a good motivation to initiate and maintain the collaboration, despite the uncertainty with regard to specific outcomes (c.f. Bieler et al. 2021). In practice, the process involved iterative cycles of visualisation, reflection and co-writing, where interpretative texts produced by the social scientists were routinely checked, refined and expanded by the data scientists. Most of the learning from the ethnographic exploration of the practices within the data science project was documented in provisional interpretative fieldnotes, diagrams and analyses which were iteratively exchanged with the data scientists for their input.

In the process of building a fair representation of the ML algorithm and a diagnosis of ethical issues, data scientists reviewed the proposed argument, made corrections where needed, and incorporated detailed and technical explanations of the process. This was an important phase of building technical understanding and clarification for the social scientists. The resulting text was a mix of discipline-specific styles and vocabularies; interpretative as well as descriptive, technical text.

Interim drafts of the co-authored text described the technical justification for the soundness of an AI/ML technique, such as accuracy and improved performance over previous comparable models in the data science literature, or bigger and more diverse datasets compared to existing ones. In further edition cycles, we attuned the text to a more (auto-)ethnographic style to combine the contributions of social scientists and data scientists in a single coherent piece. This entailed a process of finding common descriptors, concepts and terminologies – translating and inscribing technical processes and objects aimed at a broader, less specialised and interdisciplinary audience.

Inevitably, this type of interdisciplinary co-authorship raises ethical and political considerations. As the lead authors who were simultaneously writing and suggesting ways of framing the argument with the conceptual tools of critical social science, we were conscious of our own positionality. In encouraging collaborators to engage with and stand by the critique, there is a risk of imposing a narrative, speaking on behalf, and even co-opting or alienating collaborators in the process. To address this issue, we allowed sufficient time and opportunities for all collaborators to table disagreements, concerns or potential misrepresentations in working versions of the co-authored text. This process was particularly crucial when delineating a plan of action and a joint agenda of research or surfacing wider socio-political issues.

Lastly, we were faced with the question of finding an appropriate venue to publish our collaborative work. The challenges of peer-reviewing and publishing interdisciplinary research are well-known and documented in the literature, and are far from settled at the social-computational sciences nexus (McLeish and Strang 2016). Dedicated venues for interdisciplinary work are limited, and while venues like FAccT and AIES have shown openness to social science methods, concerns remain over the deficit in peer reviewers’ experience with methods and perspectives which are outside the dominant STEM fields (Laufer et al. 2022). Publishing is also crucially shaped by the pressures of funding bodies, academic impact and career development plans. For early career researchers in particular, decisions as to the structure of papers, shared authorship, where and when to publish, and how to communicate research to intended audiences can have significant career and interpersonal implications which necessarily inflect the writing process and outcomes (Kaltenbrunner et al. 2022; Nästesjö 2021). In our case, given that our research design used social scientific theory and qualitative methods, we favoured interdisciplinary venues which lean towards the social sciences but target a wide multidisciplinary readership. The co-authored paper was ultimately presented at an interdisciplinary conference (FAccT) and published in a social science journal – the JRI (Domínguez Hernández et al. 2023).

Conclusion

Critical scholars have examined the social, political and epistemic entanglements and implications of AI, big data and algorithmic systems emphasising the need for more scrutiny, regulation and transparency (Gillespie 2014; Gorwa, Binns, and Katzenbach 2020; Kitchin 2014; Martin 2015). While this scholarship has contributed significantly to the debate on the need for algorithmic governance, questions around the utility of such critiques for practitioners concerned with responsible innovation remain largely unaddressed. The calls for operationalising AI ethics and responsible innovation have certainly permeated technology companies where significant efforts have been directed at establishing ethics committees, oversight boards, responsible AI programmes, and ‘fair’ technology research and development. Yet academics remain sceptical about the widespread solution-oriented view among technology companies and practitioners to tackle the ethical problems with data-driven technology (Bietti 2021; Hu 2021). Such a trust deficit risks devaluing responsible innovation as being merely an aspirational endeavour and demands that critical social science engage more closely with the technical practices, assumptions and incentives that undergird the construction of algorithmic systems. Very often approaches to AI ethics are viewed as either ‘toothless’ or deployed from a presumed privileged vantage point relative to that of practitioners, while offering limited or no alternatives (Rességuier and Rodrigues 2020). It is therefore becoming increasingly urgent to explore how critical studies of AI and other data-driven technologies can avoid being trapped between antagonising those involved in design and becoming instrumentalised as tick box or ‘ethics washing’ exercises.

Without offering to settle this dilemma, here we wished to approach it experimentally and collaboratively. We deployed STS insights and methods in an ML project to address a shared ‘matter of concern’: the harm that can be caused by misinformation and associated with this, issues of facticity and its relations with algorithmic classification of misinformation. Our intervention was an attempt at both critiquing and constructively influencing the development of an ML tool with the aim of minimising harm while attending to the social problem at hand. We were driven by an explicit commitment to help problematise, improve, adapt or reimagine AI systems that could have significant impacts on society whilst also considering our responsibilities, as social scientists, of doing such critical work. For instance, we were conscious of the risk that the intervention could be viewed as ethics policing or an ad hoc ethics review. This was no less risky when the concrete benefits of the collaboration were neither certain nor guaranteed from the outset. That said, the experimental setting embraces the uncertain and evolving nature of interdisciplinary collaboration and thereby lends itself to adjustment as needed (Greiffenhagen, Mair, and Sharrock 2015) and – in our case – an ‘in situ’ empirical tinkering with responsible innovation.

Pre-requisites for such collaboration we suggest are open-mindedness and mutual respect across the research team (for each other’s knowledge and expertise). Despite the uncertainties relating to the collaboration, we entered the challenge with the knowledge that our collaborators were not only open to interdisciplinary work but saw its potential benefits as a mutual learning venture.

We draw three practical and methodological lessons which we hope can be of use for interdisciplinary teams concerned with embedding responsible practices in AI/ML. First, in line with previous experiences with collaborative ethnography, we emphasise the importance of collective self-critique – what Bieler et al. (2021) term ‘distributing reflexivity’ – as a key step in moving from critique to action within responsible innovation pursuits. A way to support this would be for ML researchers to engage with those with knowledge and skills in reflective practice such as social scientists and STS scholars. This goes hand in hand with our second lesson on moving from a strategic vagueness during the initial stages of collaboration, through a phase of understanding (e.g. of the technical elements of the project) and reflection (on its contingencies) to a negotiation of pragmatic synthesis and tangible outcomes, even if these materialise as provisional findings and pointers to further joint projects. Finally, when involving collaborators in the process of inscription and in the co-production of the research outcomes, collaborative ethnography offers an opportunity for all participants to jointly craft, refine and ultimately own the critiques and calls to action. We found that evaluating the success of such interactive writing projects should focus not only on the final product, but on the possibilities for collaborators to learn from one another as part of the inscription process, recognising the power dynamics at play, and finding acceptable terms for collaboration, even in the face of disagreement.

Additional information

Funding

This work was supported by the UK Research and Innovation [grant number EP/V011189/1].

Notes

1 In the rest of the paper, we use ‘interdisciplinary’ as a broad concept, noting the number of related concepts that exist (e.g. ‘multi, ‘cross’ and ‘trans’ disciplinary). Rather than examining these existing definitions, here we wish to explore empirically some of the features commonly attributed to them: integrating, linking, interacting, transcending, transgressing and transforming the boundaries between and beyond disciplines. For further reading on the concepts of inter- and transdiscipilinarity, see Alvargonzález (2011).

2 In the case of autonomous vehicles for instance there has been a strong trend toward applying the trolley problem to determine legal liability and operationalise ethics in AI (see Wu 2020).

3 For example, as stated in the 2023 instance of AIES: ‘International organizations, governments, universities, corporations, and philanthropists have recognized this need to embark on an interdisciplinary investigation to help chart a course through the new territory enabled by AI. Earlier iterations of this conference and others have seen the first fruits of these calls to action, as programs for research have been set out in many fields relevant to AI, Ethics, and Society’ (AIES 2023, para. 1).

4 https://www.rephrain.ac.uk/responsible-innovation/.

5 We note that we do not claim there is a direct causal link between the current ‘post-truth’ phenomenon and the early practice of methodological symmetry within the field of STS (see Lynch 2020). We thank one of the anonymous reviewers for flagging this.

6 i.e. based on text and images.

7 See https://www.rephrain.ac.uk/about/.

8 AREA stands for ‘Anticipate, Reflect, Engage and Act’ (www.ukri.org/who-we-are/epsrc/our-policies-and-standards/framework-for-responsible-innovation/).

9 To be sure, the proposed ML model used annotated, multimodal data (i.e. texts and images) obtained from Google’s Fact Check Explorer – which aggregates claims that have been fact-checked by hundreds of news organisations around the world – as well as from Twitter, which provides access to data about user engagements with news.