Improving media trust research through better measurement: An item response theory perspective

Abstract

While trust in news media has come to the forefront of scholarly and public debate in recent years, academic researchers have raised persistent concern that measurement issues have prevented a better understanding of the concept. This research introduces an item response theory (IRT) perspective to advance the state-of-the-art in media trust measurement beyond recent conceptual and analytical progress. I argue that standard survey instruments that concentrate on the perceived believability of news media restrict our capability to measure truly low media trust. Furthermore, I suggest an important yet previously unnoticed pathway to overcoming this restriction in a scale by Abdulla et al. that captures currency perceptions alongside believability perceptions. Using a representative survey conducted in Germany, I find robust empirical evidence that capturing currency vs. believability perceptions significantly impacts our ability to accurately measure lower vs. higher levels of media trust. The findings have implications for not only studies of media trust’s associations with antecedent and consequential constructs but any attempt to determine the true amount and divergence of citizens’ media trust. More generally, the results demonstrate how IRT aids in putting scholarly debates on the dimensionality and interplay of trust with distrust on more common and fruitful grounds.

KEYWORDS:

• Media trust
• news media
• news credibility
• measurement
• item response theory
• confirmatory item factor analysis
• survey research

Data availability statement

The data that support the findings of this study are available on the Open Science Framework at https://osf.io/6c4tu/?view_only=06a3643ca73c45dca78a95d11ecd9b58.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 My account of IRT concentrates on its unique conceptual and analytical advantages over confirmatory factor analysis as the established state-of-the-art approach in (media) trust research (e.g., Besley et al., 2021; Cho, 2006; Prochazka & Schweiger, 2019). Readers with an interest in statistical basics are referred to the various existing excellent textbooks and reviews (e.g., DeMars, 2010; Edwards, 2009; Embretson & Reise, 2013).

2 This echoes what has traditionally been called range restrictions (in correlations) (Duncan, 1984; Ferguson, 1941). It mirrors that the factors’ correlations are contingent not only on the extent to which the respective item subsets are able to measure distinct ranges of the same latent trait but also on how the latent trait is distributed in the sample at hand. In our case, this means that the currency factor will be the more strongly correlated with the balance and honesty factors, the more often the participants’ media trust falls into the range that can be validly and reliably measured by all items comprising the Abdulla et al. scale.

3 Another important but more technical reason to expect that currency perceptions reflect lower media trust than do believability perceptions resonates with the mean differences reported in Abdulla et al. (2004), Yale et al. (2015), and Prochazka and Schweiger (2019). Although the researchers applied the Abdulla et al. scale in different media systems (i.e., the USA and Germany) and to rather distinct reference objects (i.e., newspapers, TV news, and online news; specific news articles; news media as a whole), their participants, on average, rated all reference objects as more current than believable. This pattern has not received much attention in prior scale tests. Yet, it can also be considered previous tentative evidence in line with H1 when approached through the lens of classic test theory (because higher mean values are an initial sign that items are relatively better suited to measure lower latent trait levels, as further outlined in the complementary test of H1 in the SM).

4 The media trust scores obtained with Meyer’s 5-item believability scale were more or less perfectly correlated with the media trust scores obtained by combining Abdulla et al.’s (2004) honesty and balance items into an 8-item scale, r > .99. This supported that the combination did not measure another concept. The results were also thoroughly robust against dropping single believability items (including the ones asking about ‘trustworthiness' and ‘fairness'; see also Yale et al., 2015).

5 As further explicated in the note to Figure 3, the tenet also received support because a bifactor model better fitted the data than all models in Figure 2.

6 In line with the general theoretical premise that alternative media use specifically concentrates among citizens who face mainstream media with outright distrust (rather than simply maintain a critical distance toward them), the variance in generalized media trust that was exclusively captured by the currency items (i.e., encapsulated the difference between very low and intermediary to modestly low trust) showed yet substantially stronger associations with the preference for alternative news media use, both before, r = [–.35; –.38], and after adjusting for demographics, r = [–.31; –.34].

7 A complementary test of H1 in line with the premises of classic test theory, which additionally integrated both the original US data by Abdulla et al. (2004) and the data by Prochazka and Schweiger (2019) and which yielded robust additional support for H1 (including its generalizability beyond the German case as well as from generalized trust in news media to trust in different types of news media), is provided in SM Table S4.

Additional information

Notes on contributors

Frank Mangold

Frank Mangold (Dr., University of Hohenheim) is a post-doctoral researcher in the Department of Computational Social Science at GESIS – Leibniz Institute for the Social Sciences. His research interests are audience structures and media repertoires, media behaviour and media-related dispositions, opinion leadership, and quantitative methods, especially data collection and statistical modelling.