Is Multi-Method Research More Convincing Than Single-Method Research? An Analysis of International Relations Journal Articles, 1980–2018

Abstract

While some social scientists see multi-method research (MMR) as a promising strategy for strong causal inference, others argue that it does little to strengthen the validity of research. This paper offers a systematic review of how MMR has been used in mainstream International Relations (IR) and specifically in security studies. Using the TRIP Journal Article Database and Web of Science citation data, I examine whether MMR has reached its full potential. MMR has grown in prominence since the 2000s. Scholars use it most often to examine domestic rather than interstate issues. They cite MMR articles less than they cite quantitative single-method articles and about as often as they cite qualitative single-method research. This suggests that MMR is not more influential, nor perceived as more persuasive. However, this gap has decreased in recent years. The study provides insights into IR at the research design and disciplinary levels, the utility of MMR, and knowledge accumulation in social science.

Keywords:

• methodology
• multi-method research
• research design
• International Relations
• security studies

Introduction: Multi-Method Research in International Relations

Multi-method research (MMR)¹ – as opposed to single-method research (SMR) – is empirical social science research that combines qualitative and quantitative evidence in pursuit of answering a single research question.² From the 2000s onwards, methodologists have identified a “boom” in or “turn” toward MMR in the social sciences in general and in the Political Science subfields of International Relations (IR) and Comparative Politics (CP) in particular.³ Proponents of MMR see well-executed designs as a promising strategy for strong causal inference, internal and external validity, policy recommendations, and as “superior to monomethod studies” when executed well.⁴ However, this is an “ideal […] that is often discussed but rarely achieved”:⁵ Others argue that combining methods from different methodological families does little to strengthen epistemological validity due to the approaches’ arguable incommensurability and poorer execution of the employed methods.⁶

MMR has received significant attention from an epistemological and methodological standpoint, but the practice of MMR in International Relations remains underexamined empirically. If “common wisdom” indeed “seems to suggest that [qualitative and quantitative evidence allow] the researcher to make inferences with greater confidence,”⁷ is MMR work in fact particularly influential and well-received in the discipline? Experimental evidence suggests that while social scientists indicate a preference for multi-method research in the abstract, they are not more likely to be convinced by concrete MMR designs than by SMR designs.⁸

This article contributes and advances these recent debates by providing the first large-scale account of MMR practice in IR and security studies, and by offering empirical evidence on how MMR articles are practically received.⁹ It does so by, first, tracing the development of MMR in IR over 39 years, examining which methods are most commonly combined, and identifying substantive issue areas that are over- and underrepresented in MMR articles. Second, it turns to the analytical question of whether MMR work is in fact cited more frequently than work using only a single method, which would suggest more convincing inferences and greater influence.

To answer these questions, the paper draws from the comprehensive and detailed 9.033-item Teaching, Research, and International Policy Journal Article Database (TRIP JAD; 1980–2018) and merges it with Web of Science (WoS) citation data.¹⁰ The analyses control for journal-, article-, and author-related confounders that may affect both research method choice and citation counts in IR and in general.

I find that while only 4.1% of all work in the full 39-year sample combines qualitative and quantitative methods, narrowly defined, this share has nearly doubled to 7.5% in recent years.¹¹ This confirms a “boom” of MMR in IR. MMR is most commonly used in studies examining national and sub-national topics (e.g., civil war, regime type, domestic politics) and less so for international phenomena (e.g., IGOs, international norms) and canonical foci of security studies (e.g., balance of power, interstate war, deterrence). This might reflect the origins of political science MMR in comparative politics while also suggesting that the approach can be more easily applied to some issue areas than others.

Analytically, I find that MMR articles are not cited more than other empirical work in IR, but rather that they are cited less than quantitative-only work and at roughly the same rate as qualitative-only studies. These results hold across different sampling choices and modeling strategies for the full 1980–2018 timeframe. The effects persist when examining security studies literature only. However, subset analyses of more recent timeframes indicate that these relative differences may be diminishing and a preliminary analysis of IR monographs (2000–2010) does not identify less citations of MMR books relative to quantitative SMR books. This may imply that MMR articles from the recent “boom” period are increasingly well-executed and therefore well-received. It might also indicate that MMR can be applied more usefully in book-length work. On the whole, the results suggest that article-length MMR is not more influential and convincing in IR than SMR work, supporting previous experimental research with large-scale observational evidence.¹²

MMR may best meet its potential in book-length studies and larger projects, rather than in article-length work. This aligns with recent appeals for placing a greater emphasis on a division of labor within research teams in IR, as well as on dedicated research synthesis, rather than expecting any individual paper to cover a larger range of methods and approaches.¹³ In examining these issues, this study also contributes to broader ongoing debates on empirical research design, knowledge cumulation, and research reception in the social sciences.¹⁴

This article proceeds as follows. First, I summarize arguments for and against using MMR in the social sciences, derive competing observable implications from these positions, and discuss my data sources and how I operationalize MMR. Second, I present a broad descriptive account of how MMR has been used in IR from 1980 to 2018. Third, I test whether articles that use MMR are more well-received and influential in IR than articles that employ SMR. Fourth and finally, I consider implications for IR research design and publication.

Expectations, Data, and Definitions

This section lays out recent arguments for and against using qualitative and quantitative evidence within a single study and derives competing observable implications from these arguments. I then outline the sample that I will examine. I use the TRIP JAD as a representation of the overall population of English-language, mainstream, article-length IR research. Finally, I discuss different understandings, definitions, and operationalizations of multi-method research. In this study, I opt for a narrow definition for combining qualitative and quantitative evidence. My definition refers to the most common approach of combining case studies and process tracing on the one hand with inferential statistics on the other.

The Arguments For and Against Multi-Method Research; Empirical Expectations

The heightened interest in MMR has been accompanied by numerous works on how to best conduct such research, combine methods, and formalize their relationship.¹⁵ Some researchers, however, remain doubtful of the utility of mixing approaches.

Slater and Ziblatt see “the careful attention to external validity and internal validity within a single study” as a key contribution of MMR.¹⁶ In this context, they argue that quantitative methods are commonly understood to provide external validity (i.e., results’ generalizability to numerous cases) while qualitative methods probe internal validity (i.e., the strength of a causal claim for the specific case under study). That said, either type of method can be used for either type of validity.¹⁷ Lieberman’s “nested analysis” aims at synergy and a division of labor between research methods.¹⁸ Similarly, Seawright suggests an “integrated” method, with several research methods combined into a single chain of causal inquiry answering a single research question. Hesse-Bieber et al. discuss how combining qualitatively- and quantitatively-driven approaches can balance the research goals of understanding and generalization.¹⁹ These approaches use the strengths of each method: single-case studies are used to probe causal mechanisms or generate theory, regression is used for cross-case causal inference and to explain variation, etc.²⁰ Furthermore, MMR approaches can foster the reflexivity necessary for a researcher to identify their own biases.²¹

From the perspective of epistemological validity, Ahmed and Sil offer a more skeptical view, arguing that MMR merely shifts the burden of reconciling findings of methodologically disparate work from the broader field to the individual researcher.²² Similarly, Beach and Kaas argue that while related tools – such as experiments and observational regression analysis – may be usefully combined, combining approaches from different methodological families is essentially impossible due to their fundamental epistemological incommensurability.²³ Gehlbach deems the combination of methods in any one piece of research unproductive and states that it often results in “research that does many things poorly.”²⁴ Instead, he suggests a disciplinary division of labor and narrower graduate training.

Furthermore, Avenburg et al. offer compelling experimental evidence showing that social scientists are not more likely to indicate being convinced by claims based on multi-method research than by single-method research.²⁵ They presented political scientists with generic and actual examples of SMR and MMR articles. The survey participants rated generic descriptions of MMR studies as more convincing than the descriptions of quantitative SMR and as about as convincing as descriptions of qualitative SMR. However, they rated actual MMR studies as more convincing than quantitative SMR and as about as convincing as qualitative SMR with respect to external validity, and as no more convincing than either type of SMR on internal validity in various contexts. The authors conclude that adding quantitative evidence to a qualitative study therefore “adds nothing to – and perhaps even detracts from – the persuasiveness of a study.” This “decisively refut[es]” their hypothesis that “multimethod evidence is superior for all (or most) purposes.”²⁶ The present study furthers this line of inquiry by using observational data from nearly 40 years of IR research. I thereby examine the extent to which these experimental results might also be borne out in IR research practice.

Both lines of argument are plausible and could work in parallel. The purpose of this paper is to provide an empirical account of the practice and reception of multi- and single-method research in IR and security studies. Based on these arguments for and against MMR, what observable implications can we expect regarding its influence and reception? This study uses citation counts as its measure of how useful, convincing, and influential a journal article is. Citation counts serve as a measure of “intrinsic quality,” a proxy for an article’s utility in persuading an audience, and scholars’ desire to be associated with the cited work.²⁷ They are also important from a professional perspective and are a key metric for career advancement in contemporary academia.²⁸ An observable implication of MMR articles indeed being more convincing than SMR articles would be higher citation counts, all else equal. Conversely, if MMR is difficult to employ persuasively in journal-length work, we would expect it to receive less citations than other empirical work. I discuss theories regarding citation counts and competing empirical expectations in the analytical section of this paper.

The Sample: Twelve Journals, 1980–2018

This paper examines the usage of MMR in IR by using the TRIP Journal Article Database (TRIP JAD), which codes articles from twelve journals from 1980–2018 for a broad range of characteristics.²⁹ The database includes eight IR field journals and four general Political Science journals. For the former, issues 1–4 of every year are coded entirely (with special issues excluded); for the latter, only the articles deemed by TRIP coders to be IR-themed are included. I merged this database with citation and further bibliometric data extracted from Web of Science (WoS) – deemed to provide fairly high-quality data³⁰ – for a total sample of 9.033 articles. Table 6 in the Appendix summarizes the make-up of the sample by journal and year.³¹

The sample covers an array of substantive and methodological traditions.³² It offers some geographic diversity across the US, UK and continental Europe.³³ In this respect, some of the examined journals are globally visible journals that are open to non-North American authors.³⁴ Nonetheless, the sample is still limited to English-language IR and does not include critical journals or English School approaches.³⁵ Although there are no immediate alternatives to the analyzed data, this reflects and thereby perpetuates a Western hegemony in IR.³⁶ That said, the included journals are nonetheless among the most influential in the field, as judged by IR scholars from all world regions.³⁷

The format under examination – journal articles – raises another issue: Beach and Kaas note that contemporary book-length IR scholarship is expected to be multi-method.³⁸ By comparison, articles may not offer enough space to be rigorously multi-method and harness the potential of convincing causal inference through MMR. Nonetheless, publishing in journals is one the two most important “practice[s] of a scientific discipline” and ways of academic career advancement.³⁹ Therefore, journal articles and citation data remain a “useful entry point” to examining the usage of MMR in IR empirically. ⁴⁰ To probe generalizability, I also present a short analysis of IR monographs later in this article.

These considerations limit the broader population that this sample represents, and for which this paper draws generalizable conclusions: It is mainstream, highly visible, English-language, article-length IR scholarship.⁴¹

Definitions and Operationalizations of MMR

Scholars often use “method” and “methodology” interchangeably. It is worth briefly differentiating the two: methods are specific “tools and techniques,” while methodologies are “families of methods” with similar epistemological underpinnings.⁴² TRIP codes “methodology” as an article’s means of systematically linking independent and dependent variables to test hypotheses, not the precise means of data gathering (e.g., archival research) or analysis (e.g., time series analysis).⁴³ This paper employs these categories, which are well-coded, clearly differentiated, and therefore useful in answering the research questions at hand. An additional benefit of using TRIP-coded data is that it does not rely on authors self-labelling their research as MMR to potentially appeal to editors and readers.

Based on the TRIP categories, I propose three different operationalizations of multi-method research. I use the narrowest of the three for the subsequent analyses. Table 1 lists the eight methods labels featured in the JAD with brief descriptions from the codebook.⁴⁴

Table 1. Simple definition of MMR – TRIP methods categories and brief descriptions.

Method	Description
Quantitative	regression, factor analysis, ANOVA, etc.
Qualitative	“primarily case studies”
Formal modeling	formal mathematical equations, decision trees, etc.
Counterfactual	individual thought experiments^45
Analytic/conceptual	“illuminat[ing] features of IR” without empirics or a formal model^46
Descriptive	“describ[ing] (…) trends or events”^47
Policy analysis	“evaluation of options available to policy makers”
Experimental	experiments and simulations

First, a simple definition covers all articles that use more than one of the eight TRIP categories.⁴⁸

Second, a clustered definition combines related methodological groups from the TRIP database, roughly applying the above definition of ‘methodology.’⁴⁹ Table 2 lists two clusters (based on the descriptions in Table 1) which reflect this usage, which Goertz and Mahoney call the “two cultures” of research.⁵⁰ Empirical work shows that while quantitative work in Political Science is quite coherent as it relates to the quantitative “culture,” practices in qualitative work are more diverse.⁵¹ Furthermore, qualitative and quantitative approaches are difficult to differentiate as separate “cultures” of published research on features beyond the specific methods they use (e.g., on notions of causality, case selection, and generalizability).⁵²

Table 2. Clustered definition of MMR – clusters of TRIP methods categories.

Quantitative cluster	Quantitative, formal modeling, experimental
Qualitative cluster	Qualitative, counterfactual
Neither	Analytic/conceptual, descriptive, policy analysis

Third and finally, a narrow definition of MMR covers only the “quantitative” and “qualitative” labels in the TRIP data (see Table 1). This defines an article combining case studies and regression analysis on observational data (and using no further methods) as MMR. Goertz suggests that the “typical understanding” of MMR in Political Science is a combination of a case study and a quantitative or formal model.⁵³ Likewise, Rohlfing and Starke point to the combination of regression and single-case process tracing as the canonical MMR design.⁵⁴ In their thorough experimental treatment of researcher-level MMR reception in Political Science, Avenburg, Gerring, and Seawright also use a narrow definition of this sort, combining inferential statistics on the one hand with process tracing or case studies on the other. They do not consider experiments in their operationalization of quantitative research and do not cover formal theory.⁵⁵

As the little empirical work that has been done explicitly on the use of MMR in the social sciences uses this narrow definition, this study will do the same to ensure comparability. However, it is worth pointing out that this risks oversimplification and may marginalize work beyond conventional quantitative and qualitative studies.⁵⁶ The simple and clustered definitions are therefore used in robustness checks and for description. To provide the reader with an impression of this narrow MMR label and the underlying data, Table 7 in the Appendix displays the ten most-cited MMR-employing articles using this definition, and Table 8 in the Appendix shows ten randomly selected articles. The articles in both tables dedicate significant parts of their manuscripts to offering both quantitative and qualitative evidence, indicating that the combination of TRIP variables proposed above indeed captures the concept of MMR.

Descriptive Analysis: MMR in IR Over the Years, Topics, and Combinations

This section provides descriptive evidence on MMR and research methods in IR and security studies over the years, while also giving an impression of the sample used in the subsequent analyses.

MMR Over the Years and by Journal

Has there indeed been an MMR “boom” in mainstream IR?⁵⁷ Figure 1 illustrates the share of MMR articles among all articles in the sample from 1980–2018. Examining the narrow definition, MMR articles have grown over time, doubling from 2.4% for 1980–2000 to 4.5% in the early 2000s (2001–2013), and then again increasing to 7.5% in the five most recent years (2014–2018). The annual share of MMR work has consistently been around or above 5% since about 2001. This shows the increase in MMR other scholars have suspected.

Figure 1. Share of MMR in IR, 1980–2018; narrow, clustered, and simple definitions. (note: the three categories are nested within each other, from narrow to clustered to simple, so the bars are not stacked, but rather grouped with overlap.).

To further examine the sample and mainstream IR as it relates to MMR, Table 3 displays the shares of MMR, qualitative and quantitative SMR, purely conceptual work, and other approaches in each of the sample’s twelve journals.⁵⁸ Across the entire sample, 4.1% of articles mix qualitative and quantitative evidence, narrowly defined. About one in ten articles in WP has historically done so, followed by one article in 18 in IO. The other end of the table shows that journals from both quantitative and qualitative traditions can be similarly reluctant to publish MMR articles, with JPR, the American Journal of Political Science (AJPS), and EJIR having the lowest MMR article shares. Journals in the sub-field of security studies publish MMR more rarely than others do.

Table 3. Shares of MMR articles in journals (narrow definition).

		Shares
Journal	Articles	MMR	SMR Quantitative	SMR Qualitative	Conceptual only	All Other
WP	724	9.9%	21.1%	37.9%	11.5%	19.6%
IO	1059	5.5%	26.4%	35.0%	13.2%	19.9%
JOP	320	5.0%	63.8%	3.8%	5.9%	22.2%
BJPS	222	5.0%	50.9%	10.4%	6.8%	27.0%
ISQ	1388	3.8%	44.2%	21.6%	12.0%	18.4%
JCR	1106	3.7%	47.4%	4.2%	4.2%	40.6%
APSR	370	3.5%	40.5%	7.8%	13.5%	34.6%
IS	1012	3.5%	3.3%	39.2%	11.2%	42.9%
SS	647	2.9%	3.9%	59.0%	17.5%	16.7%
JPR	1248	2.9%	37.4%	21.1%	13.4%	25.2%
AJPS	327	2.1%	64.8%	1.5%	2.8%	28.8%
EJIR	611	1.8%	6.9%	48.9%	33.1%	9.3%
Total	9033	4.1%	31.2%	26.6%	12.4%	25.7%

Substantive Focus

Which issues do scholars study through an MMR lens? Table 4 displays the shares of MMR among TRIP’s 36 substantive focus areas for the entire timespan (1980–2018) and the ten most recent years (2009–2018).⁵⁹ Articles that examine sub-national phenomena and dynamics – such as civil war and crises, aid and development, elections, regime type, or domestic interest groups – seem to make greater use of MMR. In general, work that uses the state as unit of analysis – such as work on interstate war, regional integration, or balance of power – makes less use of MMR. Relatedly, many substantive foci traditionally examined in security studies are also less likely to be examined in an MMR design.

Table 4. Substantive focus areas by share of MMR, narrow definition (1980–2018, 2009–2018).

	1980–2018			2009–2018
	Articles	MMR	Share	Articles	MMR	Share
Ethnicity, Religion	480	46	9.6%	330	40	12.1%
Foreign Aid	354	29	8.2%	169	13	7.7%
Migration	152	12	7.9%	87	6	6.9%
Intrastate Conflict	1234	89	7.2%	620	56	9.0%
Gender	139	9	6.5%	108	6	5.6%
Domestic Politics	2675	166	6.2%	861	66	7.7%
Public Opinion	765	47	6.1%	360	27	7.5%
North–South Relations	283	17	6.0%	53	4	7.5%
Trade	1017	61	6.0%	359	24	6.7%
Regime Type	1236	74	6.0%	526	40	7.6%
Development	611	36	5.9%	161	15	9.3%
Monetary Policy	225	12	5.3%	59	2	3.4%
International Law	404	21	5.2%	177	14	7.9%
Terrorism	415	21	5.1%	262	19	7.3%
Hum. Intervention, Peacekeeping	398	20	5.0%	188	14	7.4%
Sanctions	127	6	4.7%	49	3	6.1%
Economic Interdependence	809	37	4.6%	269	16	5.9%
Interstate Crisis	891	38	4.3%	312	20	6.4%
Other	1824	77	4.2%	403	23	5.7%
All articles	9033	372	4.1%	3125	191	6.1%
Regional Integration	345	14	4.1%	111	7	6.3%
Environment	223	9	4.0%	100	4	4.0%
Foreign Policy	1493	60	4.0%	312	23	7.4%
IGOs	1238	49	4.0%	469	25	5.3%
Interstate War	1968	72	3.7%	564	34	6.0%
Alliances	560	20	3.6%	152	12	7.9%
NGOs	367	11	3.0%	157	7	4.5%
Weapons Systems	892	24	2.7%	148	11	7.4%
International Regimes, Norms	1238	33	2.7%	439	16	3.6%
Diplomacy	396	10	2.5%	125	6	4.8%
Bargaining, Deterrence, Strategy	1382	31	2.2%	365	18	4.9%
Balance of Power	945	21	2.2%	272	12	4.4%
WMD Proliferation	503	11	2.2%	117	8	6.8%
IR as a Discipline	740	3	0.4%	233	1	0.4%
Public Health	43	0	0.0%	30	0	0.0%

The columns for 2009–2018 show recent trends and probe whether MMR-prone issues are merely the product of substantive and methodological changes in IR over time. For example, an issue such as the balance of power, which was more salient during the Cold War, may simply have a low share of MMR articles because MMR was not as prominent in the 1980s. Table 4 shows that this does not seem to be the case. The stronger domestic focus might reflect the origins of political science MMR in Comparative Politics while also suggesting that the approach can be more readily applied to some topics and research questions than to others.

Which Methods are Commonly Combined?

How prominent are different research methods in IR, and which methods do scholars commonly combine? Figure 2 plots the standalone usage and co-occurrence of the eight methods labels from the TRIP database (i.e., the simple definition). 1,220 articles combine exactly two of these methods (13.5%). Within these, as the MMR literature suspects, the most common combination is regression and case studies (372 articles, amounting to 4.1% of all articles), followed by regression and formal modeling (269 articles or 3.0%), and regression and experiments (132 articles or 1.5%).⁶⁰ Journal articles that combine more than two of the eight methods categories – regardless of whether these are quantitative or qualitative – are very rare (75 articles or 0.8%).

Figure 2. Co-occurrence of research methods in IR (simple definition).

Figure 3. Conditional effects, journal set to Security Studies.

Figure 4. Conditional effects, journal set to International Organization.

Are MMR Articles More Influential in IR than SMR Articles are?

Building on these descriptive findings, the analytical part of this paper uses the narrow definition of MMR for its analyses. It follows similar analytical procedures as Maliniak, Powers, and Walter’s work on the gender citation gap in IR.⁶¹ The unit of analysis is the journal article.

Dependent Variable: Citation Count

I am interested in the observable influence of MMR articles, and how successful journal-length MMR articles might therefore be in IR in general and in security studies in particular. I use an article’s citation count to assess its influence. I employ data from WoS, rather than Google Scholar, because WoS only captures citations in peer-reviewed work.⁶²

While “epistemological validity” and citations are distinct features of an article, I assume that they are nonetheless meaningfully associated with one another.⁶³ The following summarizes why citations, despite limitations, usefully reflect how convincing, memorable, and influential⁶⁴ a particular article is, controlling for other factors. Kristensen summarizes three theories of citation practice taken from sociology of science and applies them to IR.⁶⁵ On the one hand, normative citation theory sees citations as rewards for high-quality work and the “intrinsic quality” of the cited article.⁶⁶ On the other hand, constructivist citation theory interprets cited articles as “the most useful persuasive devices in the defense of knowledge claims” tailored to the author’s audience (e.g., editors, reviewers, readers), at times in a “cynical” manner.⁶⁷ Often, this means that authors rely on prominent scholars in defense of an argument.⁶⁸ An intermediate position, symbolic theory, interprets citations as “symbolic identity marker[s] for position-taking,” indicating a desire to be associated with the cited work.⁶⁹ Given these multiple factors that can drive citation, the analyses account for a variety of observable sociological factors (see below).

While citation counts reflect the quality and utility of a paper in these ways, they also capture other features. Citations can indicate how interesting a paper is – be it due to a major contribution or a notable error.⁷⁰ Articles may also be cited for coining a useful phrase, as straw man arguments only loosely related to the original article, or as a controversial foil.⁷¹ Ultimately, citation counts are a direct measure of the practice of scholarship. As used here, they are intended to “reveal some important information” about the specific disciplinary setting of IR.⁷² Across academic fields, citations count the number of other papers to which the cited paper has proven “relevant and useful” and whether scholars “wish to associate their own work with it.”⁷³ Citations are therefore here generally understood as revealed preferences for that work.

Independent Variable: MMR or SMR

Based on the data, narrow definition, and descriptive evidence presented above, I operationalize the key methods variable categorically. It can take five values: “MMR,” “SMR quantitative,” “SMR qualitative,” “conceptual only,” or “all other.” One could also examine whether an MMR approach adds explanatory utility on top of and in addition to the respective usefulness of each individual approach; qualitative and quantitative. This could potentially be explored by using an interaction term or adopting a Structural Equation Modeling (SEM) approach. I pursue a simpler categorical option. Substantively, this conceptualizes article-length MMR as an altogether different research approach than standalone qualitative or quantitative studies.⁷⁴

Expectations for the Relationship between MMR and Citation Count

Building upon the arguments for and against MMR, how might we expect the combined use of both qualitative and quantitative methods to affect citation counts? Different views on MMR present competing hypotheses.

On the one hand, MMR articles may be cited less than SMR work. There are at least two concerns inherent in the mixed methods enterprise: commensurability and specialization.⁷⁵ First, issues of incommensurability can lead to conflicts in adjudicating research choices, in turn making it difficult for scholars to draw clear conclusions from an article’s results. Second, regarding specialization, MMR approaches may satisfy neither qualitative nor quantitative researchers. Given increasing methodological specialization and sophistication, scholars attempting to stay up to date on disparate methods may find it hard to stay at the cutting edge of any of them.⁷⁶ Finally, article-length work may not offer enough space to delve into multiple methods in convincing depth.

On the other hand, MMR work may be cited more than SMR articles. If MMR indeed has the potential to be particularly convincing, then we might expect MMR designs to be positively associated with citation counts. Multiple empirical sections might also give an article a greater chance at being memorable by appealing to multiple audiences.

Potential Confounders and Control Variables

I control for several factors that have been hypothesized or shown to affect both citation counts and research method choice in IR. These can be split into three categories:⁷⁷ First, features of the specific paper (e.g., issue area, article age, citations contained in the article itself); second, the journal’s characteristics (e.g., editorial practices, journal language, journal circulation and prestige); and third and finally, author-related factors (e.g., number of authors, author reputation and rank, author gender).⁷⁸ This paper controls for many of these factors, based on the data compiled by Maliniak, Powers, and Walter, and adds further potential confounders drawn from WoS.⁷⁹

Differences in citation counts between qualitative and quantitative work may also be driven by conventions and writing styles in IR sub-fields. Some fields might cite more and be more insular and self-referential, while others may delve into other disciplines more readily at the expense of referencing IR. Quantitative-only articles in the sample include 58 references on average, MMR articles contain 70, and qualitative-only articles contain 74.⁸⁰ I control for the number of citations contained in an article and add journal dummies in order to account for these potential sub-disciplinary conventions.

Because the practice of “drive-by citation” or simply citing certain canonical papers as a sub-disciplinary convention may skew citation counts, I also use sub-samples in which I remove highly-cited articles.⁸¹ Confounders such as self-citation (which is more frequently pursued by men), informal citation groups, and “reviewer-coerced citations” are also of concern.⁸² I cannot control for these.

Model Choice and Accounting for Time

The DV of citation counts is a highly dispersed count variable with a log-normal distribution. Therefore, I use a negative binomial model, as is commonly done for citation data.⁸³

In the social sciences, a paper’s peak in annual citations generally occurs in year two and three after publication.⁸⁴ I extracted citation data from WoS on 13 January 2023. TRIP covers articles published from 1980–2018. Articles from 2018 had therefore passed the initial citation stage, with at least four full years to be cited. To account for time effects, all regression models include a “years since publication” control and its squared term, as well as year fixed effects to account for idiosyncrasies by publication year.⁸⁵ The publication year is the journal issue’s year, rather than online-first dates, which are not consistently available.

Sub-Samples and Analysis Procedure

I ran regression models for four different sub-samples. First, I analyzed the entire sample of merged TRIP/WoS articles, containing 9.033 items. Second, I used a sub-sample that excluded highly-cited articles. This tackles citation phenomena such as “bursts” and “theoretical fashions” that may skew the sample statistically.⁸⁶ This sub-sample is taken to reflect ‘ordinary’ IR articles. Maliniak and colleagues suggest a threshold of three standard deviations above the mean.⁸⁷ The 9,033-item sample is thereby reduced by 109 articles to 8,924. A third sub-sample contains only empirical articles. As MMR articles are by definition empirical, this sub-sample excludes purely conceptual and primarily methodological articles.⁸⁸ This assumes that empirical work is different in kind compared to purely theoretical, conceptual, and methodological work as it relates to epistemological validity, citation dynamics and opportunities, and therefore comparability.⁸⁹ Theoretical work is generally the most-cited work in IR.⁹⁰ Similarly, the “articles of record” for new datasets or methods notes make different contributions than empirical work does, and may therefore be cited differently and be less comparable. This reduces the 9,033-item sample by 1,304 to 7,729 empirical articles. Fourth, I apply these two criteria simultaneously, for a sample of 7,664 empirical IR articles that are not among the most highly-cited.

The models for these four samples follow the same steps. The approach is in line with Maliniak et al.’s self-admitted “kitchen sink” procedure for examining citation counts in IR.⁹¹ The initial baseline model contains only the variables surrounding author characteristics, article age, number of authors, journal fixed effects, and so on.⁹² Next, variables on IR paradigm and thematic issue areas are added in the substantive model. Third, the methods model adds TRIP’s eight methods categories. Finally, the MMR model replaces the eight methods categories, using the narrow categories “MMR,” SMR Qualitative,” “SMR Quantitative,” “Conceptual only,” and “All others” (see Table 2). “SMR Quantitative” is used as the reference category. This final ‘narrow’ model is most relevant to answering the research question and therefore reported and discussed in detail for all four subsets. All other subset–model combinations are reported in the Online Appendix.

To probe the specific situation of MMR articles in security studies, I repeat the described procedure limited to articles on international security. Finally, I run the analyses for more recent timeframes to examine the extent to which MMR reception may have changed over time.

Results and Model Interpretation: Full Sample

Table 5 shows the study’s main results, the results of the negative binomial regression models for the four sub-samples when using the narrow MMR variable introduced above.⁹³ The coefficients change with the different sub-samples from Models 1–4, but the substantive results are similar across the board. Tables A1–A4 in the Online Appendix show the full regression results for the four main subset–model combinations (for a total of 16 models).⁹⁴

Table 5. Main results, full negative binomial models, different samples.

	DV: total citations
	Model 4a	Model 4b	Model 4c	Model 4d
	Full	<3 SD cit.	Emp. only	<3 SD cit. + Emp.
Constant	1.507***	2.680***	1.659***	1.750***
	(0.429)	(0.125)	(0.442)	(0.417)
Year since publication	0.360***	0.156***	0.327**	0.320***
	(0.099)	(0.007)	(0.101)	(0.097)
Year since publication^2	−0.007***	−0.003***	−0.007**	−0.006**
	(0.002)	(0.000)	(0.002)	(0.002)
Number of references	0.008***	0.006***	0.007***	0.006***
	(0.000)	(0.000)	(0.000)	(0.000)
Coauthored (Y/N)	0.128**	0.106**	0.081+	0.122**
	(0.045)	(0.041)	(0.046)	(0.039)
Author(s): all female (ref: all male)	0.116*	0.164***	0.109*	0.148***
	(0.046)	(0.044)	(0.046)	(0.043)
Author(s): mixed gender	0.041	0.068	0.045	0.048
	(0.055)	(0.052)	(0.055)	(0.051)
Tenured (Y/N)	0.158***	0.152***	0.131**	0.081*
	(0.042)	(0.036)	(0.045)	(0.035)
Tenured female (Y/N)	0.025	−0.027	0.019	0.015
	(0.066)	(0.058)	(0.067)	(0.059)
R1 university (US, Y/N)	0.171***	0.164***	0.193***	0.175***
	(0.038)	(0.035)	(0.039)	(0.034)
Journal: WP (ref: IO)	−0.241**	−0.099	−0.216**	−0.129*
	(0.076)	(0.065)	(0.079)	(0.061)
Journal: JCR	−0.477***	−0.300***	−0.431***	−0.332***
	(0.076)	(0.065)	(0.081)	(0.062)
Journal: IS	−0.616***	−0.482***	−0.556***	−0.475***
	(0.076)	(0.062)	(0.083)	(0.067)
Journal: EJIR	−0.787***	−0.554***	−0.720***	−0.581***
	(0.079)	(0.066)	(0.083)	(0.072)
Journal: JPR	−0.793***	−0.662***	−0.732***	−0.637***
	(0.071)	(0.057)	(0.072)	(0.060)
Journal: ISQ	−0.822***	−0.624***	−0.782***	−0.628***
	(0.067)	(0.053)	(0.071)	(0.054)
Journal: SS	−1.611***	−1.367***	−1.547***	−1.394***
	(0.077)	(0.070)	(0.082)	(0.074)
Journal: APSR	0.346***	0.298***	0.440***	0.433***
	(0.105)	(0.069)	(0.110)	(0.074)
Journal: AJPS	−0.160+	−0.010	−0.051	0.069
	(0.084)	(0.069)	(0.089)	(0.074)
Journal: BJPS	−0.686***	−0.601***	−0.551***	−0.491***
	(0.107)	(0.085)	(0.110)	(0.090)
Journal: JOP	−0.680***	−0.515***	−0.615***	−0.495***
	(0.080)	(0.073)	(0.084)	(0.074)
Epistemology: Non-/Post-Pos. (ref: Pos.)	−0.266***	−0.196***	−0.322***	−0.261***
	(0.060)	(0.056)	(0.073)	(0.066)
Material (Y/N)	−0.012	−0.107	0.010	−0.057
	(0.076)	(0.074)	(0.083)	(0.078)
Ideational (Y/N)	0.058	0.048	0.048	0.032
	(0.036)	(0.031)	(0.038)	(0.031)
Paradigm: Constructivist (ref: Realist)	0.150	0.027	0.090	−0.036
	(0.100)	(0.074)	(0.120)	(0.077)
Paradigm: Non-paradigmatic	−0.099	−0.093+	−0.113	−0.104+
	(0.064)	(0.056)	(0.071)	(0.058)
Paradigm: Liberal	−0.154*	−0.147*	−0.172*	−0.158*
	(0.078)	(0.069)	(0.084)	(0.066)
Paradigm: Marxist	−0.358*	−0.251+	−0.510**	−0.475**
	(0.153)	(0.141)	(0.158)	(0.146)
Paradigm: Atheoretic/non-paradigmatic	−0.277**	−0.211*	−0.301*	−0.252*
	(0.102)	(0.096)	(0.118)	(0.107)
Qual. only (ref: Quant. only)	−0.237***	−0.261***	−0.210***	−0.179***
	(0.049)	(0.044)	(0.053)	(0.044)
MMR	−0.181**	−0.215***	−0.164**	−0.164**
	(0.061)	(0.055)	(0.061)	(0.054)
Conceptual/theoretical only	0.128	−0.115
	(0.081)	(0.071)
All others	−0.203***	−0.298***	−0.219***	−0.274***
	(0.053)	(0.047)	(0.056)	(0.046)
Num.Obs.	8498	8389	7343	7278
AIC	83136.2	80609.3	71316.4	69581.5
R^2	0.019	0.111	0.016	0.120

⁺p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Note: Issue Area controls and year fixed effects not displayed. Robust SEs.

For negative binomial regressions, the coefficient represents the natural log of the expected count of citations.⁹⁵ The coefficient thus shows the expected difference in the dependent variable (on a logarithmic scale) for each additional unit of the independent variable.⁹⁶

Model 1 covers the full sample using the controls and the narrow MMR definition. The results suggest that an exclusively qualitative article receives about 80% of the citations that an exclusively quantitative article receives (e^−0.220 = 0.789, a 21.1% difference). Similarly, MMR work is estimated to receive 83.4% of the citations that an exclusively quantitative articles receives, all else equal.

Model 2 omits highly-cited articles. MMR articles again receive fewer citations than quantitative SMR articles (81%), and qualitative SMR articles fare similarly (77%). Model 3 limits the sample to empirical articles only, dropping conceptual and methodological work due to their different citation dynamics. Qualitative and MMR articles receive similar citation estimates relative to quantitative SMR (at 81% and 85%, respectively). Finally, Model 4 uses perhaps the most appropriate sample for answering the research question, comparing only empirical work that is not highly cited. Once again, MMR work is cited significantly less than quantitative SMR (85%), as is qualitative SMR (84%).

These observational results are similar to Avenburg et al.’s experimental results in that they find that MMR work is not cited more than either type of SMR. Interestingly, however, the relative positions of MMR and qualitative and quantitative SMR articles are different.⁹⁷ While the Avenburg et al. experiment finds that qualitative evidence generally fares best, the analysis of citation patterns presented here shows quantitative SMR on top. Assuming that I have sufficiently controlled for subfield citation conventions, this indicates a puzzling discrepancy between immediate research reception on the one hand and writing and publishing practice on the other.

The results from Models 3 and 4 are illustrated as conditional effects in Figures 3 and 4.⁹⁸ The journal is set to Security Studies and International Organization (the most balanced journal in the sample in terms of MMR and the two SMR categories, see Table 3), and the year to 2010, alongside further covariate values common in the field of security studies.⁹⁹ These settings again show that quantitative SMR articles are cited more frequently than both qualitative SMR and, most interestingly, MMR articles. The latter two are cited at similar rates. Furthermore, the 95% confidence intervals show the relative uncertainty when estimating citation counts.¹⁰⁰

Results and Model Interpretation: Security Studies

Following Hoagland et al., Table 9 in the Appendix replicates the main results table (Table 5) for a subsample of articles to analyze the issue area of international security broadly understood (n = 3,790).¹⁰¹ This probes the influence of MMR articles in security studies in particular.¹⁰² The same pattern emerges, although it is somewhat more pronounced: quantitative SMR papers are again estimated to receive the most citations, with qualitative SMR and MMR papers receiving less (at 85% and 83%, respectively) for the subset of empirical articles that are not highly cited (Model 8).

Results and Model Interpretation: Recent Years

Finally, how might these trends have changed over time? Tables A5 and A6 on the Online Appendix contain the results of two subset analyses. The first examines 2001–2018, as this study’s descriptive analyses identified a fairly consistent level of MMR in about 5% of all articles from 2001 onwards. The second table zooms in on the ten most recent years, 2009–2018. The analyses indicate that the overall effects identified above have likely weakened, in that the citation gap between MMR papers and quantitative and qualitative SMR papers has shrunk. The 2001–2018 analysis shows somewhat smaller effects for the MMR and SMR variables. The 2009–2018 analysis shows no statistically significant effects for the methods variables. This suggests that the reception of MMR relative to quantitative and qualitative SMR articles may be changing over time, as MMR gets further established and the execution of MMR designs improves, aided by the growing methodological MMR literature.

Robustness Checks

I reran the main regressions using an alternative procedure recommended in some parts of the bibliometric literature.¹⁰³ These robustness checks are displayed in the Online Appendix (Tables A7–A10). The checks show similar results, with some differences in significance and effect size. Once again, qualitative SMR articles are cited less than quantitative SMR articles. MMR work is also cited less than quantitative SMR work. Following this alternative procedure, the effect of MMR on citations does not track as closely to the “qualitative-only” effect as it does in the main analysis in Table 5 and as displayed in Figures 3 and 4. Rather, MMR takes a middle position between the two. This is substantively in line with the main models’ results in that MMR again does not have a stronger positive effect on citations than either type of SMR does.

When subsetting the main analyses by journal, the SMR/MMR methods variables largely retain the same signs, although in most cases the sample sizes (see Table 3) are too small to detect statistically significant results.

The results of the main analyses also hold when using the clustered rather than the narrow MMR definition laid out above. These results are shown in the Online Appendix.

Generalizability: Comparative Politics and IR Monographs

How might these findings on MMR in IR generalize to Political Science more broadly? There are some indications in the presented analyses that MMR might be used differently in IR than in other subfields and that other fields offer different empirical opportunities for combining methods. For instance, World Politics – which may be considered more of a CP than an IR journal – publishes more MMR articles than the dedicated IR journals do (recall Table 3). In the examined sample, domestic political issues are also more commonly studied using MMR than are classic IR issues (see Table 4). Furthermore, much of the methodological MMR literature reviewed above comes from CP. This all suggests that comparativists may handle MMR differently than IR scholars do. Table 10 in the Appendix shows the results of the main analyses when the issue area is limited to Comparative Politics as per the TRIP coding (n = 1,029). The analyses show a negative effect of MMR and qualitative SMR relative to quantitative SMR, but fall short of conventional statistical significance levels for the MMR variable. This suggests that MMR may be received similarly in CP as it is in IR, but this initial finding requires further investigation.

Do these findings also apply to research output beyond journal articles, such as books? With far more space to develop an argument and lay out empirics, MMR might be used more convincingly in book-length work. I conducted an analysis of Sharman and Weaver’s 500-item IR monograph dataset (2000–2010).¹⁰⁴ I find that MMR is about as common in monographs as it is in journal articles, with 4.5% (21 books) and 4.1% (372 articles) of publications using MMR narrowly defined and 16.9% (83 books) and 14.3% (1,295 articles) using MMR in its simple definition, respectively. Figure A1 in the Online Appendix plots the co-occurrence of research methods in monographs. Running versions of this article’s main regressions and using Google Scholar citations¹⁰⁵ shows that the narrow MMR variable has no clear relationship to citations. However, while falling short of conventional statistical significance (p = 0.140), the simple MMR variable has a positive association with citation counts (+0.216). A larger sample of IR monographs may provide more conclusive insights on this potentially positive relationship.

Conclusions: MMR Practice in IR

What do these results tell us about MMR practice in IR and security studies? This paper set out to first provide a comprehensive overview of MMR practices over the past 40 years and then to examine whether MMR articles are cited more than SMR articles, thus implying greater influence.

The use of regression/case study MMR designs has indeed grown in IR over the decades, at 4.1% for the entire timeframe and 7.5% in the five most recent years (2014–2018). This confirms the suspected MMR “boom” in mainstream, English-language IR.¹⁰⁶ MMR approaches are most commonly applied in research that examines sub-national phenomena and less so regarding issues that use the state as the main unit of analysis (e.g., interstate war, regional integration) or that focuses on security studies issue areas (e.g., WMD proliferation, weapons systems). This highlights avenues for potentially innovative research designs on these issues.

Turning to the analytical question of whether MMR work is cited more than qualitative or quantitative SMR work, I presented analyses combining different article and timeframe subsamples and potential confounders. The models and robustness checks indicate that MMR work is cited less than quantitative-only work and at about the same rate as qualitative-only work, all else equal. This corroborates previous experimental work on the persuasiveness of MMR evidence relative to SMR.¹⁰⁷ However, this trend may be changing, as subsamples for more recent years show that these discrepancies are shrinking. Perhaps MMR from the recent “boom” is increasingly well-executed and therefore well-received.

Further Potential Modes of Inquiry

Clearly, citation data are not the final word on how convincing or epistemologically valid MMR is. The goal of IR research is not amassing citations. Scholars should adopt other approaches to determining the efficacy of MMR in IR research. Broadening the scope of the journals under examination would be beneficial, including more critical journals and journals based beyond the US and Western Europe.¹⁰⁸ Scholars should extend the preliminary examination of IR monographs presented above. Furthermore, more sophisticated tools could be used to examine this question, such using matching methods on text to adjust for confounding.¹⁰⁹

The question of how convincing MMR is relative to other empirical strategies should be examined through multiple methods. This paper has offered a descriptive and observational, quantitative, regression-based approach. Surveys on the topic could probe the views of scholars who are conducting MMR, as well as scholars who are reading and reviewing MMR work. Interviews might be useful in surveying institutional pressures regarding publication and review, as well as concerning the ‘file drawer problem’: Qualitative and qualitative research components pointing to different conclusions may go unreported, as these types of ambiguous results are less compelling and therefore harder to publish. Finally, scholars should examine the generalizability of these findings to other social science disciplines.

Implications for Publication and Research Design in IR

According to these results, using an MMR approach does not, on average, result in more citations than do other empirical strategies. If anything, the opposite is the case – measured by citations, article-length work using MMR is penalized compared to quantitative approaches. One explanation might be that high-quality MMR is more difficult to execute than high-quality SMR. MMR requires mastery of multiple methods, and weak execution at any point can prove damaging to the overall research project.¹¹⁰

A major implication of this finding concerns the publication type under examination.¹¹¹ The word limits of article-length work may not offer enough space to convincingly apply multiple methods to a research question. Devoting only part of an article to a case study or a regression analysis may leave specialists wanting more, while simultaneously deterring non-specialists. Journal editors and reviewers should therefore be more hesitant to request further methodological components in an article.

Furthermore, from a professional perspective, citation counts are important for hiring decisions and disciplinary prestige. If MMR work is cited less than quantitative work is, scholars might choose to pursue MMR less frequently. This disincentive to conduct MMR might suppress innovative research designs and MMR’s epistemic potential.¹¹²

Another implication relates to MMR results and presentation, and concerns readers and citers. It could be that using multiple methods in pursuit of answering a single research question in effect makes arguments less parsimonious, research strategies more complex, and the subsequent results more cluttered and more difficult to reconcile with one another than most SMR approaches might, at least in article-length work.¹¹³

These findings on the reception of MMR articles in IR and security studies also relate to ongoing debates surrounding knowledge accumulation and the synthesis of results in the social sciences. The analysis suggests that rather than expecting that any individual empirical paper cover multiple methods, a more specialized approach is called for, with dedicated synthesis papers bridging these gaps.¹¹⁴ In examining these issues, this study contributes to broader ongoing debates on knowledge appraisal and research reception in the social sciences,¹¹⁵ as well as the move toward more team science in the social sciences.¹¹⁶

Acknowledgements

I thank Irene Weipert-Fenner, Jens Stappenbeck, Barış Kesgin, Julian Junk, Caroline Fehl, Ben Christian, Felix Bethke, two anonymous reviewers, and the Security Studies editors, Ronald R. Krebs and Ron Hassner, for their very helpful and constructive comments. I am grateful to Irene Entringer at TRIP and Catherine Weaver and Jason Sharman for sharing their data.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

The data and materials that support the findings of this study are available in the Harvard Dataverse at https://doi.org/10.7910/DVN/7OWHLM.

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Notes on contributors

Anton Peez

Anton Peez is a Postdoctoral Researcher at the University of Frankfurt and an Associate Fellow at Peace Research Institute Frankfurt (PRIF).

Notes

1 I use “multi-method research” rather than “mixed-method research.”

2 Jason Seawright, Multi-Method Social Science: Combining Qualitative and Quantitative Tools (Cambridge, UK: Cambridge University Press, 2016); Alejandro Avenburg, John Gerring, and Jason Seawright, “How Do Social Scientists Reach Causal Inferences? A Study of Reception,” Quality & Quantity, 2022, 2, https://doi.org/10.1007/s11135-022-01353-5.

3 On the increased usage of MMR, see Mario Luis Small, “How to Conduct a Mixed Methods Study: Recent Trends in a Rapidly Growing Literature,” Annual Review of Sociology 37, no. 1 (2011): 60, https://doi.org/10.1146/annurev.soc.012809.102657; Seawright, Multi-Method Social Science, 1–4. On MMR in IR, see Gary Goertz, “Multimethod Research,” Security Studies 25, no. 1 (2016): 3–24, https://doi.org/10.1080/09636412.2016.1134016; Jason Seawright, “Better Multimethod Design: The Promise of Integrative Multimethod Research,” Security Studies 25, no. 1 (2016): 42–49, https://doi.org/10.1080/09636412.2016.1134187. On MMR in CP, see Dan Slater and Daniel Ziblatt, “The Enduring Indispensability of the Controlled Comparison,” Comparative Political Studies 46, no. 10 (2013): 1321–22, https://doi.org/10.1177/0010414012472469.

4 On MMR and causal inference, see Evan S. Lieberman, “Nested Analysis as a Mixed-Method Strategy for Comparative Research,” American Political Science Review 99, no. 3 (2005): 435–52, https://doi.org/10.1017/S0003055405051762; Macartan Humphreys and Alan M. Jacobs, “Mixing Methods: A Bayesian Approach,” American Political Science Review 109, no. 4 (2015): 653–73, https://doi.org/10.1017/S0003055415000453; Seawright, Multi-Method Social Science; Seawright, “Better Multimethod Design”; Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 2. On MMR and external and internal validity, see Kathleen M.T. Collins, Validity in Multimethod and Mixed Research, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), https://doi.org/10.1093/oxfordhb/9780199933624.013.17; See also Slater and Ziblatt, “Controlled Comparison.” On MMR and policy recommendations, see Tanisha M. Fazal, “An Occult of Irrelevance? Multimethod Research and Engagement with the Policy World,” Security Studies 25, no. 1 (2016): 34–41, https://doi.org/10.1080/09636412.2016.1134186. On the comparison to SMR, see R. Burke Johnson and Anthony J. Onwuegbuzie, “Mixed Methods Research: A Research Paradigm Whose Time Has Come,” Educational Researcher 33, no. 7 (2004): 18, https://doi.org/10.3102/0013189X033007014.

5 John Gerring, “Comprehensive Appraisal,” in The Production of Knowledge, ed. Colin Elman, John Gerring, and James Mahoney (Cambridge University Press, 2020), 356, https://doi.org/10.1017/9781108762519.014.

6 Amel Ahmed and Rudra Sil, “Is Multi-Method Research Really ‘Better’?,” Qualitative & Multi-Method Research 7, no. 2 (2009): 2–6, https://doi.org/10.5281/ZENODO.938958; Scott Gehlbach, “The Fallacy of Multiple Methods,” Comparative Politics Newsletter, 2015, https://scottgehlbach.net/publications/the-fallacy-of-multiple-methods/; Derek Beach and Jonas Gejl Kaas, “The Great Divides: Incommensurability, the Impossibility of Mixed-Methodology, and What to Do about It,” International Studies Review 22, no. 2 (2020): 214–35, https://doi.org/10.1093/isr/viaa016.

7 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 2.

8 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

9 On social science reception, see Ibid.

10 The full TRIP JAD features 9.118 articles, some of which are individual pieces of correspondence combined under one DOI. The databases are merged by DOI, so items such as this are dropped.

11 2014–2018, the five most recent years.

12 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

13 Cyrus Samii, “Causal Empiricism in Quantitative Research,” The Journal of Politics 78, no. 3 (2016): 950–51, https://doi.org/10.1086/686690; See also John Gerring, Sebastian Karcher, and Brendan Apfeld, “Impact Metrics,” in The Production of Knowledge, ed. Colin Elman, John Gerring, and James Mahoney (Cambridge University Press, 2020), 389, https://doi.org/10.1017/9781108762519.015.

14 Colin Elman, John Gerring, and James Mahoney, eds., The Production of Knowledge: Enhancing Progress in Social Science, Strategies for Social Inquiry (Cambridge, UK: Cambridge University Press, 2020); Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

15 On such best practices, see Ingo Rohlfing and Peter Starke, “Building on Solid Ground: Robust Case Selection in Multi-Method Research,” Swiss Political Science Review 19, no. 4 (2013): 492–512, https://doi.org/10.1111/spsr.12052; Seawright, Multi-Method Social Science. On combining methods, see Julia Brannen, “Mixing Methods: The Entry of Qualitative and Quantitative Approaches into the Research Process,” International Journal of Social Research Methodology 8, no. 3 (2005): 173–84, https://doi.org/10.1080/13645570500154642; Julian Junk and Valentin Rauer, “Combining Methods: Connections and Zooms in Analysing Hybrids,” in Transformations of Security Studies: Dialogues, Diversity and Discipline, ed. Gabi Schlag, Julian Junk, and Christopher Daase (Routledge, 2015), 216–32, https://doi.org/10.4324/9781315707839. For a formal approach to combining methods, see Humphreys and Jacobs, “Mixing Methods.”

16 Slater and Ziblatt, “Controlled Comparison,” 1322.

17 Ibid., 1321.

18 Lieberman, “Nested Analysis.”

19 Sharlene Nagy Hesse-Biber, Deborah Rodriguez, and Nollaig A. Frost, A Qualitatively Driven Approach to Multimethod and Mixed Methods Research, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), https://doi.org/10.1093/oxfordhb/9780199933624.013.3.

20 Lieberman, “Nested Analysis”; Hesse-Biber, Rodriguez, and Frost, A Qualitatively Driven Approach to Multimethod and Mixed Methods Research, 5; Seawright, Multi-Method Social Science.

21 Sharlene Nagy Hesse-Biber and Amy J. Griffin, Feminist Approaches to Multimethod and Mixed Methods Research, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), 87, https://doi.org/10.1093/oxfordhb/9780199933624.013.6.

22 Ahmed and Sil, “Is Multi-Method Research Really ‘Better’?”

23 Beach and Kaas, “The Great Divides.”

24 Gehlbach, “The Fallacy of Multiple Methods,” 12; See also Samii, “Causal Empiricism,” 950–51.

25 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

26 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 5, 11.

27 Peter Marcus Kristensen, “International Relations at the End: A Sociological Autopsy,” International Studies Quarterly, 2018, 248–50, https://doi.org/10.1093/isq/sqy002; Martin Caon, Jamie Trapp, and Clive Baldock, “Topical Debate: Citations Are a Good Way to Determine the Quality of Research,” Physical and Engineering Sciences in Medicine 43, no. 4 (2020): 1146, https://doi.org/10.1007/s13246-020-00941-9.

28 Daniel Maliniak, Ryan Powers, and Barbara F. Walter, “The Gender Citation Gap in International Relations,” International Organization 67, no. 4 (2013): 895–96, https://doi.org/10.1017/S0020818313000209.

29 Daniel Maliniak, Susan Peterson, and Michael J. Tierney, “Journal Article Database Codebook. Version 2.1. Revised: 6/11/2018.,” 2018, https://trip.wm.edu/data/replication-and-other-data/TRIP_Journal%20Article%20Database_Codebook2.1.pdf; Daniel Maliniak et al., “International Relations in the US Academy,” International Studies Quarterly 55, no. 2 (2011): 437–64, https://doi.org/10.1111/j.1468-2478.2011.00653.x; Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations”; Daniel Maliniak et al., “Is International Relations a Global Discipline? Hegemony, Insularity, and Diversity in the Field,” Security Studies 27, no. 3 (2018): 448–84, https://doi.org/10.1080/09636412.2017.1416824; Jack Hoagland et al., “The Blind Men and the Elephant: Comparing the Study of International Security Across Journals,” Security Studies, 2020, 1–41, https://doi.org/10.1080/09636412.2020.1761439.

30 Gerring, Karcher, and Apfeld, “Impact Metrics,” 375–76.

31 It is important to note the changes in the annual composition of this sample, most notably that Security Studies (SS) is added in 1991 and the European Journal of International Relations (EJIR) in 1995 (their first volumes). International Studies Quarterly (ISQ) has grown from about 25 articles annually up until the early 2000s to about 60 in recent years. For a discussion of the TRIP JAD, see also Anton Peez, “Contributions and Blind Spots of Constructivist Norms Research in International Relations, 1980–2018: A Systematic Evidence and Gap Analysis,” International Studies Review 24, no. 1 (2022): viab055, https://doi.org/10.1093/isr/viab055.

32 In general, these substantive traditions include security studies in IS and SS, international organization in International Organization (IO), peace and conflict studies in the Journal of Conflict Resolution (JCR) and the Journal of Peace Research (JPR), IR theory in EJIR, as well as ISQ and World Politics (WP) as more generalist journals. Methodological traditions include qualitative approaches in International Security (IS), SS, and EJIR, quantitative work in ISQ and JPR, and pluralist approaches in IO and WP. See also Table 3.

33 Stanley Hoffmann, “An American Social Science: International Relations,” Daedalus 106, no. 3 (1977): 41–60.

34 E.g., EJIR. Mathis Lohaus and Wiebke Wemheuer-Vogelaar, “Who Publishes Where? Exploring the Geographic Diversity of Global IR Journals,” International Studies Review, 2020, viaa062, https://doi.org/10.1093/isr/viaa062.

35 E.g., Millennium and Review of International Studies, respectively.

36 Hoffmann, “An American Social Science: International Relations”; Maliniak et al., “Is International Relations a Global Discipline?”; Kristensen, “International Relations at the End,” 247.

37 Daniel Maliniak et al., “TRIP 2017 Faculty Survey (International),” 2017, https://trip.wm.edu/data/our-surveys/faculty-survey/TRIP_2017_International_Faculty_Survey__Topline_Results.pdf.

38 Beach and Kaas, “The Great Divides.”

39 The other key practice is publishing in university presses, Jason C. Sharman and Catherine E. Weaver, “Between the Covers: International Relations in Books,” PS: Political Science & Politics 46, no. 1 (2013): 125, https://doi.org/10.1017/S1049096512001291. Felix S. Bethke and Christian Bueger, “Bursts! Theoretical Fashions in the Study of International Organizations. A Bibliometric Analysis.,” 2014, 10–11, www.dropbox.com/s/2w5j69cjjnjp1ya/Bursts_Theoretical_Fashions_in_the_Study.pdf.

40 Kristensen, “International Relations at the End,” 246.

41 See also ibid., 247 on similar limitations in bibliometric work on IR.

42 Beach and Kaas, “The Great Divides,” 2; emphasis in original.

43 Maliniak, Peterson, and Tierney, “TRIP JAD Codebook V2.1.”

44 Ibid., 17–19.

45 The codebook references two works on “small-N-oriented political science research”; Ibid., 18. In other words, this does not cover ‘counterfactual’ in the sense of the potential outcomes framework for causal inference.

46 Although the codebook states that the “analytic/non-formal conceptual” label is not combined with any of the empirical methods labels, it is on several occasions, see Figure 2; Ibid. It is therefore here used for the “simple” operationalization.

47 Although the codebook states that the “descriptive” label is not combined with any of the empirical methods labels, it is on several occasions, see Figure 2; Ibid.

48 Goertz, “Multimethod Research,” 5.

49 David Collier, Henry E Brady, and Jason Seawright, “Sources of Leverage in Causal Inference: Toward an Alternative View of Methodology,” in Rethinking Social Inquiry: Diverse Tools, Shared Standards, ed. Henry E Brady and David Collier (Rowman and Littlefield, 2004), 229–66; Humphreys and Jacobs, “Mixing Methods,” 653; Very similarly, see Goertz, “Multimethod Research,” 6.

50 Gary Goertz and James Mahoney, A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences, 2013, https://doi.org/10.23943/princeton/9780691149707.001.0001.

51 David Kuehn and Ingo Rohlfing, “Are There Really Two Cultures? A Pilot Study on the Application of Qualitative and Quantitative Methods in Political Science,” European Journal of Political Research 55, no. 4 (2016): 885–905, https://doi.org/10.1111/1475-6765.12159; David Kuehn and Ingo Rohlfing, “Does the Application of Qualitative and Quantitative Methods Reflect Two Distinct Cultures? An Empirical Analysis of 180 Articles Suggests ‘No,’” 2020, 24–28, 31, https://osf.io/6uhpd/.

52 Kuehn and Rohlfing, “Are There Really Two Cultures?”

53 Goertz, “Multimethod Research,” 5–6.

54 Rohlfing and Starke, “Building on Solid Ground.”

55 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 3, 6.

56 Lisa D. Pearce, Thinking Outside the Q Boxes, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), 42, https://doi.org/10.1093/oxfordhb/9780199933624.013.4.

57 Seawright illustrates his point with the “suggestive data” of Google Scholar hits relative to OLS, while mentioning other academic trends relating to digitization that might affect the results; 2016, 2–4.

58 I.e., all work that is not any of the other four categories, based on the eight TRIP categories and narrow MMR definition. The high shares of this category for JCR and IS are largely driven by unusually large shares of formal modelling (28.0%) and descriptive work (28.2%), respectively.

59 Articles may be assigned multiple substantive foci. The average number per article is 2.9.

60 This aligns with the expectations in Rohlfing and Starke, “Building on Solid Ground.”

61 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations.”

62 Gerring, Karcher, and Apfeld, “Impact Metrics,” 375–76.

63 On “epistemological validity” see Ahmed and Sil, “Is Multi-Method Research Really ‘Better’?”

64 Gerring, Karcher, and Apfeld, “Impact Metrics,” 380.

65 Kristensen, “International Relations at the End,” 248–51; See also Gerring, Karcher, and Apfeld, “Impact Metrics” for a comprehensive discussion and critique of citation counts as an assessment metric.

66 Kristensen, “International Relations at the End,” 249.

67 Ibid., 248, 256.

68 Ibid., 250.

69 Ibid.

70 D. Lindsey, “Using Citation Counts as a Measure of Quality in Science Measuring What’s Measurable Rather than What’s Valid,” Scientometrics 15, no. 3–4 (1989): 195–96, https://doi.org/10.1007/BF02017198.

71 Examples of these three points, in order, are Benedict Anderson’s conceptualization of nations as “imagined communities”; the way Francis Fukuyama’s argument regarding the “end of history” is often cited; and Samuel Huntington’s “Clash of Civilizations.”

72 Gerring, Karcher, and Apfeld, “Impact Metrics,” 396–97.

73 Caon, Trapp, and Baldock, “Citations Are a Good Way to Determine the Quality of Research,” 1146.

74 I.e., a “‘bounded’ entity—a formalized separate research practice,” Sharlene Nagy Hesse-Biber, Introduction: Navigating a Turbulent Research Landscape, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), xxxviii, https://doi.org/10.1093/oxfordhb/9780199933624.013.1.

75 Small, “How to Conduct a Mixed Methods Study,” 77–79.

76 Ibid., 79.

77 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations”; Iman Tahamtan, Askar Safipour Afshar, and Khadijeh Ahamdzadeh, “Factors Affecting Number of Citations: A Comprehensive Review of the Literature,” Scientometrics 107, no. 3 (2016): 1195–1225, https://doi.org/10.1007/s11192-016-1889-2.

78 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations.”

79 Ibid.

80 This is based on the ‘narrow’ methods definition, see above.

81 Bethke and Bueger, “Bursts!”

82 On self-citations and citation groups, see Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 915–17. On coerced citations, see Caon, Trapp, and Baldock, “Citations Are a Good Way to Determine the Quality of Research,” 1147.

83 Andrew Gelman, Jennifer Hill, and Aki Vehtari, Regression and Other Stories (Cambridge, UK: Cambridge University Press, 2021), 266–68, https://doi.org/10.1017/9781139161879. For IR, see Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 899; in general, see Mike Thelwall and Paul Wilson, “Regression for Citation Data: An Evaluation of Different Methods,” Journal of Informetrics 8, no. 4 (2014): 963–71, https://doi.org/10.1016/j.joi.2014.09.011.

84 George A. Barnett and Edward L. Fink, “Impact of the Internet and Scholar Age Distribution on Academic Citation Age,” Journal of the American Society for Information Science and Technology 59, no. 4 (2008): 530, https://doi.org/10.1002/asi.20706.

85 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 897, 899.

86 Bethke and Bueger, “Bursts!”

87 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 905. This amount to 485+ citations for the full TRIP sample.

88 See also Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 5. This means removing all articles for which the only coded method is “analytic/non-formal conceptual”, i.e., those articles “without reference to significant empirical evidence or a formal model”, as well as those with the ‘issue area’ ‘Methodology’; Maliniak, Peterson, and Tierney, “TRIP JAD Codebook V2.1.”

89 Gerring, Karcher, and Apfeld, “Impact Metrics,” 384.

90 See also Kristensen, “International Relations at the End,” 257.

91 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations.”

92 IO is chosen as the reference category due to its prominence.

93 Note that the effect of an all-female author team is slightly positive, in contrast to Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations.” Their study’s full model examines 2,541 articles from 1980–2006 with citation data as of 2011 or 2012. The present study examines 9,033 articles from 1980–2018 with citation data as of January 2023. This is potentially in line with the original study’s suggestive finding that the gender citation may have been declining over time at the time of publication. Ibid., 912–14.

94 Models 1–4 in the main analysis displayed below are Models 4a–4d in the Online Appendix.

95 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 899.

96 Gelman, Hill, and Vehtari, Regression and Other Stories, 267.

97 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 9–12.

98 See also Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 904 for a similar figure regarding author gender and tenure status.

99 Paradigm set to ‘realist,’ epistemology to ‘positivist,’ issue area to ‘international security.’ All other variables held at means.

100 See also Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 904.

101 This covers articles that are coded as addressing international security with a substantive focus or terrorism and/or intrastate conflict; Hoagland et al., “The Blind Men and the Elephant,” 400.

102 Fazal, “An Occult of Irrelevance?”

103 Thelwall and Wilson, “Regression for Citation Data.”

104 Sharman and Weaver, “Between the Covers.”

105 The monograph dataset does not cover author characteristics, so they are not included in the models. Citation data as of 27 February 2023.

106 Goertz, “Multimethod Research.”

107 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

108 Lohaus and Wemheuer-Vogelaar, “Who Publishes Where?”

109 Margaret E. Roberts, Brandon M. Stewart, and Richard A. Nielsen, “Adjusting for Confounding with Text Matching,” American Journal of Political Science 64, no. 4 (October 2020): 887–903, https://doi.org/10.1111/ajps.12526.

110 This borrows Seawright’s metaphor; Seawright, Multi-Method Social Science.

111 See Patricia Bazeley, Writing Up Multimethod and Mixed Methods Research for Diverse Audiences, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), https://doi.org/10.1093/oxfordhb/9780199933624.013.20.

112 I thank Reviewer 1 for pointing this out.

113 I thank Caroline Fehl for stressing this point.

114 Samii, “Causal Empiricism,” 950–51; See also Gerring, Karcher, and Apfeld, “Impact Metrics,” 389.

115 Elman, Gerring, and Mahoney, The Production of Knowledge; Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

116 Thad Dunning et al., eds., Information, Accountability, and Cumulative Learning: Lessons from Metaketa I (Cambridge University Press, 2019), https://doi.org/10.1017/9781108381390.

Appendix

Table 6. Articles in the sample by journal and year.

Journal	1980	1981	1982	1983	1984	1985	1986	1987	1988	1989	1990	1991	1992	1993	1994	1995	1996	1997	1998	1999	2000	2001
ISQ	20	26	23	30	20	24	22	22	22	24	23	23	20	20	23	24	24	30	35	30	28	26
JPR	26	26	30	31	29	30	30	33	35	34	35	34	35	35	32	34	35	37	30	26	27	30
JCR	29	29	29	25	31	28	32	32	33	31	31	31	30	30	33	30	29	26	25	28	24	23
IO	30	35	29	26	26	27	28	26	25	23	20	20	26	22	21	26	23	23	32	25	28	36
IS	39	39	36	33	28	29	28	37	27	28	29	21	24	24	21	25	21	27	21	25	21	19
WP	23	23	23	20	20	23	17	19	16	19	20	17	18	17	16	16	19	20	18	15	16	18
SS												12	25	22	22	32	28	26	27	18	18	24
EJIR																20	15	15	17	15	17	16
APSR	4	2	4	7	6	5	12	8	4	10	9	7	6	11	7	5	9	13	9	4	10	7
AJPS	4	4	6	6	2	5	3	3	6	4	6	7	4	4	5	9	1	5	6	6	10	3
JOP	1	9	2	1	3	3	2	4	3	2	4	3	4	3	2	4	4	6	6	3	2	5
BJPS	4	3	1	1	2	1	3	5	2	3	3	2	2	1	2	3	3	4	6	3	4	1
Annual sum	180	196	183	180	167	175	177	189	173	178	180	177	194	189	184	228	211	232	232	198	205	208

Table

Journal	2002	2003	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013	2014	2015	2016	2017	2018	Sum	% of Sample
ISQ	24	31	36	29	39	45	37	48	52	54	61	68	71	63	56	68	66	1387	15.4%
JPR	25	25	28	26	27	27	32	27	38	37	37	35	38	43	40	34	35	1248	13.8%
JCR	25	24	27	28	29	26	24	27	26	26	29	29	27	28	28	33	31	1106	12.2%
IO	29	28	25	35	33	27	24	26	25	26	24	29	32	25	25	39	30	1059	11.7%
IS	27	23	21	23	26	25	27	27	24	24	23	22	25	25	24	26	18	1012	11.2%
WP	16	19	20	18	19	17	14	20	19	19	20	20	21	19	20	17	13	724	8.0%
SS	24	20	13	20	21	22	24	24	21	24	28	27	27	27	24	24	23	647	7.2%
EJIR	16	18	18	19	22	20	24	24	31	32	31	41	45	38	39	39	39	611	6.8%
APSR	8	15	12	15	9	11	10	6	11	12	15	13	14	14	3	7	36	370	4.1%
AJPS	10	5	14	15	15	17	14	14	10	20	13	14	12	8		7	30	327	3.6%
JOP	10	7	6	4	6	21	13	9	20	8	16	15	17	16	19	24	33	320	3.5%
BJPS	3	7	2	6	8	13	5	12	8	9	15	12	17	14		11	21	222	2.5%
Annual sum	217	222	222	238	254	271	248	264	285	291	312	325	346	320	278	329	375	9033

Table 7. The ten most-cited MMR papers (narrow definition) in the sample.

	Authors	Title	Journal	Year	Vol.	Iss.	Citations
1	Hellman	Winners Take All: The Politics of Partial eform in Postcommunist Transitions	WP	1998	50	2	732
2	Walter	The Critical Barrier to Civil War Settlement	IO	1997	51	3	545
3	Mansfield; Snyder	Democratization and the Danger of War	IS	1995	20	1	515
4	Pape	Why Economic Sanctions Do Not Work	IS	1997	22	2	492
5	Salehyan; Gleditsch	Refugees and the Spread of Civil War	IO	2006	60	2	449
6	True; Mintrom	Transnational Networks and Policy Diffusion: The Case of Gender Mainstreaming	ISQ	2001	45	1	393
7	Hafner-Burton	Sticks and Stones: Naming and Shaming the Human Rights Enforcement Problem	IO	2008	62	4	388
8	Nielson; Tierney	Delegation to International Organizations: Agency Theory and World Bank Environmental Reform	IO	2003	57	2	372
9	Harff	No Lessons Learned from the Holocaust? Assessing Risks of Genocide and Political Mass (…)	APSR	2003	97	1	369
10	Hafner-Burton	Trading Human Rights: How Preferential Trade Agreements Influence Government Repression	IO	2005	59	3	349

Table 8. Ten randomly selected MMR papers (narrow definition) from the sample.

	Authors	Title	Journal	Year	Vol	Iss	Citations
1	Rasler	A Comparative Analysis of Monadic and Dyadic Perspectives on the 1976 Lebanese Crisis	ISQ	1984	28	4	3
2	Van Der Wusten; Nijman; Thijsse	Security Policies of European Countries Outside the Soviet Sphere	JPR	1985	22	4	1
3	Pugh; Cobble	Non-Nationalist Voting in Bosnian Municipal Elections	JPR	2001	38	1	30
4	Mansfield; Snyder	Democratic Transitions, Institutional Strength, and War	IO	2002	56	2	172
5	Barrington; Herron; Silver	The Motherland Is Calling: Views of Homeland among Russians in the Near Abroad	WP	2003	55	2	37
6	Bussmann; Schneider; Wiesehomeier	Foreign Economic Liberalization and Peace: The Case of Sub-Saharan Africa	EJIR	2005	11	4	28
7	Kanavou	How Peace Agreements Are Derailed: The Evolution of Values in Cyprus, 1959-74	JPR	2006	43	3	4
8	Hertog	Defying the Resource Curse: Explaining Successful State-Owned Enterprises in Rentier States	WP	2010	62	2	73
9	Milewicz; Elsig	The Hidden World of Multilateralism: Treaty Commitments of Newly Democratized (…)	ISQ	2014	58	2	12
10	Hendrix; Young	State Capacity and Terrorism: A Two-Dimensional Approach	SS	2014	23	2	41

Table 9. Full negative binomial models, different samples, international security subsample.

	DV: total citations
	Model 5	Model 6	Model 7	Model 8
	Full	<3 SD cit.	Emp. only	<3 SD cit. + Emp.
Constant	1.290*	1.347*	1.355*	1.373*
	(0.586)	(0.562)	(0.594)	(0.569)
Year since publication	0.462***	0.447***	0.436**	0.426**
	(0.139)	(0.135)	(0.142)	(0.137)
Year since publication^2	−0.009**	−0.009**	−0.009**	−0.009**
	(0.003)	(0.003)	(0.003)	(0.003)
Number of references	0.007***	0.006***	0.006***	0.005***
	(0.001)	(0.001)	(0.001)	(0.001)
Coauthored (Y/N)	0.128*	0.145**	0.105	0.135*
	(0.064)	(0.054)	(0.067)	(0.056)
Author(s): all female (ref: all male)	0.214**	0.246***	0.206**	0.236***
	(0.067)	(0.062)	(0.067)	(0.064)
Author(s): mixed gender	0.162*	0.177*	0.123	0.133+
	(0.081)	(0.076)	(0.081)	(0.076)
Tenured (Y/N)	0.066	0.065	0.065	0.043
	(0.059)	(0.048)	(0.062)	(0.050)
Tenured female (Y/N)	0.010	−0.007	0.061	0.053
	(0.098)	(0.095)	(0.105)	(0.100)
R1 university (US, Y/N)	0.166**	0.148**	0.165**	0.149**
	(0.058)	(0.049)	(0.060)	(0.050)
Journal: WP (ref: IO)	−0.079	0.042	−0.152	−0.032
	(0.131)	(0.123)	(0.132)	(0.123)
Journal: JCR	−0.380***	−0.356***	−0.392***	−0.356***
	(0.110)	(0.083)	(0.113)	(0.084)
Journal: IS	−0.485***	−0.463***	−0.457***	−0.434***
	(0.111)	(0.089)	(0.115)	(0.092)
Journal: EJIR	−0.647***	−0.696***	−0.806***	−0.768***
	(0.165)	(0.143)	(0.158)	(0.147)
Journal: JPR	−0.629***	−0.587***	−0.605***	−0.558***
	(0.103)	(0.087)	(0.107)	(0.089)
Journal: ISQ	−0.721***	−0.654***	−0.744***	−0.647***
	(0.106)	(0.087)	(0.106)	(0.088)
Journal: SS	−1.551***	−1.443***	−1.479***	−1.378***
	(0.112)	(0.097)	(0.115)	(0.100)
Journal: APSR	0.515**	0.366***	0.593**	0.443***
	(0.172)	(0.111)	(0.182)	(0.112)
Journal: AJPS	−0.065	0.031	−0.076	0.030
	(0.138)	(0.122)	(0.139)	(0.121)
Journal: BJPS	−1.225***	−1.148***	−1.196***	−1.111***
	(0.171)	(0.160)	(0.171)	(0.159)
Journal: JOP	−0.779***	−0.707***	−0.772***	−0.689***
	(0.125)	(0.111)	(0.126)	(0.111)
Epistemology: Non-/Post-Pos. (ref: Pos.)	−0.155	−0.156+	−0.218*	−0.165
	(0.095)	(0.090)	(0.110)	(0.105)
Material (Y/N)	−0.184	−0.212+	−0.093	−0.123
	(0.121)	(0.112)	(0.122)	(0.117)
Ideational (Y/N)	0.004	−0.018	−0.008	−0.028
	(0.054)	(0.046)	(0.055)	(0.047)
Paradigm: Constructivist (ref: Realist)	−0.100	−0.060	−0.092	−0.032
	(0.132)	(0.107)	(0.125)	(0.110)
Paradigm: Non-paradigmatic	−0.097	−0.076	−0.076	−0.047
	(0.080)	(0.065)	(0.083)	(0.066)
Paradigm: Liberal	−0.245*	−0.191*	−0.211+	−0.156+
	(0.107)	(0.091)	(0.109)	(0.092)
Paradigm: Marxist	−0.929**	−0.851**	−0.792**	−0.734**
	(0.286)	(0.270)	(0.282)	(0.268)
Paradigm: Atheoretic/non-paradigmatic	−0.216	−0.141	−0.287+	−0.210
	(0.142)	(0.130)	(0.155)	(0.143)
Qual. only (ref: Quant. only)	−0.209**	−0.191**	−0.176**	−0.159*
	(0.066)	(0.062)	(0.068)	(0.064)
MMR	−0.211*	−0.215**	−0.185*	−0.186*
	(0.085)	(0.079)	(0.084)	(0.078)
Conceptual/theoretical only	−0.009	−0.040
	(0.109)	(0.100)
All others	−0.318***	−0.324***	−0.297***	−0.308***
	(0.075)	(0.068)	(0.074)	(0.066)
Observations	3565	3535	3284	3259
AIC	34603.3	33812.8	31816.1	31142.1
R^2	0.011	0.111	0.010	0.115

+p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Note: Year fixed effects not displayed. Robust SEs.

Table 10. Full negative binomial models, different samples, issue area: comparative politics.

	DV: total citations
	Model 9	Model 10	Model 11	Model 12
	Full	<3 SD cit.	Emp. only	<3 SD cit. + Emp.
Constant	1.490+	1.784*	1.681*	1.915*
	(0.831)	(0.800)	(0.827)	(0.798)
Year since publication	0.302+	0.259	0.277+	0.257
	(0.167)	(0.162)	(0.165)	(0.162)
Year since publication^2	−0.007+	−0.006	−0.006+	−0.006
	(0.004)	(0.004)	(0.004)	(0.003)
Number of references	0.005***	0.004***	0.005***	0.004***
	(0.001)	(0.001)	(0.001)	(0.001)
Coauthored (Y/N)	0.193*	0.205*	0.191*	0.200*
	(0.084)	(0.080)	(0.086)	(0.082)
Author(s): all female (ref: all male)	0.135	0.142	0.113	0.117
	(0.097)	(0.095)	(0.099)	(0.096)
Author(s): mixed gender	−0.029	−0.044	−0.040	−0.058
	(0.089)	(0.086)	(0.090)	(0.085)
Tenured (Y/N)	−0.080	−0.087	−0.076	−0.083
	(0.083)	(0.074)	(0.085)	(0.075)
Tenured female (Y/N)	−0.059	−0.054	−0.050	−0.035
	(0.117)	(0.113)	(0.117)	(0.113)
R1 university (US, Y/N)	0.192*	0.167*	0.210**	0.166*
	(0.075)	(0.072)	(0.076)	(0.073)
Journal: WP (ref: IO)	0.253+	0.218	0.161	0.161
	(0.152)	(0.151)	(0.158)	(0.152)
Journal: JCR	0.001	−0.086	−0.051	−0.121
	(0.163)	(0.159)	(0.171)	(0.161)
Journal: IS	0.066	−0.100	−0.077	−0.246
	(0.298)	(0.256)	(0.308)	(0.258)
Journal: EJIR	−0.203	−0.299	−0.177	−0.229
	(0.305)	(0.300)	(0.293)	(0.289)
Journal: JPR	−0.122	−0.144	−0.159	−0.171
	(0.145)	(0.149)	(0.153)	(0.152)
Journal: ISQ	−0.419**	−0.442**	−0.501**	−0.488**
	(0.161)	(0.162)	(0.164)	(0.162)
Journal: SS	−1.198***	−1.220***	−1.284***	−1.262***
	(0.259)	(0.258)	(0.263)	(0.260)
Journal: APSR	0.928***	0.798***	0.794***	0.765***
	(0.200)	(0.191)	(0.199)	(0.196)
Journal: AJPS	0.373+	0.258	0.302	0.221
	(0.191)	(0.177)	(0.195)	(0.178)
Journal: BJPS	−0.012	−0.068	−0.075	−0.088
	(0.205)	(0.206)	(0.209)	(0.208)
Journal: JOP	−0.071	−0.090	−0.131	−0.126
	(0.173)	(0.173)	(0.178)	(0.174)
Epistemology: Non-/Post-Pos. (ref: Pos.)	−0.503**	−0.388*	−0.492*	−0.408*
	(0.188)	(0.180)	(0.199)	(0.190)
Material (Y/N)	0.289	0.319+	0.183	0.219
	(0.183)	(0.165)	(0.187)	(0.160)
Ideational (Y/N)	−0.035	−0.015	−0.037	−0.037
	(0.070)	(0.066)	(0.070)	(0.066)
Paradigm: Constructivist (ref: Realist)	−0.181	−0.210	−0.159	−0.217
	(0.366)	(0.361)	(0.355)	(0.362)
Paradigm: Non-paradigmatic	−0.375	−0.427	−0.338	−0.416
	(0.334)	(0.330)	(0.319)	(0.329)
Paradigm: Liberal	−0.429	−0.469	−0.355	−0.433
	(0.358)	(0.355)	(0.345)	(0.354)
Paradigm: Marxist	−1.264**	−1.350**	−1.180**	−1.274**
	(0.427)	(0.427)	(0.417)	(0.423)
Paradigm: Atheoretic/non-paradigmatic	−0.315	−0.334	−0.273	−0.298
	(0.398)	(0.394)	(0.398)	(0.401)
Qual. only (ref: Quant. only)	−0.276**	−0.268**	−0.278**	−0.251**
	(0.098)	(0.095)	(0.100)	(0.096)
MMR	−0.173	−0.154	−0.170	−0.139
	(0.114)	(0.108)	(0.114)	(0.108)
Conceptual/theoretical only	0.523+	0.281
	(0.281)	(0.271)
All others	−0.198*	−0.255**	−0.206*	−0.253**
	(0.100)	(0.091)	(0.100)	(0.092)
Num.Obs.	1029	1021	1002	996
AIC	9923.0	9717.9	9622.5	9459.6
R^2	0.107	0.199	0.073	0.209

+p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Note: Year fixed effects not displayed. Robust SEs.