“It’s Not so Easy to Measure impact”: A Qualitative Analysis of How Universities Measure and Evaluate Their Communication

References

• Adhikari, B., Hlaing, P. H., Robinson, M. R., Ruecker, A., Tan, N. H., Jatupornpimol, N., Chanviriyavuth, R., & Cheah, P. Y. (2019). Evaluation of the Pint of Science festival in Thailand. Public Library of Science ONE, 7(14), e0219983. https://doi.org/10.1371/journal.pone.0219983
   Google Scholar
• AMEC. (2020). Barcelona Principles 3.0. https://amecorg.com/barcelona-principles-3-0-translations/
   Google Scholar
• Andersen, T. O., Dissing, A. S., Varga, T. V., Rod, N. H., & Delcea, C. (2021). The SmartSleep Experiment: Evaluation of changes in night-time smartphone behavior following a mass media citizen science campaign. Public Library of Science ONE, 16(7), e0253783. https://doi.org/10.1371/journal.pone.0253783
   Google Scholar
• Besley, J. C. (2020). Five thoughts about improving science communication as an organizational activity. Journal of Communication Management, 24(3), 155–161. https://doi.org/10.1108/JCOM-03-2020-0022
   Web of Science ®Google Scholar
• Brønn, P. (2014). How others see us: Leaders’ perceptions of communication and communication managers. Journal of Communication Management, 18(1), 58–79. https://doi.org/10.1108/JCOM-03-2013-0028
   Google Scholar
• Bühler, H., Naderer, G., Koch, R., & Schuster, C. (2007). Hochschul-PR in Deutschland: Ziele, Strategien und perspektiven [Public relations of German higher education institutions: Goals, strategies, and perspectives]. Deutscher Universitäts-Verlag.
   Google Scholar
• Buhmann, A., & Brønn, P. S. (2018). Applying Ajzen’s theory of planned behavior to predict practitioners’ intentions to measure and evaluate communication outcomes. Corporate Communications, 23(3), 377–391. https://doi.org/10.1108/CCIJ-11-2017-0107
   Google Scholar
• Buhmann, A., & Likely, F. (2018). Evaluation and measurement. In R. L. Heath & W. Johansen (Eds.), The international encyclopedia of strategic communication (Vol. 1, pp. 625–640). Wiley Blackwell.
   Google Scholar
• Buhmann, A., & Volk, S. C. (2022). Measurement and evaluation: Framework, methods, and critique. In J. Falkheimer & M. Heide (Eds.), Research handbook on strategic communication (pp. 475–489). Edward Elgar Publishing. https://doi.org/10.4337/9781800379893.00039
   Google Scholar
• Busto Salinas, L. (2013). University communication departments. Analysis of the situation in Spain. Estudios sobre el mensaje periodístico, 19, 641–649. https://doi.org/10.5209/rev_ESMP.2013.v19.42147
   Google Scholar
• Chen, G., Xin, Y., & Chen, N.-S. (2017). Informal learning in science museum: Development and evaluation of a mobile exhibit label system with iBeacon technology. Educational Technology Research & Development, 65(3), 719–741. https://doi.org/10.1007/s11423-016-9506-x
   Web of Science ®Google Scholar
• Economou, E., Luck, E., & Bartlett, J. (2023). Between rules, norms and shared understandings: How institutional pressures shape the implementation of data-driven communications. Journal of Communication Management, 27(1), 103–119. https://doi.org/10.1108/JCOM-01-2022-0009
   Web of Science ®Google Scholar
• Engwall, L. (2008). Minerva and the media. Universities protecting and promoting themselves. In C. Mazza, P. Quattrone, & A. Riccaboni (Eds.), European universities in transition: Issues, models and cases (pp. 31–48). Edward Elgar.
   Google Scholar
• Entradas, M., & Bauer, M. W. (Eds.). (2022). Public communication of research universities. Routledge.
   Google Scholar
• Entradas, M., Marcinkowski, F., Bauer, M. W., Pellegrini, G., & Wolniak, R. (2023). University central offices are moving away from doing towards facilitating science communication: A European cross-comparison. Public Library of Science ONE, 18(10), e0290504. https://doi.org/10.1371/journal.pone.0290504
   Google Scholar
• Fitzpatrick, K. R., & Weissman, P. L. (2021). Public relations in the age of data: Corporate perspectives on social media analytics (SMA). Journal of Communication Management, 25(4), 401–416. https://doi.org/10.1108/JCOM-09-2020-0092
   Web of Science ®Google Scholar
• Fredriksson, M., & Pallas, J. (2018). New public management. In R. L. Heath & W. Johansen (Eds.), The international encyclopedia of strategic communication (pp. 1–6). John Wiley & Sons. https://doi.org/10.1002/9781119010722.iesc0119
   Google Scholar
• Fürst, S., Vogler, D., Schäfer, M. S., & Sörensen, I. (2024). From “minimalists” to “professional all-rounders”: Typologizing Swiss universities’ communication practices and structures. The European Journal of Communication Research. Manuscript In Press. Preprint available at https://osf.io/vrk8g
   Google Scholar
• Fürst, S., Volk, S. C., Schäfer, M. S., Vogler, D., & Sörensen, I. (2022). Assessing changes in the public communication of higher education institutions: A survey of leaders of Swiss universities and colleges. Studies in Communication Sciences, 22(3), 515–534. https://doi.org/10.24434/j.scoms.2022.03.3489
   Google Scholar
• Gilkerson, N. D., Swenson, R., & Likely, F. (2019). Maturity as a way forward for improving organizations’ communication evaluation and measurement practices. Journal of Communication Management, 23(3), 246–264. https://doi.org/10.1108/JCOM-12-2018-0130
   Web of Science ®Google Scholar
• Grand, A., & Sardo, A. M. (2017). What works in the field? Evaluating informal science events. Frontiers in Communication, 2, 22. https://doi.org/10.3389/fcomm.2017.00022
   Google Scholar
• Höhn, T. D. (2011). Wissenschafts-PR. Eine Studie zur Öffentlichkeitsarbeit von Hochschulen und außeruniversitären Forschungseinrichtungen [Science PR: A study on public relations at higher education institutions and non-university research organizations]. UVK.
   Google Scholar
• Holtzhausen, D., & Zerfass, A. (2013). Strategic communication – Pillars and perspectives of an alternative paradigm. In A. Zerfaß, L. Rademacher, & S. Wehmeier (Eds.), Organisationskommunikation und Public Relations (pp. 73–94). Springer VS. https://doi.org/10.1007/978-3-531-18961-1_4
   Google Scholar
• Jensen, E. (2014). The problems with science communication evaluation. Journal of Science Communication, 13(1), C04. https://doi.org/10.22323/2.13010304
   Google Scholar
• Jensen, E., & Gerber, A. (2020). Evidence-based science communication. Frontiers in Communication, 4, 78. https://doi.org/10.3389/fcomm.2019.00078
   Google Scholar
• King, H., Steiner, K., Hobson, M., Robinson, A., & Clipson, H. (2015). Highlighting the value of evidence-based evaluation: Pushing back on demands for ‘impact’. Journal of Science Communication, 14(2), A02. https://doi.org/10.22323/2.14020202
   Google Scholar
• Krücken, G. (2021). Multiple competitions in higher education: a conceptual approach. The Innovation, 23(2), 163–181. https://doi.org/10.1080/14479338.2019.1684652
   Google Scholar
• Laredo, P. (2007). Revisiting the third mission of universities: Toward a renewed categorization of university activities? Higher Education Policy, 20(4), 441–456. https://doi.org/10.1057/palgrave.hep.8300169
   Google Scholar
• Lemon, L. L., & VanDyke, M. S. (2023). Addressing grand challenges: Perceptions of interdisciplinary research and how communication structures facilitate interdisciplinary research at US research-intensive universities. Journal of Communication Management, 27(4), 522–538. https://doi.org/10.1108/JCOM-04-2022-0035
   Web of Science ®Google Scholar
• Lemon, L. L., & VanDyke, M. S. (2024). Pandemic problems in the ivory tower: Exploring employee engagement during the COVID-19 crisis. International Journal of Strategic Communication, 18(1), 38–55. https://doi.org/10.1080/1553118X.2023.2235333
   Google Scholar
• Lepori, B., Huisman, J., & Seeber, M. (2014). Convergence and differentiation processes in Swiss higher education: An empirical analysis. Studies in Higher Education, 39(2), 197–218. https://doi.org/10.1080/03075079.2011.647765
   Web of Science ®Google Scholar
• Luoma–aho, V., & Canel, M.-J. (2020). Introduction to public sector communication. In V. Luoma–aho & M.-J. Canel (Eds.), The handbook of public sector communication (pp. 1–25). Wiley Blackwell. https://doi.org/10.1002/9781119263203.ch0
   Google Scholar
• Macnamara, J. (2015). Breaking the measurement and evaluation deadlock: A new approach and model. Journal of Communication Management, 19(4), 371–387. https://doi.org/10.1108/JCOM-04-2014-0020
   Web of Science ®Google Scholar
• Macnamara, J., & Gregory, A. (2018). Expanding evaluation to progress strategic communication: Beyond message tracking to open listening. International Journal of Strategic Communication, 12(4), 469–486. https://doi.org/10.1080/1553118X.2018.1450255
   Google Scholar
• Marcinkowski, F., Kohring, M., Friedrichsmeier, A., & Fürst, S. (2013). Neue Governance und die Öffentlichkeit der Hochschulen [New governance and the publics of higher education institutions. In E. Grande, D. Jansen, O. Jarren, A. Rip, U. Schimank, & P. Weingart (Eds.), Neue Governance der Wissenschaft: Reorganisation – externe Anforderungen – Medialisierung (pp. 257–288). Transcript. https://doi.org/10.1515/transcript.9783839422724.257
   Google Scholar
• Marcinkowski, F., Kohring, M., Fürst, S., & Friedrichsmeier, A. (2014). Organizational influence on scientists’ efforts to go public: An empirical investigation. Science Communication, 36(1), 56–80. https://doi.org/10.1177/1075547013494022
   Web of Science ®Google Scholar
• Mayring, P. (2014). Qualitative Content Analysis. Theoretical Foundation, Basic Procedures and Software Solution. https://nbn-resolving.org/urn:nbn:de:0168-ssoar-395173
   Google Scholar
• Musselin, C. (2007). Are universities specific organisations? In G. Krücken, A. Kosmützky, & M. Torka (Eds.), Towards a multiversity? Universities between global trends and national traditions (pp. 63–84). Transcript.
   Google Scholar
• Niemann, P., van den Bogaert, V., & Ziegler, R. (Eds.). (2023). Evaluationsmethoden der Wissenschaftskommunikation [Evaluation methods in science communication]. Springer VS. https://doi.org/10.1007/978-3-658-39582-7
   Google Scholar
• Nothhaft, H., & Stensson, H. (2019). Explaining the measurement and evaluation stasis: A thought experiment and a note on functional stupidity. Journal of Communication Management, 23(3), 213–227. https://doi.org/10.1108/JCOM-12-2018-0135
   Web of Science ®Google Scholar
• Pellegrini, G. (2021). Evaluating science communication: Concepts and tools for realistic assessment. In M. Bucchi & B. Trench (Eds.), Routledge handbook of public communication of science and technology (pp. 305–322). Routledge. https://doi.org/10.4324/9781003039242
   Google Scholar
• Pennisi, L., & Lackey, N. Q. (2018). A multiyear evaluation of the NaturePalooza science festival. The Journal of Extension, 56(7). Article 8. https://doi.org/10.34068/joe.56.07.08
   Google Scholar
• Pinheiro, R., Langa, P. V., & Pausits, A. (2015). One and two equals three? The third mission of higher education institutions. European Journal of Higher Education, 3(5), 233–249. https://doi.org/10.1080/21568235.2015.1044552
   Google Scholar
• Rädiker, S., & Kuckartz, U. (2019). Analyse qualitativer Daten mit MAXQDA. Text, Audio und Video [analyzing qualitative data with MAXQDA: Text, audio, and video]. Springer VS. https://doi.org/10.1007/978-3-658-22095-2
   Google Scholar
• Raupp, J., & Osterheider, A. (2019). Evaluation von hochschulkommunikation [evaluation of university communication. In B. Fähnrich, J. Metag, S. Post, & M. S. Schäfer (Eds.), Forschungsfeld Hochschulkommunikation (pp. 181–205). Springer VS.
   Google Scholar
• Rodgers, S., Wang, Z., Maras, M. A., Burgoyne, S., Balakrishnan, B., Stemmle, J., & Schultz, J. C. (2018). Decoding science: Development and evaluation of a science communication training program using a triangulated framework. Science Communication, 40(1), 3–32. https://doi.org/10.1177/1075547017747285
   Web of Science ®Google Scholar
• Romenti, S., Murtarelli, G., Miglietta, A., & Gregory, A. (2019). Investigating the role of contextual factors in effectively executing communication evaluation and measurement. Journal of Communication Management, 23(3), 228–245. https://doi.org/10.1108/JCOM-12-2018-0131
   Web of Science ®Google Scholar
• Sauter-Sachs, S. (1992). Public Relations der Universität am Beispiel der Universität Zürich [Public relations of universities: The case of the University of Zurich]. Haupt.
   Google Scholar
• Schwetje, T., Hauser, C., Böschen, S., & Leßmöllmann, A. (2020). Communicating science in higher education and research institutions. Journal of Communication Management, 24(3), 189–205. https://doi.org/10.1108/JCOM-06-2019-0094
   Web of Science ®Google Scholar
• Swenson, R., Gilkerson, N., Likely, F., Anderson, F. W., & Ziviani, M. (2019). Insights from industry leaders: A maturity model for strengthening communication measurement and evaluation. International Journal of Strategic Communication, 13(1), 1–21. https://doi.org/10.1080/1553118X.2018.1533555
   Google Scholar
• Volk, S. C. (2023). Evaluation der Wissenschaftskommunikation: Modelle, Stufen, Methoden [Evaluation of science communication: Models, phases, methods]. In P. Niemann, V. van den Bogaert, & R. Ziegler (Eds.), Evaluationsmethoden der Wissenschaftskommunikation (pp. 33–49). Springer VS. https://doi.org/10.1007/978-3-658-39582-7_3
   Google Scholar
• Volk, S. C., & Buhmann, A. (2023). Digital corporate communication and measurement and evaluation. In V. Luoma-Aho & M. Badham (Eds.), Handbook on Digital Corporate Communication (pp. 118–133). Edward Elgar Publishing. https://doi.org/10.4337/9781802201963.00018
   Google Scholar
• Watson, T., & Zerfass, A. (2011). Return on investment in public relations. A critique of concepts used by practitioners from communication and management sciences perspectives. PRism, 8(1), 1–14.
   Google Scholar
• Weingart, P., & Joubert, M. (2019). The conflation of motives of science communication — causes, consequences, remedies. Journal of Science Communication, 18(3), 1–13. https://doi.org/10.22323/2.18030401
   Web of Science ®Google Scholar
• Zerfass, A., Verčič, D., & Volk, S. C. (2017). Communication evaluation and measurement: Skills, practices and utilization in European organizations. Corporate Communications: An International Journal, 22(1), 2–18. https://doi.org/10.1108/CCIJ-08-2016-0056
   Google Scholar
• Zerfass, A., & Volk, S. C. (2020). Aligning and linking communication with organizational goals. In V. Luoma–aho & M. Canel (Eds.), The handbook of public sector communication (pp. 417–434). Wiley. https://doi.org/10.1002/9781119263203.ch27
   Google Scholar
• Ziegler, R., Hedder, I. R., & Fischer, L. (2021). Evaluation of science communication: Current practices, challenges, and future implications. Frontiers in Communication, 6, 669744. https://doi.org/10.3389/fcomm.2021.669744
   Google Scholar