A relations-based evaluation framework to support collaborative research and innovation (R&I) networks

References

• Abbott, A., and Q. Schiermeier. 2019. EU Science. The next €100 billion. Nature 569. 22 May 2019. https://www.nature.com/articles/d41586-019-01566-z.
   Google Scholar
• Adelle, C., N. Elema, E. Chakauya, and D. Benson. 2018. “Evaluating Homegrown Research Networks in Africa.” South African Journal of Science 114 (3-4): 1–7. doi:10.17159/sajs.2018/20170070.
   Web of Science ®Google Scholar
• Alkin, M. C., A. T. Vo, and C. A. Christie. 2012. “The Evaluator’s Role in Valuing: Who and with Whom.” In Promoting Valuation in the Public Interest: Informing Policies for Judging Value in Evaluation, edited by G. Julnes, 29–41. Hoboken, NJ: Wiley.
   Google Scholar
• Amoroso, S., A. Coad, and N. Grassano. 2017. “European R&D Networks: A Snapshot from the 7th EU Framework Programme.” Economics of Innovation and New Technology 42 (5): 1170–1183.
   Google Scholar
• Aschhoff, B., and T. Schmidt. 2008. “Empirical Evidence on the Success of R&D Cooperation—Happy Together?” Review of Industrial Organization 33 (1): 41–62. doi:10.1007/s11151-008-9179-7
   Web of Science ®Google Scholar
• Balland, P., R. Boschma, and J. Ravet. 2019. “Network Dynamics in Collaborative Research in the EU, 2003–2017.” European Planning Studies 27 (9): 1811–1818. doi:10.1080/09654313.2019.1641187
   Web of Science ®Google Scholar
• Barajas, A., E. Huergob, and L. Moreno. 2012. “Measuring the Economic Impact of Research Joint Ventures Supported by the EU Framework Programme.” The Journal of Technology Transfer 37 (6): 917–942. doi:10.1007/s10961-011-9222-y
   Web of Science ®Google Scholar
• Bear, J. B., and A. Williams Woolley. 2011. “The Role of Gender in Team Collaboration and Performance.” Interdisciplinary Science Reviews 36 (2): 146–153. doi:10.1179/030801811X13013181961473
   Web of Science ®Google Scholar
• Behn, R. D. 2003. “Why Measure Performance? Different Purposes Require Different Measures” Public Administration Review 63: 586–606. doi:10.1111/1540-6210.00322
   Web of Science ®Google Scholar
• Bonfima, L., A. Segattob, and A. Takahashi. 2018. “The Structural, Relational and Cognitive Dimensions of Social Capital on Innovation and Technology in Interorganizational and Intraorganizational Settings.” International Journal of Innovation 6 (3): 232–255. doi:10.5585/iji.v6i3.306
   Google Scholar
• Bozeman, B., and C. Boardman. 2014. Research Collaboration and Team Science: A State-of-the-art Review and Agenda. Springer.
   Google Scholar
• Breschi, S., and F. Malerba. 2011. “Assessing the Scientific and Technological Output of EU Framework Programmes: Evidence from the FP6 Projects in the ICT Field.” Scientometrics 88 (1): 239–257. doi:10.1007/s11192-011-0378-x
   Web of Science ®Google Scholar
• Broekel, T., and W. Mueller. 2018. “Critical Links in Knowledge Networks – What About Proximities and Gatekeeper Organisations?” Industry and Innovation 25. doi:10.1080/13662716.2017.1343130.
   Web of Science ®Google Scholar
• Brown, Jonathon D. 1986. “Evaluations of Self and Others: Self-Enhancement Biases in Social Judgments.” Social Cognition 4 (4): 353–376. doi:10.1521/soco.1986.4.4.353
   Web of Science ®Google Scholar
• Bruce, A., C. Lyall, J. Tait, and R. Williams. 2004. “Interdisciplinary Integration in Europe: The Case of the Fifth Framework Programme.” Futures 36 (4): 457–470. doi:10.1016/j.futures.2003.10.003
   Web of Science ®Google Scholar
• Burt, R. 2004. “Structural Holes and Good Ideas.” American Journal of Sociology 110 (2): 349–399. doi:10.1086/421787
   Web of Science ®Google Scholar
• Campbell, L. G., S. Mehtani, M. E. Dozier, and J. Rinehart. 2013. “Gender-Heterogeneous Working Groups Produce Higher Quality Science.” PLoS ONE 8 (10): e79147. Doi:10.1371/journal.pone.0079147.
   PubMed Web of Science ®Google Scholar
• Carr, G., D. P. Loucks, and G. Blöschi. 2018. “Gaining Insight Into Interdisciplinary Research and Education Programmes: A Framework for Evaluation.” Research Policy 47 (1): 35–48. doi:10.1016/j.respol.2017.09.010
   Web of Science ®Google Scholar
• Cassi, L., N. Corrocher, F. Malerba, and N. Vonortas. 2008. “Research Networks as Infrastructure for Knowledge Diffusion in European Regions.” Economics of Innovation and New Technology 17 (7-8): 663–676. doi:10.1080/10438590701785603
   Google Scholar
• Chevalier, R. 2009. “The Changing Role of Evaluators and Evaluation.” In Handbook of Improving Performance in the Workplace: Volumes 1-3, 354–374.
   Google Scholar
• Cincera, M., L. Kempen, B. von Pottelsberghe, R. Veugelers, and C. Villegas. 2003. “Productivity Growth, R&D and the Role of International Collaborative Agreements: Some Evidence from Belgium Manufacturing Companies.” Brussels Economic Review 46 (3): 107–140.
   Google Scholar
• Claridge, T. 2018. “Dimensions of Social Capital-structural, cognitive, and relational.” Social Capital Research 1: 1–4.
   Google Scholar
• Collins, H., R. Evans, and M. Gorman. 2007. “Trading Zones and Interactional Expertise.” Studies in History and Philosophy of Science Part A 38 (4): 657–666. doi:10.1016/j.shpsa.2007.09.003
   Web of Science ®Google Scholar
• Cooke, N. J., and M. L. Hilton. 2015. Enhancing the Effectiveness of Team Science. Washington, DC: The National Academies Press.
   Google Scholar
• Costa, A. L., and B. Kallick. 1993. “Through the Lens of a Critical Friend.” Educational Leadership: Journal of the Department of Supervision and Curriculum Development 51 (2): 49–51.
   Web of Science ®Google Scholar
• Després, C., N. Brais, and S. Avellan. 2004. “Collaborative Planning for Retrofitting Suburbs: Transdisciplinarity and Intersubjectivity in Action.” Futures 36: 471–486. doi:10.1016/j.futures.2003.10.004
   Web of Science ®Google Scholar
• Díez-Vial, I., and A. Montoro-Sánchez. 2020. “International Gatekeepers: How to Integrate Domestic Networks and International Relations.” International Business Review 29 (6): 101751. doi:10.1016/j.ibusrev.2020.101751
   Web of Science ®Google Scholar
• Duncker, E. 2001. “Symbolic Communication in Multidisciplinary Cooperations.” Science, Technology, & Human Values 26: 349–338. doi:10.1177/016224390102600305
   Web of Science ®Google Scholar
• European Commission. 2014. Horizon 2020 Work Programme 2014–2015; 10 December 2013. Revised 20 March 2014. European Commission: Brussels, Belgium.
   Google Scholar
• European Commission. 2015. Commitment and Coherence – Ex-Post Evaluation of the 7th EU Framework Programme (2007-2013). EU report.
   Google Scholar
• European Commission. 2017. Key Findings from the Horizon 2020 Interim Evaluation. https://publications.europa.eu/en/publication-detail/-/publication/46cfbdd3-5c78-11e7-954d-01aa75ed71a1/language-en/format-PDF/source-77918402.
   Google Scholar
• European Commission. 2018. “A New Horizon for Europe.” https://ec.europa.eu/info/sites/info/files/research_and_innovation/contact/documents/horizon_europe_impact_assessment_book_web_version.pdf.
   Google Scholar
• European Commission. 2019. Europe in May 2019 Preparing for a More United, Stronger and More Democratic Union in an Increasingly Uncertain World. The European Commission’s contribution to the informal EU27 leaders’ meeting in Sibiu (Romania) on 9 May 2019. https://ec.europa.eu/commission/sites/beta-political/files/comm_sibiu_06-05_en.pdf.
   Google Scholar
• European Commission. 2020. “Report on the Results of the Online Consultation, DG Research and Innovation.” https://ec.europa.eu/info/sites/info/files/research_and_innovation/contact/documents/ec_rtd_hecodesign-implementation_112019.pdf.
   Google Scholar
• European Commission. no date. Responsible Research & Innovation in Horizon 2020. https://ec.europa.eu/programmes/horizon2020/en/h2020-section/responsible-research-innovation#:~:text = Responsible%20Research%20and%20Innovation%20%28RRI%29%20implies%20that%20societal,with%20the%20values%2C%20needs%20and%20expectations%20of%20society.
   Google Scholar
• Feinstein, O. 2006. Evaluation as a Learning Tool. Paper presented at the International Congress on Human Development, Madrid 2006.
   Google Scholar
• Fetterman, D. M. 1994. “Empowerment Evaluation.” Evaluation Practice 15 (1): 1–15. doi:10.1177/109821409401500101
   Google Scholar
• Fitzpatrick, J. L., J. R. Sanders, and B. R. Worthen. 2011. Program Evaluation: Alternative Approaches and Practical Guidelines. New York: Pearson Education.
   Google Scholar
• Galison, P. 2010. Trading with the Enemy 3. Trading Zones and Interactional Expertise: Creating New Kinds of Collaboration, 25.
   Google Scholar
• Georghiou, L., and D. Roessner. 2000. “Evaluating Technology Programs: Tools and Methods.” Research Policy 29: 657–678. doi:10.1016/S0048-7333(99)00094-3
   Web of Science ®Google Scholar
• Head, B. W. 2008. “Assessing Network-Based Collaborations: Effectiveness for Whom?” Public Management Review 10 (6): 733–749. doi:10.1080/14719030802423087.
   Web of Science ®Google Scholar
• Hearn, S., and E. Mendizabal. 2011. “Not Everything That Connects is a Network.” ODI Background paper. https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/6313.pdf.
   Google Scholar
• Hoekman, J., K. Frenken, and F. Van Oort. 2009. “The Geography of Collaborative Knowledge Production in Europe.” The Annals of Regional Science 43 (3): 721–738. doi:10.1007/s00168-008-0252-9
   Web of Science ®Google Scholar
• Hollaender, K., M. C. Loibl, and A. Wilts. 2008. “Management.” In Handbook of Transdiciplinary Research. Proposed by the Swiss Academies of Arts and Sciences, edited by G. Hirsch Hadorn, H. Hoffmann-Riem, S. Biber-Klemm, W. Grossenbacher-Mansuy, D. Joye, C. Pohl, U. Wiesmann, and E. Zemp, 385–398. Heidelberg: Springer.
   Google Scholar
• Karlsson, O. 2001. “Critical Dialogue: Its Value and Meaning.” Evaluation 7 (2): 211–227. doi:10.1177/135638900100700206.
   Google Scholar
• Kenis, P., and K. G. Provan. 2009. “Towards an Exogenous Theory of Public Network Performance.” Public Administration 87 (3): 440–456. doi:10.1111/j.1467-9299.2009.01775.x
   Web of Science ®Google Scholar
• Klein, J. T. 2008. “Evaluation of Interdisciplinary and Transdisciplinary Research: A Literature Review.” American Journal of Preventive Medicine 35 (2): S116–S123. doi:10.1016/j.amepre.2008.05.010
   PubMed Web of Science ®Google Scholar
• König, B., K. Diehl, K. Tscherning, and K. Helming. 2013. “A Framework for Structuring Interdisciplinary Research Management.” Research Policy 42 (1): 261–272. doi:10.1016/j.respol.2012.05.006
   Web of Science ®Google Scholar
• Kuhlmann, S., P. Stegmaier, and K. Konrad. 2019. “The Tentative Governance of Emerging Science and Technology - A Conceptual Introduction.” Research Policy 48 (5): 1091–1097. doi:10.1016/j.respol.2019.01.006
   Web of Science ®Google Scholar
• Kyvik, S., and I. Reymert. 2017. “Research Collaboration in Groups and Networks: Differences Across Academic Fields.” Scientometrics 113 (2): 951–967. doi:10.1007/s11192-017-2497-5.
   PubMed Web of Science ®Google Scholar
• Łącka, I. 2015. “Evaluation of the Effectiveness of Relations in Network Organizations.” In Management of Network Organizations, edited by W. Sroka, and Š Hittmár, 137–152. Cham: Springer.
   Google Scholar
• Laredo, P. 1998. “The Networks Promoted by the Framework Programme and the Questions They Raise About its Formulation and Implementation.” Research Policy 27 (6): 589–598. doi:10.1016/S0048-7333(98)00055-9
   Web of Science ®Google Scholar
• Lewin, K. 1951. Field Theory in Social Science: Selected Theoretical Papers (Edited by Dorwin Cartwright.).
   Google Scholar
• Loughran, J., and N. Brubaker. 2015. “Working with a Critical Friend: A Self-Study of Executive Coaching.” Studying Teacher Education 11 (3): 255–271. Doi: 10.1080/17425964.2015.1078786.
   Web of Science ®Google Scholar
• Luukkonen, T. 1998. “The Difficulties in Assessing the Impact of EU Framework Programs.” Research Policy 27 (6): 599–610. doi:10.1016/S0048-7333(98)00058-4.
   Web of Science ®Google Scholar
• Luukkonen, T. 2000. “Additionality of EU Framework Programmes.” Research Policy 29: 711–724. doi:10.1016/S0048-7333(99)00041-4
   Web of Science ®Google Scholar
• Mazzucato, M. 2018. “Mission-Oriented Research & Innovation in the European Union a Problem-Solving Approach to Fuel Innovation-Led Growth.” EU Report. https://ec.europa.eu/jrc/communities/en/community/european-tto-circle/article/E28098mission-oriented-research-and-innovation-european-union-E28093.
   Google Scholar
• Mazzucato, M. 2019. Governing Missions in the European Union. EU Report. https://ec.europa.eu/info/publications/governing-missions-governing-missions-european-union_de.
   Google Scholar
• McAdam, D., S. Tarrow, and C. Tilly. 2003. “Dynamics of Contention.” Social Movement Studies 2 (1): 99–102. doi:10.1080/14742837.2003.10035370
   Google Scholar
• McNair, L. M., M. Davitt, and G. P. Batten. 2015. “Outside the ‘Comfort Zone’: Impacts of Interdisciplinary Research Collaboration on Research, Pedagogy, and Disciplinary Knowledge Production.” Engineering Studies 7 (1): 47–79. doi:10.1080/19378629.2015.1014817.
   Web of Science ®Google Scholar
• Nahapiet, J., and S. Ghoshal. 1998. “Social Capital, Intellectual Capital, and the Organizational Advantage.” The Academy of Management Review 23 (2): 242–266. doi:10.2307/259373
   Web of Science ®Google Scholar
• Niehoff, J. 2019. “European Partnerships Status of Preparation, DG RTD.” Published online on 15 May 2019. https://www.era-learn.eu/news-events/events/era-learn-thematic-partnership-workshop-towards-horizon-europe/20190515_europeanpartnerships_jn.pdf.
   Google Scholar
• Olejniczak, K., E. Raimondo, and T. Kupiec. 2016. “Evaluation Units as Knowledge Brokers: Testing and Calibrating an Innovative Framework.” Evaluation 22 (2): 168–189. doi:10.1177/1356389016638752
   Web of Science ®Google Scholar
• Ostrom, E. 1999. “Institutional Rational Choice: An Assessment of the Institutional Analysis and Development Framework.” In Theories of the Policy Process, 21–64. Routledge.
   Google Scholar
• Pandza, K., T. A. Wilkins, and E. A. Alfoldi. 2011. “Collaborative Diversity in a Nanotechnology Innovation System: Evidence from the EU Framework Programme.” Technovation 31 (9): 476–489. doi:10.1016/j.technovation.2011.05.003
   Web of Science ®Google Scholar
• Patton, M. Q. 2006. “Evaluation for the Way We Work.” The Nonprofit Quarterly 13 (1): 28–33. http://www.scribd.com/doc/8233067/Michael-Quinn-Patton-Developmental-Evaluation-2006.
   Google Scholar
• Patton, M. Q. 2008. Utilization-focused Evaluation. 4th ed. Thousand Oaks, CA: Sage.
   Google Scholar
• Patton, M. Q. 2015. “State of the Art and Practice of Developmental Evaluation: Answers to Common and Recurring Questions.” In Developmental Evaluation Exemplars Principles in Practice, edited by M. Q. Patton, K. McKegg, and N. Wehipeihana, 1–25. New York: The Guilford Press.
   Google Scholar
• Patton, M. Q. 2016. “Developmental Evaluation.” Seminar held at Kolding, Denmark September 8, 2016 http://danskevalueringsselskab.dk/wp-content/uploads/2018/03/Seminar-1-Developmental.pdf.
   Google Scholar
• Patton, M. Q. 2020a. Blue Marble Evaluation: Premises and Principles. Guilford Publications.
   Google Scholar
• Patton, M. Q. 2020b. “Evaluation Criteria for Evaluating Transformation: Implications for the Coronavirus Pandemic and the Global Climate Emergency.” American Journal of Evaluation, doi:10.1177/1098214020933689.
   Web of Science ®Google Scholar
• Ponds, R., F. G. van Oort, and K. Frenken. 2007. “The Geographical and Institutional Proximity of Research Collaboration.” Papers in Regional Science 86 (3): 423–443. doi:10.1111/j.1435-5957.2007.00126.x
   Web of Science ®Google Scholar
• Provan, K. G., K. R. Isett, and H. B. Milward. 2004. “Cooperation and Compromise: A Network Response to Conflicting Institutional Pressures in Community Mental Health.” Nonprofit and Voluntary Sector Quarterly 33: 489–514. doi:10.1177/0899764004265718
   Web of Science ®Google Scholar
• Provan, K. G., and H. B. Milward. 1999. “Do Networks Really Work?: A Framework for Evaluating Public-Sector Organisational Networks.” Academy of Management Proceedings 1: A1–A6. doi:10.5465/apbpp.1999.27628083.
   Google Scholar
• Provan, K., and J. Sydow. 2008. “Evaluating Inter-Organizational Relationships.” In The Oxford Handbook of Inter-Organizational Relations, edited by S. Cropper, C. Huxham, M. Ebers, and P. Smith Ring. New York, N.Y.: Oxford University Press. doi:10.1093/oxfordhb/9780199282944.003.0026
   Google Scholar
• Puljak, L., and S. G. Vari. 2014. “Significance of Research Networking for Enhancing Collaboration and Research Productivity.” Croatian Medical Journal 55 (3): 181–183. Doi:10.3325/cmj.2014.55.181.
   PubMed Web of Science ®Google Scholar
• Quinn, R E. 1988. Beyond Rational Management. Jossey-Bass, San Francisco, CA.
   Google Scholar
• Romero, D., N. Galeano, and A. Molina. 2009. “Mechanisms for Assessing and Enhancing Organisations’ Readiness for Collaboration in Collaborative Networks.” International Journal of Production Research 47 (17): 4691–4710. doi:10.1080/00207540902847280.
   Web of Science ®Google Scholar
• Schwandt, T. A. 2001. “Understanding Dialogue as Practice.” Evaluation 7 (2): 228–237. doi:10.1177/13563890122209658.
   Google Scholar
• Scriven, M. 1967. “The Methodology of Evaluation.” In Perspectives of Curriculum Evaluation, edited by R. W. Tyler, R. M. Gagne, and M. Scriven, 39–83. Chicago, IL: Rand McNally.
   Google Scholar
• Scriven, M. 1991. Evaluation Thesaurus. 4th ed. Newbury Park, CA: Sage.
   Google Scholar
• Spaapen, J, and L Van Drooge. 2011. “Introducing ‘productive interactions’ in social impact assessment.” Research evaluation 20 (3): 211–218.
   Web of Science ®Google Scholar
• Stevahn, L, J A King, G Ghere, and J Minnema. 2005. “Establishing essential competencies for program evaluators.” American Journal of Evaluation 26 (1): 43–59.
   Web of Science ®Google Scholar
• Stevahn, L, J A King, G Ghere, and J Minnema. 2005. “Evaluator competencies in university-based evaluation training programs.” Canadian Journal of Program Evaluation 20 (2).
   Google Scholar
• Stokols, D., K. L. Hall, B. K. Taylor, and R. P. Moser. 2008. “The Science of Team Science: Overview of the Field and Introduction to the Supplement.” American Journal of Preventive Medicine 35 (2 Suppl): S77–S89. doi:10.1016/j.amepre.2008.05.002.
   PubMed Web of Science ®Google Scholar
• Thomas-Hunt, M. C., and K. W. Phillips. 2004. “When What You Know Is Not Enough: Expertise and Gender Dynamics in Task Groups.” Personality and Social Psychology Bulletin 30 (12): 1585–1598. doi:10.1177/0146167204271186.
   PubMed Web of Science ®Google Scholar
• Tomlinson, P. R. 2012. “Industry Institutions, Social Capital, and Firm Participation in Industrial Development.” Industrial and Corporate Change 21 (1): 1–29. doi:10.1093/icc/dtr028
   Web of Science ®Google Scholar
• Ukrainski, K., H. Kanep, M. Kirs, and E. Karo. 2019. “International R&D Networks of Firms: A Country-Level Analysis of the EU Framework Programmes.” Ordnungspolitische Diskurse, No. 2019-01, OrdnungsPolitisches Portal, s.l.
   Google Scholar
• Volkov, B. B. 2011. “Beyond Being an Evaluator: The Multiplicity of Roles of the Internal Evaluator.” New Directions for Evaluation 132: 25–42. doi:10.1002/ev.394
   Google Scholar
• Walter, A. I., S. Helgenberger, A. Wiek, and R. W. Scholz. 2007. “Measuring Societal Effects of Transdisciplinary Research Projects: Design and Application of an Evaluation Method.” Evaluation and Program Planning 30 (4): 325–338. doi:10.1016/j.evalprogplan.2007.08.002
   PubMed Web of Science ®Google Scholar
• Weiss, C. 1995. “Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families.” In New Approaches to Evaluating Community Initiatives’, edited by J. Connell, A. Kubisch, L. Schorr, and C. Weiss. Washington, DC: Aspen Institute.
   Google Scholar
• Wiltz, L. K. 2005. “I Need a Bigger Suitcase: The Evaluator Role in non-Formal Education.” New Directions for Evaluation 108: 13–28. doi:10.1002/ev.168.
   Google Scholar