Human–Artificial Intelligence Collaboration in Prediction: A Field Experiment in the Retail Industry

References

• Agrawal, A.; Gans, J.S.; and Goldfarb, A. Exploring the impact of artificial Intelligence: Prediction versus judgment. Information Economics and Policy, 47 (2019), 1–6.
   Web of Science ®Google Scholar
• Al-Amoudi, I.; and Latsis, J. Anormative black boxes: Artificial intelligence and health policy, In E. Lazega and I. Al-Amoudi (eds.), Post-Human Institutions and Organisations: Confronting the Matrix. Oxfordshire, UK: Taylor & Francis, 2019, pp. 119–142.
   Google Scholar
• Amar, J.; Rahimi, S.; Surak, Z.; and von Bismarck, N. AI-driven operations forecasting in data-light environments. McKinsey & Company. 2022. https://www.mckinsey.com/capabilities/operations/our-insights/ai-driven-operations-forecasting-in-data-light-environments ( accessed October 2022).
   Google Scholar
• Balasubramanian, N.; Ye, Y.; and Xu, M. Substituting human decision-making with machine learning: Implications for organizational learning. Academy of Management Review, 47, 3 (2022), 448–465.
   Web of Science ®Google Scholar
• Beckman, C. M.; Haunschild, P. R.; and Phillips, D. J. Friends or strangers? Firm-specific uncertainty, market uncertainty, and network partner selection. Organization Science, 15, 3 (2004), 259–275.
   Web of Science ®Google Scholar
• Benbya H.; Davenport T.H., and Pachidi S. Artificial intelligence in organizations: Current state and future opportunities. MIS Quarterly Executive, 19, 4 (2020), 9–21.
   Web of Science ®Google Scholar
• Benbya, H.; Nan, N.; Tanriverdi, H.; and Yoo, Y. Complexity and information systems research in the emerging digital world. Mis Quarterly, 44, 1 (2020), 1–17.
   Google Scholar
• Benbya, H.; Pachidi, S.; and Jarvenpaa, S. Special issue editorial: Artificial intelligence in organizations: implications for information systems research. Journal of the Association for Information Systems, 22, 2 (2021), 10.
   Web of Science ®Google Scholar
• Berente, N.; Gu, B.; Recker, J.; and Santhanam, R. Managing artificial intelligence. MIS Quarterly, 45, 3 (2021), 1433–1450.
   Google Scholar
• Blattberg, R. C.; and Hoch, S. J. Database models and managerial intuition: 50% model + 50% manager. Management Science, 36, 8 (1990), 887–899.
   Web of Science ®Google Scholar
• Brehmer, B. Response consistency in probabilistic inference tasks. Organizational Behavior and Human Performance, 22, 1 (1978), 103–115.
   Google Scholar
• Brown, A.; Chouldechova, A.; Putnam-Hornstein, E.; Tobin, A.; and Vaithianathan, R. Toward algorithmic accountability in public services: A qualitative study of affected community perspectives on algorithmic decision-making in child welfare services. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. (2019), 1–12.
   Google Scholar
• Brown, S. Machine learning, explained. 2021. https://mitsloan.mit.edu/ideas-made-to-matter/machine-learning-explained ( accessed October 2022).
   Google Scholar
• Brynjolfsson, E.; and Mitchell, T. What can machine learning do? Workforce implications. Science, 358, 6370 (2017), 1530–1534.
   PubMed Web of Science ®Google Scholar
• Brynjolfsson, E.; Rock, D.; and Syverson, C. Artificial intelligence and the modern productivity paradox: A clash of expectations and statistics. In The Economics of Artificial Intelligence: An Agenda. Chicago: University of Chicago Press, 2018, pp. 23–57.
   Google Scholar
• Cattaneo, M.D.; Drukker, D.M.; and Holland, A.D. Estimation of multivalued treatment effects under conditional independence. The Stata Journal, 13, 3 (2013), 407–450.
   Web of Science ®Google Scholar
• Chandra, S.; Shirish, A.; and Srivastava, S.C. To be or not to be … Human? Theorizing the role of human-like competencies in conversational artificial intelligence agents. Journal of Management Information Systems, 39, 4 (2022), 969–1005
   Web of Science ®Google Scholar
• Choudhury, P.; Starr, E.; and Agarwal, R. Machine learning and human capital complementarities: Experimental evidence on bias mitigation. Strategic Management Journal, 41, 8 (2020), 1381–1411.
   Web of Science ®Google Scholar
• Cyert, R.M.; and March, J.G. A Behavioral Theory of the Firm. Englewood Cliffs, NJ: Prentice-Hall, 1963, pp. 169–187.
   Google Scholar
• Dastin, J. Amazon scraps secret AI recruiting tool that showed bias against women. 2018. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G ( accessed October 20, 2020).
   Google Scholar
• Davenport, T.H.; and Kirby, J. Beyond automation. Harvard Business Review, 93, 6 (2015), 58–65.
   Web of Science ®Google Scholar
• De-Arteaga, M.; Fogliato, R.; and Chouldechova, A. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. (2020), 1–12.
   Google Scholar
• E2open. 2021 Forecasting and Inventory Benchmark Study. 2021. https://cdn.brandfolder.io/IDKCKNW5/at/68qzkp9n3ttw39ncvxhvcx7n/E2open_2021_Forecasting_and_Inventory_Benchmark_Study_Full_Report.pdf ( accessed October 2022).
   Google Scholar
• Farwell, D.C. Dss mechanisms for judgmental flexibility: An exploratory survey. Journal of Management Information Systems, 1, 2 (1984), 72–79.
   Google Scholar
• Fisher, M.; and Rajaram, K. Accurate retail testing of fashion merchandise: Methodology and application. Marketing Science, 19, 3 (2000), 266–278.
   Web of Science ®Google Scholar
• Füegener, A.; Grahl, J.; Gupta, A.; and Ketter, W. Cognitive challenges in human–artificial intelligence collaboration: investigating the path toward productive delegation. Information Systems Research, 33, 2 (2021), 678–696.
   Web of Science ®Google Scholar
• Gaissmaier, W.; and Schooler, L. The smart potential behind probability matching. Cognition, 109, 3 (2008), 416–422.
   PubMed Web of Science ®Google Scholar
• Galbraith, J. Designing Complex Organizations. Reading, MA: Addison-Wesley, 1973.
   Google Scholar
• Gartner. Current use cases for machine learning in supply chain solutions. 2018. https://www.gartner.com/en/documents/3875876/current-use-cases-for-machine-learning-in-supply-chain-p ( accessed September 7, 2020).
   Google Scholar
• Gaur, V.; Kesavan, S.; Raman, A.; and Fisher, M.L. Estimating demand uncertainty using judgmental forecasts. Manufacturing & Service Operations Management, 9, 4 (2007), 480–491.
   Google Scholar
• Germain, R.; Claycomb, C.; and Dröge, C. Supply chain variability, organizational structure, and performance: The moderating effect of demand unpredictability. Journal of Operations Management, 26, 5 (2008), 557–570.
   Web of Science ®Google Scholar
• Glebbeek, A.C.; and Bax, E.H. Is high employee turnover really harmful? An empirical test using company records. Academy of Management Journal, 47, 2 (2004), 277–286.
   Web of Science ®Google Scholar
• Glynn, A.N.; and Quinn, K.M. An introduction to the augmented inverse propensity weighted estimator. Political Analysis, 18, 1 (2010), 36–56.
   Web of Science ®Google Scholar
• Goodwin, Paul, Brent Moritz, and Enno Siemsen. 2018. “Forecast Decisions.” In The Handbook of Behavioral Operations, edited by Karen Donohue, Elena Katok, and Stephen Leider, 433–458. Hoboken, NJ: John Wiley & Sons, Inc.
   Google Scholar
• Guszcza, J.; Lewis, H.; and Evans-Greenwood, P. Cognitive collaboration: Why humans and computers think better together. Deloitte Review, 20 (2017), 8–29.
   Google Scholar
• Guthrie, P.M. Supply chains need AI, but AI needs humans. 2020. https://www.kinaxis.com/ko/node/2873 ( accessed October 20, 2020).
   Google Scholar
• Harvey, N.; and Reimers, S. Trend damping: Under‐adjustment, experimental artifact, or adaptation to features of the natural environment? Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 2 (2013) 589–607.
   PubMed Web of Science ®Google Scholar
• Harvey, N.; Ewart, T.; and West, R. Effects of data noise on statistical judgement. Thinking & Reasoning, 3, 2 (1997), 111–132.
   Google Scholar
• Hogarth, R. M.; and Makridakis, S. Forecasting and planning: An evaluation. Management science, 27, 2 (1981), 115–138.
   Web of Science ®Google Scholar
• Hong, L.; Lamberson, P. J.; and Page, S. E. Hybrid predictive ensembles: Synergies between human and computational forecasts. Journal of Social Computing, 2, 2 (2021), 89–102.
   PubMedGoogle Scholar
• Huber, J.; and Stuckenschmidt, H. Daily retail demand forecasting using machine learning with emphasis on calendric special days. International Journal of Forecasting, 36, 4 (2020), 1420–1438.
   Web of Science ®Google Scholar
• Johnson, M.; and Vera, A. No AI is an island: the case for teaming intelligence. AI Magazine, 40, 1 (2019), 16–28.
   Web of Science ®Google Scholar
• Kahneman, D.; and Tversky, A. On the psychology of prediction. Psychological review, 80, 4 (1973), 237.
   Web of Science ®Google Scholar
• Kim, C.N.; and McLeod Jr, R. Expert, linear models, and nonlinear models of expert decision making in bankruptcy prediction: A lens model analysis. Journal of Management Information Systems, 16, 1(1999), 189–206.
   Google Scholar
• Kleinberg, J.; Ludwig, J.; Mullainathan, S.; and Obermeyer, Z. Prediction policy problems, American Economic Review, 105, 2 (2015), 491–495.
   PubMedGoogle Scholar
• Kliegr T.; Bahník S.; and Fürnkranz J. A review of possible effects of cognitive biases on interpretation of rule-based machine learning models. Artificial Intelligence, 295 (2021), 103458.
   Web of Science ®Google Scholar
• Kremer, M.; Moritz, B.; and Siemsen, E. Demand forecasting behavior: System neglect and change detection. Management Science, 57, 10 (2011), 1827-1843.
   Google Scholar
• Kumar, V.; Rajan, B.; Venkatesan, R.; and Lecinski, J. Understanding the role of artificial intelligence in personalized engagement marketing. California Management Review, 61, 4 (2019), 135–155.
   Web of Science ®Google Scholar
• Lai, V.; Chen, C.; Liao, Q. V.; Smith-Renner, A., and Tan, C. Towards a science of human-ai decision making: A survey of empirical studies. arXiv preprint arXiv:2112.11471 (2021). https://arxiv.org/pdf/2112.11471.pdf
   Google Scholar
• Launay V. EN. The impact of high frequency trading. 2019. https://www.centralcharts.com/en/gm/1-learn/5-trading/16-automatic/312-the-impact-of-high-frequency-trading ( accessed September 7, 2020).
   Google Scholar
• Lawrence, M.; and O’Connor, M. Exploring judgmental forecasting. International Journal of Forecasting, 8, 1 (1992), 15–26.
   Web of Science ®Google Scholar
• Lawrence, M.; Goodwin, P.; O’Connor, M.; and Önkal, D. Judgmental forecasting: A review of progress over the last 25 years. International Journal of Forecasting, 22, 3 (2006), 493–518.
   Web of Science ®Google Scholar
• Li, W.; and Chai, Y. Assessing and enhancing adversarial robustness of predictive analytics: An empirically tested design framework. Journal of Management Information Systems, 39, 2 (2022), 542–572.
   Web of Science ®Google Scholar
• Lindebaum, D.; Vesa, M.; and Den Hond, F. Insights from “the machine stops” to better understand rational assumptions in algorithmic decision making and its implications for organizations. Academy of Management Review, 45, 1 (2020), 247–263.
   Web of Science ®Google Scholar
• Massey, C.; and Wu, G. Detecting regime shifts: The causes of under‐and overreaction. Management Science, 51, 6 (2005), 932–947.
   Web of Science ®Google Scholar
• McLaren, J. 4 Keys to Better Retail Promotion Forecasting and Replenishment. 2020. https://www.relexsolutions.com/resources/4-keys-to-better-retail-promotion-forecasting-and-replenishment/ ( accessed December 15, 2020).
   Google Scholar
• Metcalfe, J. S.; Perelman, B. S.; Boothe, D. L.; and Mcdowell, K. Systemic oversimplification limits the potential for human-AI partnership. IEEE Access, 9 (2021), 70242–70260.
   Web of Science ®Google Scholar
• OECD Framework for the classification of AI systems. OECD Digital Economy papers. Feb 2022. No. 323
   Google Scholar
• Peeters, M.M.M.; van Diggelen, J.; van den Bosch, K.; Bronkhorst, A.; Neerincx, M.A.; Schraagen, J.M.; and Raaijmakers, S. Hybrid collective intelligence in a human–AI society. AI and Society, 36 (2021), 217–238.
   Web of Science ®Google Scholar
• Rai A.; Constantinides, P.; and Sarker, S. Next-generation digital platforms: Toward human-AI hybrids. Mis Quarterly, 43, 1 (2019), iii–ix.
   Google Scholar
• Raisch, S.; and Krakowski, S. 2021. Artificial intelligence and management: The automation-augmentation paradox. Academy of Management Review, 46, 1 (2021), 192–210.
   Web of Science ®Google Scholar
• Ransbotham, S.; Khodabandeh, S.; Kiron, D.; Candelon, F.; Chu, M., and LaFountain, B. Expanding AI’s impact with organizational learning. MIT Sloan Management Review. Research Report, (2020). https://web-assets.bcg.com/f1/79/cf4f7dce459686cfee20edf3117c/mit-bcg-expanding-ai-impact-withorganizational-learning-oct-2020.pdf
   Google Scholar
• Rastogi, C.; Leqi, L.; Holstein, K., and Heidari, H. A unifying framework for combining complementary strengths of humans and ML toward better predictive decision-making. arXiv preprint arXiv:2204.10806, (2022). https://arxiv.org/pdf/2204.10806.pdf
   Google Scholar
• Revilla, E.; and Rodríguez-Prado, B. Building ambidexterity through creativity mechanisms: Contextual drivers of innovation success. Research Policy, 47, 9 (2018), 1611–1625.
   Web of Science ®Google Scholar
• Saenz, M.J.; Revilla, E., and C. Simón. Designing AI systems with human-machine teams. MIT Sloan Management Review, 61, 3 (2020), 1–5.
   Google Scholar
• Seifert, M.; and Hadida, A.L. On the relative importance of linear model and human judge(s) in combined forecasting. Organizational Behavior and Human Decision Processes, 120, 1 (2013), 24–36.
   Web of Science ®Google Scholar
• Seifert, M.; Siemsen, E.; Hadida, A.L.; and Eisingerich, A.B. Effective judgmental forecasting in the context of fashion products. Journal of Operations Management, 36 (2015), 33–45.
   Web of Science ®Google Scholar
• Sheffi Y. The new (Ab)normal: Reshaping Business and Supply Chain Strategy Beyond COVID-19. MIT CTL Media, 2020. https://books.google.com/books?hl=en&lr=&id=lR8AEAAAQBAJ&oi=fnd&pg=PT13&dq=new+normal+early+childhood+management+education&ots=vAmPvnAcTw&sig=g-7OCoIMrlr1jAPoAYU7Kq2OMaE
   Google Scholar
• Shrestha, Y.R.; Ben-Menahem, S.M.; and Von Krogh, G. Organizational decision-making structures in the age of artificial intelligence. California Management Review, 61, 4 (2019), 66–83.
   Web of Science ®Google Scholar
• Simon, H.A. Models of Man; Social and Rational. New York: Wiley, 1957.
   Google Scholar
• Smith, B.C. The Promise of Artificial Intelligence: Reckoning and Judgment. Cambridge, MA: MIT Press, 2019.
   Google Scholar
• Swink, M.; and Jacobs, B.W. Six sigma adoption: Operating performance impacts and contextual drivers of success. Journal of Operations Management, 30, 6 (2012), 437–453.
   Web of Science ®Google Scholar
• Tan, Z. Bounded, efficient and doubly robust estimation with inverse weighting. Biometrika, 97, 3 (2010), 661–682.
   Web of Science ®Google Scholar
• Teodorescu, M.H.; Morse, L.; Awwad, Y., and Kane, G.C. Failures of fairness in automation require a deeper understanding of human-ML augmentation. MIS Quarterly, 45, 3 (2021), 1483–1500.
   Web of Science ®Google Scholar
• Theocharis, Z.; and Harvey, N. When does more mean worse? Accuracy of judgmental forecasting is nonlinearly related to length of data series. Omega, 87 (2019), 10–19.
   Web of Science ®Google Scholar
• Tofangchi, S.; Hanelt, A.; Marz, D.; and Kolbe, L.M. Handling the efficiency–personalization trade-off in service robotics: A machine-learning approach. Journal of Management Information Systems, 38, 1 (2021), 246–276.
   Web of Science ®Google Scholar
• Van Den Broek, E.; Levina, N.; and Sergeeva, A. In pursuit of data: Negotiating data tensions between data scientists and users of AI tools. Academy of Management Proceedings, 2022, 1 (2022), 16942.
   Google Scholar
• Van Den Broek, E.; Sergeeva, A.; and Huysman, M. When the machine meets the expert: An ethnography of developing AI for hiring. MIS Quarterly, 45, 3 (2021), 1557–1580
   Web of Science ®Google Scholar
• Van Donselaar, K.H.; Gaur, V.; Van Woensel, T.; Broekmeulen, R.A.; and Fransoo, J.C. Ordering behavior in retail stores and implications for automated replenishment. Management Science, 56, 5 (2010), 766–784.
   Web of Science ®Google Scholar
• Wang, W.; and Benbasat, I. Recommendation agents for electronic commerce: Effects of explanation facilities on trusting beliefs. Journal of Management Information Systems, 23, 4 (2007), 217–246.
   Web of Science ®Google Scholar
• Wilson, H.J.; and Daugherty, P.R. Collaborative intelligence: humans and AI are joining forces. Harvard Business Review, 96, 4 (2018), 114–123.
   Web of Science ®Google Scholar
• Winship, C. Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge, UK: Cambridge University Press, 2007.
   Google Scholar
• You, S.; Yang, C.L.; and Li, X. Algorithmic versus human advice: Does presenting prediction performance matter for algorithm appreciation? Journal of Management Information Systems, 39, 2 (2022), 336–365.
   Web of Science ®Google Scholar
• Zanzotto, F.M. Human-in-the-loop artificial intelligence. Journal of Artificial Intelligence Research, 1 64 (2019), 243–252.
   Google Scholar
• Zheng, N.N.; Liu, Z.Y.; Ren, P.J.; Ma, Y.Q.; Chen, S.T.; Yu, S.Y.; Xue, J.R.; Chen, B.D.; and F.Y. Wang. Hybrid-augmented intelligence: collaboration and cognition. Frontiers of Information Technology & Electronic Engineering, 18, 2 (2017), 153–179.
   Web of Science ®Google Scholar
• Ma, Y. Human-Machine Teaming for Intelligent Demand Planning. Thesis Master of Engineering in SCM. DSpace@MIT, Massachusetts Institute of Technology (2020).
   Google Scholar