Automated Visual Analysis for the Study of Social Media Effects: Opportunities, Approaches, and Challenges

References

• Agüera y Arcas, A., Mitchell, M., & Todorov, A. (2017). Physiognomy’s new clothes. https://medium.com/@blaisea/physiognomys-new-clothes-f2d4b59fdd6a
   Google Scholar
• Allen, J., Howland, B., Mobius, M., Rothschild, D., & Watts, D. J. (2020). Evaluating the fake news problem at the scale of the information ecosystem. Science Advances, 6(14), eaay3539. 10.1126/sciadv.aay3539
   PubMed Web of Science ®Google Scholar
• Alpaydin, E. (2020). Introduction to machine learning. MIT Press.
   Google Scholar
• Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015). VQA: Visual question answering. Proceedings of the IEEE international conference on computer vision. 2425–2433.
   Google Scholar
• Araujo, T., Lock, I., & van de Velde, B. (2020). Automated visual content analysis (AVCA) in communication research: A protocol for large scale image classification with pre-trained computer vision models. Communication Methods and Measures, 14(4), 239–265. 10.1080/19312458.2020.1810648
   Web of Science ®Google Scholar
• Aubrey, J. S., & Frisby, C. M. (2011). Sexual objectification in music videos: A content analysis comparing gender and genre. Mass Communication and Society, 14(4), 475–501. 10.1080/15205436.2010.513468
   Web of Science ®Google Scholar
• Auxier, B., & Anderson, M. (2021). Social media use in 2021. Pew Research Center. https://www.pewresearch.org/internet/2021/04/07/social-media-use-in-2021/
   Google Scholar
• Baltrušaitis, T., Ahuja, C., & Morency, L. P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443. 10.1109/TPAMI.2018.2798607
   PubMed Web of Science ®Google Scholar
• Bastos, M., Mercea, D., & Goveia, F. (2021). Guy next door and implausibly attractive young women: The visual frames of social media propaganda. New Media & Society, Advance online publication. 10.1177/14614448211026580
   Web of Science ®Google Scholar
• Bastos, M. Mercea, D. & Goveia, F.(2023). Guy next door and implausibly attractive young women: The visual frames of social media propaganda. New Media & Society, 25(8), 2014–2033.
   Web of Science ®Google Scholar
• Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big?. Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623.
   Google Scholar
• Beskow, D. M., Kumar, S., & Carley, K. M. (2020). The evolution of political memes: Detecting and characterizing internet memes with multi-modal deep learning. Information Processing & Management, 57(2), 102170. 10.1016/j.ipm.2019.102170
   Web of Science ®Google Scholar
• Bossetta, M., & Schmøkel, R. (2023). Cross-platform emotions and audience engagement in social media political campaigning: Comparing candidates’ Facebook and Instagram images in the 2020 US election. Political Communication, 40(1), 48–68. 10.1080/10584609.2022.2128949
   Web of Science ®Google Scholar
• Boussalis, C., & Coan, T. G. (2021). Facing the electorate: Computational approaches to the study of nonverbal communication and voter impression formation. Political Communication, 38(1–2), 75–97. 10.1080/10584609.2020.1784327
   Web of Science ®Google Scholar
• Brands, C., Kruikemeier, S., & Trilling, D. (2021). Insta (nt) famous? Visual self-presentation and the use of masculine and feminine issues by female politicians on instagram. Information, Communication & Society, 24(14), 2016–2036. 10.1080/1369118X.2021.1962942
   Web of Science ®Google Scholar
• Brown, Z., & Tiggemann, M. (2016). Attractive celebrity and peer images on instagram: Effect on women’s mood and body image. Body Image, 19, 37–43. 10.1016/j.bodyim.2016.08.007
   PubMed Web of Science ®Google Scholar
• Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Conference on fairness, accountability and transparency, New York, NY, USA. 77–91. PMLR.
   Google Scholar
• Carrotte, E. R., Prichard, I., & Lim, M. S. C. (2017). “Fitspiration” on social media: A content analysis of gendered images. Journal of Medical Internet Research, 19(3), e6368. 10.2196/jmir.6368
   Web of Science ®Google Scholar
• Casas, A., & Webb Williams, N. (2019). Images that matter: Online protests and the mobilizing role of pictures. Political Research Quarterly, 72(2), 360–375. 10.1177/1065912918786805
   Web of Science ®Google Scholar
• Casas, A. & Williams, N. W.(2019). Images that matter: Online protests and the mobilizing role of pictures. Political Research Quarterly, 72(2), 360–375.
   Web of Science ®Google Scholar
• Cataldo, I., De Luca, I., Giorgetti, V., Cicconcelli, D., Bersani, F. S., Imperatori, C., Abdi, S., Negri, A., Esposito, G., & Corazza, O. (2021). Fitspiration on social media: Body-image and other psychopathological risks among young adults. A narrative review. Emerging Trends in Drugs, Addictions, and Health, 1, 100010. 10.1016/j.etdah.2021.100010
   Google Scholar
• Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014). Return of the devil in the details: Delving deep into convolutional nets. https://arxiv.org/abs/1405.3531
   Google Scholar
• Chen, K., Kim, S. J., Gao, Q., & Raschka, S. (2022). Visual framing of science conspiracy videos: Integrating machine learning with communication theories to study the use of color and brightness. Computational Communication Research, 4(1). 10.5117/CCR2022.1.003.CHEN
   Google Scholar
• Chen, Y., Sherren, K., Smit, M., & Lee, K. Y. (2021). Using social media images as data in social science research. New Media & Society, 25(4), 849–871. 10.1177/14614448211038761
   Web of Science ®Google Scholar
• Crawford, K. (2021). The atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press.
   Google Scholar
• Dan, V., & Arendt, F. (2021). Visual cues to the hidden agenda: Investigating the effects of ideology-related visual subtle backdrop cues in political communication. The International Journal of Press/politics, 26(1), 22–45. 10.1177/1940161220936593
   Web of Science ®Google Scholar
• D’Errico, F. & Poggi, I.(2019, January). Tracking a leader’s humility and its emotions from body, face and voice. In Web Intelligence (Vol. 17, No. (1), pp. 63–74). IOS Press.
   Google Scholar
• de Vreese, C. H. D., & Semetko, H. A. (2004). News matters: Influences on the vote in the Danish 2000 euro referendum campaign. European Journal of Political Research, 43(5), 699–722. 10.1111/j.0304-4130.2004.00171.x
   Web of Science ®Google Scholar
• De Vries, T., Misra, I., Wang, C., & Van der Maaten, L. (2019). Does object recognition work for everyone? Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, Long Beach, California, USA. 52–59.
   Google Scholar
• Dietrich, B. J. (2021). Using motion detection to measure social polarization in the US house of representatives. Political Analysis, 29(2), 250–259. 10.1017/pan.2020.25
   Web of Science ®Google Scholar
• Easton, S., Morton, K., Tappy, Z., Francis, D., & Dennison, L. (2018). Young people’s experiences of viewing the fitspiration social media trend: Qualitative study. Journal of Medical Internet Research, 20(6), e9156. 10.2196/jmir.9156
   Web of Science ®Google Scholar
• Farkas, X., & Bene, M. (2021). Images, politicians, and social media: Patterns and effects of politicians’ image-based political communication strategies on social media. The International Journal of Press/politics, 26(1), 119–142. 10.1177/1940161220959553
   Web of Science ®Google Scholar
• Fredrickson, B. L., & Roberts, T.-A. (1997). Objectification theory: Toward understanding women’s lived experiences and mental health risks. Psychology of Women Quarterly, 21(2), 173–206. 10.1111/j.1471-6402.1997.tb00108.x
   Web of Science ®Google Scholar
• Gil de Zúñiga, H., Molyneux, L., & Zheng, P. (2014). Social media, political expression, and political participation: Panel analysis of lagged and concurrent relationships. Journal of Communication, 64(4), 612–634. 10.1111/jcom.12103
   Web of Science ®Google Scholar
• Grabe, M. E., & Bucy, E. P. (2009). Image bite politics: News and the visual framing of elections. Oxford University Press.
   Google Scholar
• Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B., & Lazer, D. (2019). Fake news on Twitter during the 2016 US presidential election. Science, 363(6425), 374–378. 10.1126/science.aau2706
   PubMed Web of Science ®Google Scholar
• Haim, M., & Jungblut, M. (2021). Politicians’ self-depiction and their news portrayal: Evidence from 28 countries using visual computational analysis. Political Communication, 38(1–2), 55–74. 10.1080/10584609.2020.1753869
   Web of Science ®Google Scholar
• Hiaeshutter-Rice, D., Neuner, F. G., & Soroka, S. (2021). Cued by culture: Political imagery and partisan evaluations. Political Behavior, Advance online publication. 10.1007/s11109-021-09726-6
   Web of Science ®Google Scholar
• Hornik, R., Binns, S., Emery, S., Epstein, V. M., Jeong, M., Kim, K., Kim, Y., Kranzler, E. C., Jesch, E., Lee, S. J., Levin, A. V., Liu, J., O’Donnell, M. B., Siegel, L., Tran, H., Williams, S., Yang, Q., & Gibson, L. A. (2022). The effects of tobacco coverage in the public communication environment on young people’s decisions to smoke combustible cigarettes. Journal of Communication, 72(2), 187–213. 10.1093/joc/jqab052
   PubMed Web of Science ®Google Scholar
• Houts, P. S., Doak, C. C., Doak, L. G., & Loscalzo, M. J. (2006). The role of pictures in improving health communication: A review of research on attention, comprehension, recall, and adherence. Patient Education and Counseling, 61(2), 173–190. 10.1016/j.pec.2005.05.004
   PubMed Web of Science ®Google Scholar
• Joo, J., Bucy, E. P., & Seidel, C. (2019). Automated coding of televised leader displays: Detecting nonverbal political behavior with computer vision and deep learning. International Journal of Communication, 13. https://ijoc.org/index.php/ijoc/article/view/10725
   Web of Science ®Google Scholar
• Joo, J., Li, W., Steen, F. F., & Zhu, S. C. (2014). Visual persuasion: Inferring communicative intents of images. Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, Ohio, USA. 216–223.
   Google Scholar
• Joo, J., & Steinert-Threlkeld, Z. C. (2022). Image as data: Automated content analysis for visual presentations of political actors and events. Computational Communication Research, 4(1). 10.5117/CCR2022.1.001.JOO
   Google Scholar
• Jungblut, M., & Haim, M. (2021). Visual gender stereotyping in campaign communication: Evidence on female and male candidate imagery in 28 countries. Communication Research, Advance online publication. 10.1177/00936502211023333
   Web of Science ®Google Scholar
• Keyes, O. (2018). The misgendering machines: Trans/HCI implications of automatic gender recognition. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 1–22. 10.1145/3274357
   Google Scholar
• Kim, Y. & Kim, J. H.(2018). Using computer vision techniques on Instagram to link users’ personalities and genders to the features of their photos: An exploratory study. Information Processing & Management, 54(6), 1101–1114.
   Web of Science ®Google Scholar
• Kleemans, M., Daalmans, S., Carbaat, I., & Anschütz, D. (2018). Picture perfect: The direct effect of manipulated instagram photos on body image in adolescent girls. Media Psychology, 21(1), 93–110. 10.1080/15213269.2016.1257392
   Web of Science ®Google Scholar
• Kovacheva, A. Wiener, H. J. Kareklas, I. & Muehling, D.(2022). Online engagement with memes and comments about climate change. Sustainability, 14(14), 8900.
   Web of Science ®Google Scholar
• Krcmar, M., Ewoldsen, D. R., & Koerner, A. (2016). Communication science theory and research: An advanced introduction. Routledge.
   Google Scholar
• Kress, G., & Van Leeuwen, T. (1998). Front pages: (the critical) analysis of newspaper layout. In A. Bell & P. Garrett (Eds.), Approaches to media discourse (pp. 186–219). Blackwell.
   Google Scholar
• Krippendorff, K. (2018). Content analysis: An introduction to its methodology. Sage.
   Google Scholar
• Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. 10.1145/3065386
   Web of Science ®Google Scholar
• Liu, F., Ford, D., Parnin, C., & Dabbish, L. (2017). Selfies as social movements: Influences on participation and perceived impact on stereotypes. Proceedings of the ACM on Human-Computer Interaction, 1(72), 1–21.
   Google Scholar
• Lorenz-Spreen, P., Oswald, L., Lewandowsky, S., & Hertwig, R. (2023). A systematic review of worldwide causal and correlational evidence on digital media and democracy. Nature Human Behaviour, 7(1), 74–101. 10.1038/s41562-022-01460-1
   PubMed Web of Science ®Google Scholar
• Lu, Y., & Shen, C. (2023). Unpacking multimodal fact-checking: Features and engagement of fact-checking videos on Chinese TikTok (Douyin). Social Media+ Society, 9(1), 205630512211504. 10.1177/20563051221150406
   Web of Science ®Google Scholar
• Manago, A. M., Graham, M. B., Greenfield, P. M., & Salimkhan, G. (2008). Self-presentation and gender on MySpace. Journal of Applied Developmental Psychology, 29(6), 446–458. 10.1016/j.appdev.2008.07.001
   Web of Science ®Google Scholar
• Molder, A. L., Lakind, A., Clemmons, Z. E., & Chen, K. (2022). Framing the global youth climate movement: A qualitative content analysis of Greta Thunberg’s moral, hopeful, and motivational framing on instagram. The International Journal of Press/politics, 27(3), 668–695. 10.1177/19401612211055691
   Web of Science ®Google Scholar
• Mooseder, A., Brantner, C., Zamith, R., & Pfeffer, J. (2023). (Social) media logics and visualizing climate change: 10 years of# climatechange images on twitter. Social Media+ Society, 9(1), 205630512311643. 10.1177/20563051231164310
   Web of Science ®Google Scholar
• Muise, D., Lu, Y., Pan, J., & Reeves, B. (2022). Selectively localized: Temporal and visual structure of smartphone screen activity across media environments. Mobile Media & Communication, 10(3), 487–509. 10.1177/20501579221080333
   Web of Science ®Google Scholar
• Murashka, V., Liu, J., & Peng, Y. (2021). Fitspiration on instagram: Identifying topic clusters in user comments to posts with objectification features. Health Communication, 36(12), 1537–1548. 10.1080/10410236.2020.1773702
   PubMed Web of Science ®Google Scholar
• Nesi, J., Choukas-Bradley, S., & Prinstein, M. J. (2018). Transformation of adolescent peer relations in the social media context: Part 1—a theoretical framework and application to dyadic peer relationships. Clinical Child and Family Psychology Review, 21(3), 267–294. 10.1007/s10567-018-0261-x
   PubMed Web of Science ®Google Scholar
• Ohme, J., Araujo, T., Boeschoten, L., Freelon, D., Ram, N., Reeves, B. B., & Robinson, T. N. (2023). Digital trace data collection for social media effects research: APIs, data donation, and (screen) tracking. Communication Methods and Measures, Advance online publication, 1–18. 10.1080/19312458.2023.2181319
   Web of Science ®Google Scholar
• Olivola, C. Y., & Todorov, A. (2010). Elected in 100 milliseconds: Appearance-based trait inferences and voting. Journal of Nonverbal Behavior, 34(2), 83–110. 10.1007/s10919-009-0082-1
   Web of Science ®Google Scholar
• OpenAI, 2023 GPT-4 technical report https://arxiv.org/abs/2303.08774 doi:10.48550/arXiv.2303.08774
   Google Scholar
• Otto, L. P., Thomas, F., Glogger, I., & De Vreese, C. H. (2022). Linking media content and survey data in a dynamic and digital media environment–mobile longitudinal linkage analysis. Digital Journalism, 10(1), 200–215. 10.1080/21670811.2021.1890169
   Web of Science ®Google Scholar
• Peng, Y.(2017). Time travel with one click: Effects of digital filters on perceptions of photographs. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA (pp. 6000–6011).
   Google Scholar
• Peng, Y. (2018). Same candidates, different faces: Uncovering media bias in visual portrayals of presidential candidates with computer vision. Journal of Communication, 68(5), 920–941. 10.1093/joc/jqy041
   Web of Science ®Google Scholar
• Peng, Y. (2021). What makes politicians’ instagram posts popular? analyzing social media strategies of candidates and office holders with computer vision. The International Journal of Press/politics, 26(1), 143–166. 10.1177/1940161220964769
   Web of Science ®Google Scholar
• Peng, Y., Lu, Y., & Shen, C. (2023). An agenda for studying credibility perceptions of visual misinformation. Political Communication, 40(2), 225–237. Advance online publication 10.1080/10584609.2023.2175398
   Web of Science ®Google Scholar
• Phillips, P. J., Jiang, F., Narvekar, A., Ayyad, J., & O’Toole, A. J. (2011). An other-race effect for face recognition algorithms. ACM Transactions on Applied Perception (TAP), 8(2), 1–11. 10.1145/1870076.1870082
   Web of Science ®Google Scholar
• Pouwels, J. L., Valkenburg, P. M., Beyens, I., van Driel, I. I., & Keijsers, L. (2021). Social media use and friendship closeness in adolescents’ daily lives: An experience sampling study. Developmental Psychology, 57(2), 309–323. 10.1037/dev0001148
   PubMed Web of Science ®Google Scholar
• Radford, A. Kim, J. W. Hallacy, C. Ramesh, A. Goh, G. Agarwal, S. & Sutskever, I.(2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (pp. 8748–8763). PMLR.
   Google Scholar
• Ramesh, A. Dhariwal, P. Nichol, A. Chu, C. & Chen, M. arXiv preprint arXiv:2204.06125. (2022). Hierarchical text-conditional image generation with clip latents. 1(2), 3.
   Google Scholar
• Reece, A. G. & Danforth, C. M.(2017). Instagram photos reveal predictive markers of depression. EPJ data science, 6(1), 15.
   Web of Science ®Google Scholar
• Riddle, K., & Martins, N. (2022). A content analysis of American primetime television: A 20-year update of the national television violence studies. Journal of Communication, 72(1), 33–58. 10.1093/joc/jqab043
   Web of Science ®Google Scholar
• Schwemmer, C., Knight, C., Bello-Pardo, E. D., Oklobdzija, S., Schoonvelde, M., & Lockhart, J. W. (2020). Diagnosing gender bias in image recognition systems. Socius: Sociological Research for a Dynamic World, 6, 6. 10.1177/2378023120967171
   Google Scholar
• Shah, D. V., Sun, Z., Bucy, E. P., Kim, S. J., Sun, Y., Li, M., & Sethares, W. (2023). Building an ICCN multimodal classifier of aggressive political debate style: Towards a computational understanding of candidate performance over time. Communication Methods and Measures, Advance online publication, 1–18. 10.1080/19312458.2023.2227093
   Web of Science ®Google Scholar
• Smeulders, A. W., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000). Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), 1349–1380. 10.1109/34.895972
   Web of Science ®Google Scholar
• Steinert-Threlkeld, Z. C., Chan, A. M., & Joo, J. (2022). How state and protester violence affect protest dynamics. The Journal of Politics, 84(2), 798–813. 10.1086/715600
   PubMed Web of Science ®Google Scholar
• Tiggemann, M., & Zaccardo, M. (2018). ‘Strong is the new skinny’: A content analysis of #fitspiration images on instagram. Journal of Health Psychology, 23(8), 1003–1011. 10.1177/1359105316639436
   PubMed Web of Science ®Google Scholar
• Valkenburg, P. M. (2017). Understanding self-effects in social media. Human Communication Research, 43(4), 477–490. 10.1111/hcre.12113
   Web of Science ®Google Scholar
• Valkenburg, P., Beyens, I., Pouwels, J. L., van Driel, I. I., & Keijsers, L. (2021). Social media use and adolescents’ self-esteem: Heading for a person-specific media effects paradigm. Journal of Communication, 71(1), 56–78. 10.1093/joc/jqaa039
   Web of Science ®Google Scholar
• Valkenburg, P. M. Peter, J. & Walther, J. B.(2016). Media effects: Theory and research. Annual Review of Psychology, 67, 315–338.
   PubMed Web of Science ®Google Scholar
• van Atteveldt, W., & Peng, T. Q. (Eds.). (2021). Computational methods for communication science. Routledge.
   Google Scholar
• Vendemia, M. A., & DeAndrea, D. C. (2018). The effects of viewing thin, sexualized selfies on instagram: Investigating the role of image source and awareness of photo editing practices. Body Image, 27, 118–127. 10.1016/j.bodyim.2018.08.013
   PubMed Web of Science ®Google Scholar
• Webb Williams, N., Casas, A., & Wilkerson, J. D. (2020). Images as data for social science research: An introduction to convolutional neural nets for image classification. Cambridge University Press.
   Google Scholar
• Wojcieszak, M., de Leeuw, S., Menchen-Trevino, E., Lee, S., Huang-Isherwood, K. M., & Weeks, B. (2021). No polarization from partisan news: over-time evidence from trace data. The International Journal of Press/politics, Advance online publication. 10.1177/19401612211047194
   PubMed Web of Science ®Google Scholar
• Xi, N., Ma, D., Liou, M., Steinert-Threlkeld, Z. C., Anastasopoulos, J., & Joo, J. (2020). Understanding the political ideology of legislators from social media images. Proceedings of the international AAAI conference on web and social media, Atlanta, Georgia, USA. 726–737.
   Google Scholar
• Yang, Y. Davis, T. & Hindman, M.(2023). Visual misinformation on Facebook. Journal of Communication, 73(4), 316–328.
   Web of Science ®Google Scholar
• Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019). From recognition to cognition: Visual commonsense reasoning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, USA. 6720–6731.
   Google Scholar
• Zhang, H., & Pan, J. (2019). Casm: A deep-learning approach for identifying collective action events with text and image data from social media. Sociological Methodology, 49(1), 1–57. 10.1177/0081175019860244
   Google Scholar
• Zhang, H., & Peng, Y. (2022). Image clustering: An unsupervised approach to categorize visual data in social science research. Sociological Methods & Research, Advance online publication. 10.1177/00491241221082603
   Web of Science ®Google Scholar
• Zhang, B. & Pinto, J.(2021). Changing the world one meme at a time: The effects of climate change memes on civic engagement intentions. Environmental Communication, 15(6), 749–764.
   Google Scholar