Feeling, thinking, and not seeing: how images engage and disengage in an information-saturated world – a neurophenomenological perspective

ABSTRACT

We can no longer realise images solely through semiotic theories of interpretation and judgement fixed to earlier modes of communication, such as print. Instead, we engage with images through various networked digital devices and online social interactions. These engagements offer many possible experiences with images, some of which we have agency and others that are purely autonomic and some interactions resembling a healthy-bodied manifestation of visual agnosia or the loss of ability to recognise and identify. This paper utilises research from the ‘Fast Image’ study, a study of graphic design students and practitioners and their views on photographic image use, comparing print and online media. The author uses an interpretive approach supported by mixed data-gathering methods, including photo-elicitation, interviews and semi-structured questions. This discussion encourages advancing visual literacy and visual culture discourses to incorporate a Neurophenomenological approach toward understanding the effects of emerging technologies and viewing environments on photographic image use. These effects include sensory and cognitive responses to images and the influences of external stimuli on our phenomenological apprehension of images.

KEYWORDS:

• Digital images
• photography
• neurophenomenology
• perception
• technology
• visual literacy

Introduction

With the increasing coalescence of human and digital interaction, our bodies and minds adapt to evolving digital environments, responding to the affordances of the particular technologies we use. We interact with images in many ways in these environments, via desktop computers in study or work settings, during travel, while multitasking on smartphones, in social situations, or seeking entertainment. Our interactions with images give us emotional affectivity, provided, for example, through personal and intimate interactions with images viewed on smartphones. However, at other times our interactions prevent us from a singular, focused engagement with them, occasioning not seeing or only the partial seeing of images.

Using data gathered from the author’s ‘Fast Image’ study (Marotta 2022) on photographic image apprehension by graphic design students and practitioners and supporting literature, this paper describes how our interactions with images occur, the resulting experiences during the viewing process and the final effect on our phenomenological apprehension of images. These interactions are discussed primarily concerning the digital environment; digital environments are those where a change in cognitive functioning is most significant (Marotta 2022). However, comparisons with images viewed in the printed environment are made when necessary to highlight differences between participant reactions toward those viewed in digital mediums, the nature of the change in engagement with images, and how apprehension occurs to achieve a phenomenological outcome.

Scope of the fast image study

The ‘Fast Image’ study aimed to ascertain and describe if and how our apprehension of images has changed from print to digital mediums and to what degree technology influences these changes. The researcher investigated the image-viewing experiences and practices of five established Graphic Design and Advertising industry practitioners and ten Graphic Design students from three Australian universities through a photo elicitation exercise where participants compared their image-viewing experiences in the printed Australian Geographic Magazine, Tasmania 2016 edition, with its online counterpart, the Australian Geographic Magazine Website using the same set of images. In addition, further information related to image-viewing experiences was collected using research methods such as questionnaires, focus groups and individual interviews. These methods were designed to gain insight into participants’ experiences in other situations using other technologies and platforms, such as smartphones and social media.

While this paper uses first-person approaches to provide insights into the image-viewing experience, the discussion encourages including third-person perspectives using a neurophenomenological approach to understanding image apprehension concerning our interactions with the viewing environment and the interdependence of the mind, brain, and body toward phenomenological experience.

The neurophenomenological perspective

Francisco Varela (1996) originally proposed Neurophenomenology as a methodological framework for understanding consciousness through embodied and situated lived experience combining the first-person experiential approaches of phenomenology with third-person neuroscientific methods (Gallagher 2009). Neurophenomenology is derived from what Varela calls the enactive approach in cognitive science (Varela, Thompson, and Rosch 1993). This approach includes the idea that cognitive processes emerge from the interdependent sensorimotor actions of the body, nervous system, and environment (Thompson, Lutz, and Cosmelli 2005).

To use these ideas to help explain the relationship between our cognitive and biological responses to images and our final apprehension of them, we must understand how the nervous system and the brain work together to provide a phenomenological outcome to our online visual experiences. The interplay between the nervous system and the brain shapes the resultant interpretation of what we see, indicating that mental phenomena arise from the brain’s ‘self-organised activity’ (Fazelpour and Thompson 2015, 223). Kahn (2013) posits that self-organisation is our biological response to environmental change. In other words, the brain influences and adapts to what we see based on the functional interactions of its components. Furthermore, we respond to our encounters with objects and the technological environment through the connections between our experiences, emotions, feelings and thoughts. So how do we experience images through technology and our environment, and how do they affect our visualisation to apprehension process?

An investigation of the literature provides context through a neuropsychological perspective to the question above. It is presented through pertinent theories of the brain’s impact on behavioural aspects of the nervous system concerning the physical, mental and emotional features of the human experience when viewing images in a digital environment.

Links between phenomenological interpretation and brain functions indicate the importance of people’s kinesthetic experiences toward images. To explain this idea further, let us consider vision primarily enabled through optical, physiological and neural processes as the first part of a neurological to phenomenological sequence. This process is illustrated in Figure 1. Albertazzi (2013) argues the standard scientific practice has been to consider the meanings and qualities of visual phenomena as extraneous to these processes. However, there is an increasing body of work from scientific (Khachouf, Poletti, and Pagnoni 2013; Luck and Vogel 2013; Zeki 2007) and philosophical (Hansen 2011; Hayles 2012; Malabou 2008; Massumi 2002) fields seeking to validate and articulate connections between the objective and subjective in visual phenomena. These connections allow for a phenomenological experiencing of our physiological awareness of the world. (Albertazzi 2013; Bagdasaryan and Le Van Quyen 2013) argue that the phenomenological results of our physiological awareness of the world.

Figure 1. ACIVA Model of Visual Apprehension. Marotta 2022.

Hansen offers further insight by describing ‘a completely different kind of image – our perceptual image of the world outside’ (Hansen 2011, 84) arriving from the process of seeing. This action includes our conception of the mental image (the one formed in the brain), beginning with the image we see transmitted to and processed by the brain. Hansen is referring to the process whereby the mental image is converted to the phenomenological image in the biological to phenomenological sequence of seeing. However, what happens during the conversion process, and how does this process affect our phenomenological apprehension of the image? Hansen’s observation suggests a meaningful connection between the neurological and the phenomenological that could be further examined through combined scientific and social sciences studies (neurophenomenology), offering the possibility for valuable insights into the influence of technology on cognitive and somatosensory (perception through the senses) visual information processing and the subsequent phenomenological experience (Marotta 2022).

When the body and mind cooperate in the translation of stimuli [the image] through the act of visual perception, phenomenological affirmation is observed through pre-reflective human experience operating with other brain faculties, such as autobiographical memory activation, as indicated in the participant comment below. Malibou (2008) posits that these biological to mental translations are more than scientifically predicated.

One of the images I remember was the school group … because of the happiness of my family before we had problems. I remember happiness, enjoying my childhood. (8s)

The comment above aligns with other participant responses, indicating the affective role memory plays in image recall, evoking the emotion of happiness from the participant and associating it with the image viewed. Emotionally experiencing images involves a dynamic interaction between neural processes and bodily sensations (Marotta 2022).

Therefore, a contemporary philosophical and neuroscientific approach to investigating the cooperation of body and mind toward images will offer new and richer possibilities for understanding how we engage with and understand them.

The following sections expand upon the neurophenomenological aspects of the image-viewing experience. These include understanding our perceptual experience of images through relevant neuroscientific theories and how external and temporal effects and sensorial experiences influence neurological and biological actions toward phenomenological apprehension.

External and temporal effects on image viewing

Visual psychophysics

Physical stimuli in the outside world impact our visual processing of images and actions toward them. Visual Psychophysics, or the study of visual perception, examines this connection between external stimulus and human action (Lu and Barbara Dosher 2014). In the context of this discussion, it is a scientific field that helps us understand how external stimuli affect how we view and use images involving the temporal aspects of viewing and the brain’s responses during this process.

Considering how external factors influence our visual capabilities and neurological and phenomenological processes is essential in identifying the extent of their impact on our viewing experience. This will help us understand how visual apprehension operates concerning the type of cognitive responses generated during the image-viewing process, how this occurs and to what intensity we experience and understand images relative to the viewing environment.

Visual noise and visual masking

Participant responses during the original study indicated the impact of the external effects of visual noise on the cognition of information, as illustrated by the comment below concerning viewing images on a smartphone.

Smartphones are very distracting. There are too many things there for you to focus on the image. (1s)

This discussion defines ‘visual noise’ as random visual stimuli affecting our attention and hindering our ability to see. Associations are evident between online-related cognitive impacts and structural changes in the brain, including the effects of ‘rapid attention shifting, distractibility, reduced deliberations and impaired executive control abilities’ (Loh and Kanai 2016, 506).

For example, visual masking is an instance of visual noise. It is a phenomenon of visual perception that produces situations that interfere with the viewing task and contributes to the experience of ‘looking but not seeing’ – visual access without phenomenal visual consciousness, preventing the progression toward higher-order processing, such as assessing the semantic properties of the image. The impediment toward visual apprehension can occur when competing stimuli mask the image being viewed, as demonstrated by the following participant’s response when asked whether exposure to multiple online information streams affects the communication value of the image.

There’s so much there. It’s like having a noticeboard. There’s stuff overlapping, sooner or later you just switch off. I think it affects the viewers ability to absorb information. (10s)

The communication value of images viewed in this situation is diminished due to the tendency to ‘switch off’ in an environment of competing stimuli, creating conditions where the image is not seen or the viewer chooses not to it.

In addition, visual masking theory concerns how stimuli are registered in the mind, including examining ‘feature, object and scene representations, attentional control mechanisms, and phenomenal awareness’ (Breitmeyer 2008, 9). The registration of stimuli is a feature of attention, a vital consideration playing a crucial role in determining whether the visual cortex can process the features of objects or scenes effectively to reach a conscious representation. Lamme (2004) argues that attention does not determine whether stimuli reach a conscious state but determines whether ‘a (conscious) report about stimuli is possible’ (Lamme 2004, 863). To identify the conditions in which stimuli can reach a conscious state, we need to acknowledge the characteristic differences among various stimuli [images] and the diversity of the contexts in which they appear. The following participant expresses the differences in the visual experience comparing desktop computer and smartphone viewing contexts. Furthermore, the participant’s comment indicates different digital devices and viewing environments have conditions that facilitate or prevent focused attention.

If the image is on a [desktop] computer, there are a lot of other things happening. There might be a menu bar, pop up ads or this and that happening. There are a lot of things fighting for attention. If it’s [image] on a Smartphone, there might not be as many things happening around it, so I can focus more on the image. (11p)

Temporal aspects of visual masking

The effects of visual masking on the image-viewing process can be further described through spatiotemporal qualities. To illustrate this process, let us call the primary image to be viewed the ‘target’ image. The presence of another image or stimulus, referred to as a ‘mask’, affects the viewing of the target image, occluding visibility and inhibiting critical information about the image, particularly if the mask image is of high contrast and luminance (Bacon-Macé et al. 2005).

Consequently, we experience the effects of visual masking as visual sensations relative to the presence of other stimuli appearing at temporal intervals. These stimuli can be of ‘different sensory types and origin moving through time and space’ (Hansen 2011, 84), processed by the brain and perceived at a conscious and unconscious level (Lamme 2004). Visual masking affects the way the image appears in consciousness because the stimuli we see before the target image, which Herzog and Koch refer to as ‘feature inheritance’ (Herzog and Christof Koch 2001, 4271); features of one object incorrectly associated with another, affect the act of viewing and bias the way the image is apprehended (Herzog and Christof Koch 2001). Images are considered among the objects to which Herzog and Koch refer. For example, on a Smartphone, one can view a succession of images on an image carousel by swiping them in and out of view. Previous images viewed, those observed in proximity to the target image and those following impact our cognition of the target image as indicated in the following two participant comments. The second comment indicates the temporal nature and the possible impression on memory (memorising a representation of the viewed image at a different time to the one on which we transpose the memory) of Herzog & Koch’s ‘feature inheritance’ effects.

On your phone, you’ve got different things happening at the same time. You’re not concentrating on one specific thing. It affects the way you interpret images because you focus less on them. (11p)

If you see something one day and see another thing another day, what you saw previously in another image could affect how you see a different image. It’s just that kind of business to everything. It affects people for sure. (5s)

In addition to the effects of competing images, the qualities of the viewing technology affect the visibility of the ‘target’ image by diminishing its visibility or altering our attentional intensity and how we interpret the same image under different viewing conditions (see participant comment 8s below). In a digital environment, the rate at which competing visual stimuli (masks) enter the visual field and potentially alter comprehension is more intense than in the printed environment (Marotta 2022). The following participant comments illustrate these effects, with participant 14p indicating viewing the image in print offers more potential for concentrated attention.

On a smartphone and tablet you’re not really putting a lot of attention into the images; you’re just skimming through content mostly. (7s)

You’re never really there when you have all these other things happening. You just have to turn off all your digital devices to really absorb all the information. (2s)

When you’re looking at print you don’t have as many distractions and therefore, you’re concentrating more on the image and what it conveys. (14p)

On digital devices, you're not getting the same output on every single device. The image each person sees can be interpreted differently depending on which device they're viewing it. There may be difference in contrast, in colours or shades. This can make significant differences in the way we interpret images. (8s)

The temporal applications of the regions in the brain's visual cortex are a further influence on the visual masking process. How these regions react toward visual stimuli can reduce or increase the degree to which masking affects our cognition of images. Let us consider phenomenologically experienced visual apprehension as involving the mutual interaction of relevant regions of the brain's visual cortex. These regions are specialised to process different attributes of the visual scene at different micro-temporal moments. This processing occurs in temporal sequence requiring all pertinent areas to operate in unity for apprehension to occur. Zeki describes this as visual consciousness made up of ‘different microconsciousnesses’ (Zeki 2005 , p. 1178) resulting from visual input, cognitive processing and apprehension.

Zeki is referring to the asynchronous processing of visual information by the relevant regions within the brain’s visual cortex (Zeki 2005). There is a relationship between these areas and the temporal correlates of the visual cortex, which process different components of the optical system, such as colour, motion, location and orientation in temporal sequence (Ilias et al. 2014; Zhou et al. 2018).

Accordingly, if given adequate time to process visual input, each attribute of the viewed image is handled in ‘perfect temporal’ and ‘spatial registration’ (Zeki 2007 , p. 583). However, in many digital viewing situations of fragmented attention and short bursts of viewing time, signals from one area of the visual cortex do not bind because they are not able to reach those of another in time (Zeki 2007), resulting in only partial processing of the visual scene. Consequently, the resulting phenomenological experience of the image and the quality of semantic decoding are adversely affected.

Therefore, identifying which situations and technologies intensify visual masking effects and are favourable to the temporal processing aspects of the brain’s visual cortex can help us minimise their impact and develop technologies and methods to enhance viewing and create arresting images in visually cluttered environments effectively.

However, despite the hindering effects of visual masking and the reliance on adequate processing time for visual apprehension, an image may have qualities pertinent to the viewer, allowing it to become visible and override masking effects. For example, the brain’s unconscious processes (‘automatic senses’) may favour one image among many. This favourability may be due to the viewer’s subjective response to salient characteristics within the viewed image, as indicated by the following participant:

It’s like walking around the art gallery. I wait till the picture grabs me. I won’t go and find it; I wait till it finds me. It’s the same when you swipe through things really quickly on a smartphone. Something goes ‘ah stop’, and then that’s the one you spend more time on. (10s)

The automatic process described above validates visual inputs by the brain’s motor processes before the conscious representation of these inputs. These processes allow stimuli not consciously detected or recognised to affect a viewer’s behaviour (Jaśkowski, Skalska, and Verleger 2003) and indicate the relationship between human perception, cognition, and unconscious and conscious processing dynamics; we can process visual information without awareness. Breitmeyer (2014) posits that much of the processing of visual information before conscious registration is, by definition, unconscious.

These cognitive operations mean that various types of visual processing precede perception influencing human sensibilities toward the image, something beyond the ability for direct human intervention. Incorporating these processes into a more extensive ecological theory of visual perception will expand philosophical discussions with image apprehension at its centre (Hansen 2015).

I'm not seeing

The previous section discussed the effects of visual masking as a component of visual noise which can affect our ability to apprehend images in visually cluttered digital environments. This section speculatively presents that visual noise creates a viewing environment where we demonstrate similar impacts to neurological disorders such as visual agnosia. The introduction of this idea extends the preceding discussion to include the occlusive effects of this neuropsychological condition that prevents a person from understanding the meaning of a visually presented stimulus [image], despite the affected individual having an intact sensory and low-level vision (detecting image elements) and typical language and semantic function (Behrmann and Nishimura 2010).

For example, in many instances, digital environments expose us to an overabundance of potentially unnecessary information. This exposure increases the working memory’s load resulting in distraction from the main message or image, making it difficult to perceive and interpret visual information. This paper identifies the relationship between ‘digitally induced’ visual agnosia and our apprehension of images and encourages further research from a neurophenomenological perspective. The following participant comments indicate visual agnosia characteristics during their image-viewing experience.

I can watch a thousand of images and I will absorb just one part of the information from that image. (8s)

You are presented with so much information and you’re just scrolling through it. You’re not absorbing all the information you see. At the end of the day everything is just cluttered in your head and you don’t absorb any real information at all. (7s)

There are two forms of visual agnosia pertinent to this discussion, apperceptive and associative. Biran and Coslett (2003) describe apperceptive visual agnosia as being unable to construct a good perceptual representation or working model from visual input. Associative visual agnosia indicates perception occurs, but recognition does not. This results from the viewer seeing but unable to group image elements despite allocating attention toward it (Abrams and Law 2002).

These agnosias prevent the image from being fully apprehended in cluttered digital environments because working memory, or the ‘small amount’ of the information held in mind and used to execute cognitive tasks such as information processing and comprehension (Cowan 2014) becomes overloaded with extraneous information and has little chance of association with items stored in long term memory that would help phenomenological apprehension.

A detailed definition of the different forms of visual agnosia and how each of them affects our viewing experience is beyond the scope of this paper. However, the author recommends further investigation into identifying the qualities of the various sources of visual noise experienced within the digital media ecosystem and how each source affects our neurological processing of images.

Affect and arousal: experiencing images sensorially

The impact of external factors in the viewing environment on visual cognitive processes has been described. In addition, the previous section discussed the biological aspects of perception using visual psychophysics theories (visual masking and visual agnosia) and their influence on our phenomenal interpretation of images. The following discussion will focus on another critical part of visual perception, our emotional affective response toward images. How are we affected by what we see, and how does this affect influence our cognisance of images? Exploring the answers to these questions is essential because affective experiences, positive and negative, including excitement, shock, happiness and sadness, influence our thoughts and behaviours toward images. Clore and Palmer (2009) describe affect as an embodied response to pleasure or displeasure toward something, connecting affect and cognition. This connection allows affective reactions to provide information about the value and importance of what we experience (Clore and Palmer 2009). People experience their affect as being a reaction to ‘whatever happens to be in mind at the time’ (Clore and Palmer 2009, 25), with positive affects adding value to one's current thoughts and disposition, in turn promoting higher-level cognitive responses including reasoning, and comprehension (Clore et al. 2001; Clore and Palmer 2009).

Alternatively, negative affect leads to a perceptual, stimulus-bound focus invalidating and inhibiting higher-level cognitions (bottom-up processing). However, there are challenges in applying these aspects of affect theory too broadly to the image-viewing experience, particularly if memory processes are involved. For example, if extraordinary enough, a negative response to an image can allow information to progress to higher-order cognitive domains and facilitate a semantic analysis of the image, as noted in the example below.

To further elaborate, let us consider affective responses or subjectively experienced feelings as being delivered through the primary motivational system. This process allows us to express emotion toward stimuli [images] (Frank and Wilson 2020), with affect as the conscious subjective aspect of emotion (Cacioppo and Berntson 1999). Biological drives such as arousal and the desire for emotional experiences are part of the primary motivational system (Niven and Miles 2013) and make up the physiological bases for higher-level affects, such as curiosity and surprise (Buck 1999); the brain, processing sensory information from the external world in the context of sensations from the body. This ‘body to brain’ sequence allows experientially felt interactions to progress to higher-order cognitive processing of the mind. Consequently, how we are affected when viewing images impacts how we experience and process them. For example, when asked to recall a memorable image, the following participant describes Malcolm Browne’s 1963 image of the self-immolating Monk in Saigon and describes the averse affect of shock when recalling the image together with the semantic associations.

The burning monk was such a shocking image. For me it’s hugely memorable because of the loss of life and the sacrifice and all those kinds of things. That’s one that sticks in my head. (9s)

The example above indicates affect is instrumental in promoting cognitive, relational processing, responding indirectly to stimuli (from memory) using prior knowledge. In addition, the response illustrates the cogency of affect on the higher-order cognitive memory processes of recall and retrieval. Clore and Palmer (2009) suggest that this action occurs by the averse effect acting as a conduit for embodied information retrieved from long-term memory.

Storbeck and Clore (2008) posit that affect moderates many of the phenomena in cognitive psychology. The arousal dimension of affect often promotes and intensifies engagement with stimuli, enhancing long-term memory of events. Affective responses such as happiness, sadness and excitement are part of a biological and phenomenological process whereby in ‘emotionally arousing situations’ (Storbeck and Clore 2008, 1834), androgenic hormones are released and enhance memory processes, through which personal experience draws attention to the arousing event (Storbeck and Clore 2008).

In the following example, emotional connections to a photo work with affective responses toward perception, giving significance to a photo’s salient features that may not be possible without emotion derived from personal experience. Emotional influences shape visual perception (Zadra and Clore 2011) and influence attention, judgment, thought, and affective responses. These influences motivate us toward a deeper interaction with the image, beginning with viewing and progressing toward further semantic decoding in an immediate sensory way. Massumi (2002) argues the affective experiences we derive from varying intensities of physical and biological interactions we have with stimuli [images] are the autonomic starting point, which, together with emotion, leads toward higher-level cognitive activities of the mind. Emotional experiences are not fixed states but are shaped by our subjective interpretations and the contexts we encounter images. Therefore, emotions become part of a broader system of signification situated within narratives and action-reaction patterns embedded in embodied and affective processes.

Emotion is qualified intensity, the conventional, consensual point of insertion of intensity into semantically and semiotically formed progressions, into narrativizable action-reaction circuits, into function and meaning (Massumi 2002, 28).

When asked to recall a memorable image, the following participant recalls one that was emotionally salient, associating feelings with the representational content of family.

Seeing images of the 9/11 memorial that I took when my whole family was together. That feels good because we rarely get together. (1s)

The following participant expressed the quality of engagement with an image through an internally felt physical response, ‘spark a feeling inside of me’, which triggered the verbal, externally expressed affective response of ‘wow’ toward engaging visual content. In addition, in the Instagram environment, a personal aspect of social connectedness is evident with the image, a metaphor for a friend's voice.

If I see an image on Instagram, for example, and if it is taken well at a beautiful location, it would spark a feeling inside of me seeing it this way. ‘Wow.’ So, in a way, this image is you speaking to me. (6s)

Therefore, affect and emotion are essential in perceptual experience. The previous participants’ comments indicate emotions arise immediately upon the perception of emotionally evocative stimuli expressed as feelings and behaviour, which involve cognitive processes helping us conceptualise what we see. Furthermore, Bergson posits that ‘there is no perception without affection’ (Bergson 1911, 60), meaning that every act of perceiving an object (or image) offers the potential for one’s body to act on that object as a sensory self-affection of the body. This challenges the mindset that affect is ‘a distinct psychological phenomenon separate from cognition and perception’ (Siegel et al. 2018, 496) and indicates a knowing in an immediate and sensory way (Tomkins 2008).

The following section discusses the corporeal aspects of our interaction with images, expressed through qualities afforded by the viewing technologies of print and screen. These interactions are presented as physical interactions of touch and sensation, externally felt and internally processed in combination with higher-level cognitive brain functions.

Physical and haptic experiences

As a constituent of the visual process, the senses intensify the viewing experience, indelibly transferring the experience of the visual scene to memory; the sensations mediated by the skin link with the outside world (Zimmermann 1989). The combination of the senses offers an immersive experience, stimulating richer responses to what we see, activated through the tactility provided by the viewing medium. Massumi describes tactility as ‘the sensibility of the skin as the surface of contact between the perceiving subject and the perceived object’ (Massumi 1996, 58).

The tactile perception of our environment can be understood as responses to external and internal stimuli; tactile sensibility (exteroceptive), and visceral sensibility (interoceptive) (Massumi 2002). External environmental stimuli trigger haptic sensations by impacting the internal pathways of thought (exteroceptive), often leading to cognition. In addition, others are activated by internal stimuli related to the body’s physiological state (interoception).

A comprehensive body of literature (Barsalou 2010; Glenberg, Witt, and Metcalfe 2013; Shapiro 2011) supports the involvement of the human body’s sensorimotor characteristics in higher cognition. Additionally, bodily actions and reactions toward environmental stimuli influence cognitive processes, actively registering information from the visual scene through motor behaviour, including movements of the eyes and hands together with sensory receptors such as eyes, nose, and skin (Gibson, 1966). We understand through an ecology of world and body; they are a ‘constituent of, and not merely a causal influence on, cognition’ (Shapiro 2011, 4).

The following participant response implies interoceptive visceral sensibilities emanating from an internally generated positive affect expressed toward the ‘visually appealing’ image as a feeling or sensation (‘vibes’).

It was a very visually appealing image. It gives you positive vibes. (7s)

The following participant responses indicate extroperceptive tactile sensibilities operating. These sensibilities promote further engagement with the viewed image, leading to higher-order cognitive processes of perception, allowing for the interpretation of information and the acquisition of meaning, as indicated in the second response.

If you’re viewing an image online, it’s not tactile like something in print. Sometimes the texture of the paper feels nice, or the smell of the ink. (11s)

It was a tangible book, meaning I could feel the true meaning of the image. (1s)

Gibson (2014) posits that ‘visual control of the hands is inseparably connected with the visual perception of objects’ and that manipulation of a surface ‘subserves many other forms of behaviour’ (Gibson 2014, 224) facilitating and influencing the quality of perception as indicated by the following participant comments.

I like the experience of holding something physical in your hands. It gives you more. (3s)

It’s hands-on, interactive, more authentic. (5s)

The images stuck in my head a little better maybe because of the physical feel of paper while analysing. (2s)

Participants in the study did not express these sensory phenomenological experiences as much toward their digital viewing experiences as they did when viewing images in print. The tactility afforded by screen technology attempts to simulate physical bodily actions. However, the rich kinesthesia experienced through touching a malleable surface such as paper, the synergism of pressure exerted onto the skin and blood vessels, and the subsequent neuronal message to the brain indicates a connection between a more intense neurological percept of reality with experiential apprehension when interacting with printed images.

Nevertheless, increasingly, newer technologies such as virtual, augmented, and mixed realities continue to replicate these sensory experiences with some success (Gatter, Hüttl-Maack, and Rauschnabel 2021; Kautish and Khare 2022), the inclusion of which is beyond the scope of this paper. However, it must be noted that through touch screen affordances, devices such as tablets and smartphones can offer the richness of another type of sensory engagement, one combining vision and the tactility of image manipulation, stimulating the visual senses and resulting in immersive, affective responses toward the image-viewing experience.

Kroupi et al. (2016) regard immersive qualities as enhancing depth perception, reality and emotion in immersive multimedia environments. Participants enjoyed the interactive possibilities offered by screen technologies allowing them to engage with visual elements. They described these viewing experiences in terms of zooming in and out and the ability to change the magnification of the image. These interactive affordances offered an immersive 3D quality to the viewing engagement, as expressed by the following participant.

You really experience the panoramic 360 view. If it’s a big landscape of a mountain range, you can zoom in over there. It’s not just a flat thing anymore. It’s got so much more depth, its nearly like a movie now. (5s)

Consequently, the resultant hedonic experiences acquired through various visual technologies play an essential part in acquiring knowledge and meaning (Bloch 2008; Dhanani 1994; Fulkerson 2020; Paterson 2016). Therefore, when we are interacting with images, we are participating in an ecology of tactile functionality and multisensory embodied experience through haptic perception (Marks 2008).

A neuroscientific to phenomenological framework: the aciva model (Affective, cognitive interaction toward visual apprehension)

The previous sections indicated that the impetus for image apprehension comes mainly from the interaction between the brain, body, mind, and environment. Furthermore, many elements in the visual apprehension process activate specific cognitive pathways during this interaction. They contextualise our responses and shape the possibilities of how images are perceived. These elements include semiotic content, emotional valence, arousal, personal intent when viewing, the qualities of the viewing device, the context of use and the source of the image.

We encounter combinations of these elements simultaneously or sequentially – streams of stimuli that trigger either the appetitive or aversive motivational systems (Schupp et al. 2012). One system organises a person’s response to rewarding or appetitive stimuli, and the other organises the individual’s response to unpleasant or aversive stimuli. Although applied mainly in the fields of Psychology (Corr 2013; Buck 1999) and Neuroscience (Bissonette et al. 2014) and as a framework to understand adaptive and reactive behaviours towards threat and reward cues as part of an organism’s drive to survive (Andreatta and Pauli 2015: Lang and Bradley 2013), aspects of these systems can be applied toward understanding the image viewing process. These include their role in enhancing or diminishing the processes of memory and perception (Tomkins 1982), attention allocation, information intake (Lang and Bradley 2013), and motivation (Clore and Huntsinger 2007) for engagement or disengagement toward an image (see Figure 1).

In addition to the influence of the appetitive and aversive motivational systems on our perceptive processes, the influences of bottom-up and top-down processing on our cognition (Theeuwes 2010) can further aid our understanding of how we apprehend images.

Bottom-up and top-down processing help us make sense of what we see. Our encounters with images, when experienced sensorially, shape our perception during bottom-up processing without the imposition of preconceived ideas. At the same time, we can use our pre-existing knowledge and expectations in top-down processing to interpret what we see. Firstly, our reaction to these encounters can be automatic in response to motivationally relevant images and promote ongoing cognitive processing (Lang 2006). In this reaction, the image is experienced sensorially (bottom-up processing) or by cognitive control, where sensory information is used to guide and select motor actions. This process includes deciding whether to stop (aversive) or continue (appetitive) the viewing process (Scott 2016). Secondly, our reaction can be intentional and semantically motivated (top-down processing). For example, when directing our intention toward an activity such as a Google image search. In top-down processing, perception is partly driven by prior learned experience projected onto antecedent cortical areas with top-level processing creating a ‘predictive’ coding of the image (Gilbert and Wu 2013).

In the bottom-up processing example, the nature of the encounter affects how individual emotional cues are processed and can ‘facilitate attention allocation, information intake and sympathetic arousal’ (Lang and Bradley 2013, 230)^. These emotional cues determine whether an image is salient enough to motivate the viewer toward deeper engagement making the image more memorable. In the Google image search example (top-down processing), the viewer is looking for a particular type of image, forming a semantic expectation in the ‘mind’s eye’ of one already known. The relevant neurons then carry this information (Kim, Park, and Suh 2020), facilitating the recognition of the desired image choice, which is then available for further semiotic analysis. This process represents the phenomenological act of intentionality toward the image, an a priori of lived experience. Therefore, each processual stage can progress or diminish engagement with the image being viewed and affects the quality of the final interpretive outcome (see Figure 1).

Figure 1, the ACIVA model (Affective and Cognitive Interaction toward Visual Apprehension), illustrates the cognitive process from the initial reception of the image through to our phenomenological apprehension of it. The process of apprehension can progress from an intellectual type of top-down processing or a bottom-up process driven by the senses. Beginning with the example of bottom-up cognitive processing, we can see that after the visual input (Figure 1, step 1), a corporeal interaction (Figure 1, step 2) occurs, affecting motivation and the quality of interaction (engagement or disengagement) one will have toward an image. Finally, image processing occurs on a conscious or unconscious level, translating the affective experience into conscious perception (somatosensory process, Figure 1, step 3b) (Marotta 2022).

For example, observing the top-down cognitive process in the ACIVA model, apprehension can occur either consciously if the image appears in the dominant visual sphere of focus or unconsciously (Figure 1, step 3a) when the brain automatically processes all or part of an image that may be invisible or masked. Dehaene refers to this as subliminal priming when an image is experienced below the perception threshold (Dehaene 2009). If the image is perceived consciously, decoding occurs through phenomenologically interpreted semiotic modes of analysis (Figure1, step 4). These include world view, group norms and symbolic interactionist perspectives of understanding meaning through people’s social interactions – a part of the social media image environment.

Top-down and bottom-up processing can also influence each other (Schipper, Ernst, and Fahle 2008) through a feedback loop involving the interaction between prior knowledge and sensory information (Dura-Bernal, Wennekers, and Denham 2012). For example, when the makeup of an image has enough salience to attract the viewer’s attention, the image is initially stored in short-term memory (Figure 1, step 3c). This action occurs in the following way: Sensory information from the somatosensory (Figure 1, step 3b) region of the brain is transferred (fed forward) to higher-order areas of the brain (Bullier 2001) for higher-level processing. The feedback connections from these areas then move information in the reverse direction (Khorsand, Moore, and Soltani 2015), modulating responses by activating cortical processes such as attention and figure-ground segregation or grouping, which allows us to perceive the [image] more fully (Roelfsema, Tolboom, and Khayat 2007). The image can then be further processed and semantically decoded using higher neural functions (Figure 1, steps 3–4).

Conclusion and recommendations

This paper aims to expand current photographic image theory research and practice. It highlights the need to consolidate mind, brain, and body approaches to visual literacy pedagogy and practices when investigating how we apprehend photographic images. While not exhaustively presenting the findings, this paper has focused on particular insights from the ‘Fast Image’ study. On a practical level, the integration of the ACIVA model of image apprehension (Figure 1) can be adopted and incorporated into visual literacy curriculum at a tertiary level. The purpose of this model is to engage students to understand better how images make meaning through a neurophenomenological process, how visuality and our engagement with images are changing and how the different and evolving presentations of the image affect our emotional and critical engagement and experience at each stage of the visual apprehension process illustrated in Figure 1.

On a theoretical level, this paper encourages further discussion into the changing nature and our understanding of image apprehension and the need to evolve interpretive and semiotic image theories to incorporate neuroscientific ideas involving the coalescence of mind, brain and body as we attempt to create meaning in what we see. In addition, we need to consider the fluidity of images, observing them moving through diverse and emerging temporal spaces provided by new visual and experiential technologies such as Generative A.I. and virtual, augmented and mixed realities. These visual and technical spaces come with their associated affordances, where images are accessed in environments that bring intensities of visual noise and intent, influencing the degree and the quality to which we experience images. How are images understood in these spaces?

Additionally, the synergy between neuroscientific and phenomenological aspects of image apprehension can be further explored using science-oriented research techniques such as eye tracking, EEG (electroencephalogram) and fMRI (Functional Magnetic Resonance Imaging) to map visual activity and stimuli to brain responses during image viewing. These techniques were previously limited to research investigating cognitive functional aspects of viewing, such as effects of viewing conditions on perceptual load (Lavie 2005), internet search tasks and reading text (Small et al. 2009), visual masking effects (Ansorge et al. 2008) and ‘real world’ scene perceptions (Wu, Wick, and Pomplun 2014). In contrast, others related to visual communication focus on areas such as online advertisement cues (Yen and Chiang 2021), evaluating the appeal of body representations in artistic and photographic images through neurocognitive processing (Lutz et al. 2013) and investigating consumers’ sensorimotor, cognitive, and affective responses to marketing stimuli (Suomala et al. 2012). Little research has examined photographic image apprehension incorporating neurophenomenological approaches. Combining the advantages of quantitative and qualitative research methods into a research project offers the possibility for a broader and richer range of findings not confined to only one approach (Creswell and Plano Clark 2007; Johnson and Onwuegbuzie 2004).

Limitations

In the original ‘Fast Image’ study, the small sample size of 10 students and five practitioners made comprehensive statistical analysis difficult (Marotta 2022). However, the study was conducted from a qualitative standpoint and designed as an exploratory gathering of information that successfully provided insights into the experiential and cognitive aspects of image viewing which are the focus of this paper.

Final note

The researcher acknowledges the experiential and cognitive overlap of effect and affect when we engage with images in different situations, including digital and print environments. Although each viewing situation has affordances triggering the apprehension process and influencing how it occurs differently, viewing in these situations is experienced through degrees of bottom-up and top-down processing (Figure 1), some fast and some automatic, of which we are unaware (Varela, Thompson, and Rosch 1993) and others more intentional and contemplative.

The emerging neuroscientific and philosophical theories of Varela (1996), Bechara, Damasio, and Damasio (2000), Lombardi (2019), Massumi (2002), Dehaene (2009) and Hansen (2011; 2015) are essential in understanding how we apprehend images across different technologies. These theories are about human cognition that is embodied and enactive (Lombardi 2019; Varela 1996), shifting the discussion about the way we understand and engage with images, one that is inclusive of but not solely technologically or cognitively driven but acknowledges the interaction of mind, brain and body combining in different intensities of experience and cognition leading to a wide range of possibilities of understandings or misunderstandings. From this perspective, through various human cognitive and biological actions influenced by environments and technologies, we feel, think and do not see the image.

Acknowledgements

Ethical approval for this project was given by Central Queensland University Human Ethics Committee: H15/07-175

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Thomas Marotta

Tom Marotta (Ph.D.) is a sessional lecturer at Central Queensland University with interests in the field of visual communication and digital media.