Vision is (probably) Informationally Encapsulated from Flavor (Part II)

By Daniel Tippens

Some notes on attention and sensory penetration

I intend to argue that vision is informationally encapsulated (just “encapsulated” from here on out) from flavor. In the last post, we discussed the ideas of encapsulation and sensory penetration. The notion of encapsulation that I am working with takes place at the level of experience, and sensory penetration takes place when one sense modality directly influences another sense modality’s respective experience and is not a result of a quirk. If one sense X is sensorily penetrated by Y, then X is *not* encapsulated from Y.

There is still one more note here which needs to be made. At the beginning of the last post, I said that the following steps precede visual experience:

First: Our eyes receive inputs.
Second: Brain regions dedicated to vision, such as V1-V4, process those inputs.
Third: An output is produced.
Fourth: We have a visual experience.

This isn’t quite accurate, or, at least, not detailed enough. There is still another step that should be inserted, that of attentional selection:

First: Our eyes receive inputs.
Second: Selective attention selects certain inputs to enter into processing within the visual system.
Third: Brain regions dedicated to vision, such as V1-V4, process those inputs.
Fourth: An output is produced.
Fifth: We have a visual experience (which may or may not just be step 4).

When we look at things in our environment, we can selectively attend to some things over others. Indeed, at this very moment you are selectively attending to this word instead of the word “output” several lines above. Our eyes are bombarded with inputs at any given moment, far too many for our visual system to be able to handle. So, attention acts as a gatekeeper, allowing some inputs to enter into further visual processing, and not others [1].

There are two types of attention: overt and covert. Overt attention is when one actually moves one’s eyes in the direction of the stimulus they wind up attending to. This could be captured by recalling the case of the firetruck in the last post. Covert attention takes place when one’s eyes remain fixed, but one shifts one’s attention around nonetheless. Think of when you are at a party, in a conversation with someone, and you want to keep an eye on what your spouse is doing in the corner of the room. In order to avoid being rude, you are likely to keep your eyes fixed on your interlocutor, but covertly attend to your spouse.

We need to discuss covert attention in order to rule out one last case that does not count as sensory penetration. We said that sensory penetration is when one sense directly influences another sense’s resulting experience and is not a result of a quirk (thereby ruling out synesthesia). But now consider this case involving covert attention.

Imagine that you are at the movie theatre. On the screen, everything is silent, and characters are moving around. You are attending to the center of the movie screen. A sound is presented on the left-hand side of the screen, thereby drawing your covert attention to it. Your visual experience has been influenced by audition and was not the result of a quirk, but it still doesn’t count as a case of sensory penetration. The reason is that the sound merely shifted your attention, which changed the inputs that went through for visual processing, but didn’t alter the processing of those inputs. So, our final understanding of sensory penetration is when one sense directly influences another sense’s resulting experience, is not a result of a quirk, and is not a result of a shift in attention alone.

What is flavor?

In order to argue that vision is encapsulated from flavor, we should get clear on what flavor is. Many people confuse “taste” with “flavor [2].” Experientially, taste is the experience which results from gustatory receptors on the tongue and around the mouth. It is widely believed that there are 5 basic tastes (and 5 basic taste experiences): sweet, sour, bitter, salty, and umami.

But we don’t just experience the five basic tastes. We experience chocolate, watermelon, and creamy soup. We experience flavors. In fact, most of our flavor experience doesn’t come from taste, so where do these extra experiential qualities come from? Enter olfaction (smell) [3], which comes in two types: orthonasal and retronasal. Orthonasal olfaction takes place when odorants (the inputs to our olfactory receptors in the nose) enter the nose from the outside. When you think of orthonasal olfaction, think of sniffing as the paradigm action which facilitates this kind of smell.

Though orthonasal olfaction doesn’t really play a role in flavor experience, retronasal olfaction does. Retronasal olfaction takes place when odorants enter the nose through the nasal cavity in the back of the mouth. If you have ever smoked hookah, basically the entirety of the flavor you experience is due to retronasal olfaction. If you eat a skittle and pinch your nose, you will notice that much of its flavor seems to disappear. Retronasal olfaction’s role in flavor experiences also explains why food tastes so different when you have a cold.

Interestingly, our flavor experiences seem to be sensorily penetrated by every single modality. Touch can influence flavor perception [4], as when we eat a creamy soup. Audition can also play a role, as when we experience the flavor of potato chips to differ depending on their crunchiness [5]. Vision, too, has some causal influence. The color of the food, or even its packaging, can influence our flavor experiences [6].

It is important to note the difference between a sensory experience partially constituting the experience of flavor, as opposed to merely causally influencing it. The color of a food might causally influence our flavor experience, but it doesn’t constitute it. We don’t experience the colors as being spatially located in the mouth, and things don’t taste red. Taste and retronasal olfactory experiences constitute the experience of flavor, but colors only causally influence, i.e sensorily penetrate, flavor experiences.

So flavor experience, for the purposes of this paper, is the experience one has thanks to retronasal olfaction and taste which can be sensorily penetrated, i.e directly causally influenced by, the other sense modalities, and is spatially located in the mouth [7]. What the flavor system tracks are, *ahem*, flavors; chemical properties of the foodstuff that one ingests. As Barry Smith has put it, “Chefs and nature create flavors, not brains [8].”

Vision is encapsulated from flavor, an inference to the best explanation argument.

Here I present a three-part argument for the claim that vision is encapsulated from flavor at the level of experience, i.e that flavor does not sensorily penetrate vision at the level of experience. There are two things that typically trigger sensory penetration which cross-modal perception researchers have uncovered. One is known as the spatial-temporal rule, and the other is cross-modal correspondence. The first part of my argument seeks to show that the spatial-temporal rule is never satisfied between flavor and vision due to their anatomical relationship. The second part is an introspective argument to support the conclusion that while cross-modal correspondence may allow for flavor to sensorily penetrate vision at the level of processing, it does not obtain at the level of experience. The third part uses the conclusions from the first two arguments as the basis of an inference to the best explanation claim that vision is encapsulated from flavor.

Cross-modal perception researchers tend to agree that the reason our sense modalities exchange information with one another is to resolve conflict about a common subject matter. This is known as the principle of intersensory discrepancy [9]. Consider again the ventriloquist effect. When one sees the puppet’s mouth moving and hears the ventriloquist’s voice, audition and vision are receiving discrepant information about the source of the speech [10]. Audition and vision recognize the disagreement, and then resolve the conflict. One auditorily experiences the source of the speech as coming from the puppet’s mouth.

Now, when do two senses recognize a common subject matter? One of the most important triggers is spatial-temporal overlap [11]. The puppet’s mouth movements and the ventriloquist’s voice are spatially and temporally close in proximity. Indeed, the ventriloquist is intentionally matching his puppet’s mouth movements with his speech sounds. When this coincidence in space and time obtains between stimuli in two different senses, sensory penetration occurs. This is known as the spatial-temporal rule (just the “spatial rule” from now on).

So, it is a necessary condition that the spatial rule to obtain in order for sensory penetration, of the kind in the ventriloquist effect, to obtain. Note that I said of the kind in the ventriloquist effect, as we will see later that there is another kind of sensory penetration which occurs.

The question is, then, do flavor and vision ever satisfy the spatial rule? If not, this kind of sensory penetration can’t occur between them. I think not, because in order for the spatial rule to be satisfied, two sense modalities must have the same object in both of their receptive fields at the same time. The two modalities must have overlapping receptive fields, within which some common object rests. This simply never is the case for vision and flavor.

Here, “receptive field” refers to the physical field of space that a sense modality has access to at a given time when one’s body remains fixed. This means that receptive fields between modalities overlap in different ways. First, some receptive fields subsume others. The receptive field of audition subsumes, for the most part, the receptive fields of all of the other modalities. I can hear things that I see, smell, touch, and taste, but I cannot taste, smell, touch, or see everything that I hear. Consider when you hear a seagull far off in the distance behind you, and you keep your body fixed.

Second, some receptive fields partially overlap. I can see some of the things that I smell or touch, but not all of them, as is the case with potent objects touching my back. Third, receptive fields have potential overlap. When I shout out to you from behind, you can turn your head and bring your visual receptive field to overlap with your auditory receptive field.

In order for the spatial rule to obtain, an object must be within the overlapping receptive fields of the two modalities. This simply never happens with flavor and vision, for they stand in a very unique anatomical relationship with one another. The receptive field of vision is always outside of the mouth, a place that flavor simply doesn’t have access to, not even potentially.

It is worth noting that vision is the only sense modality that stands in this relationship with flavor. Touch and audition both have partial overlapping receptive fields with flavor, as you can feel the texture of the food as you hear your teeth chomping on it. But for the visual system, flavor takes place right under our very noses, and for the flavor system, the visual system’s receptive field is always just out of reach. As such, vision and flavor never satisfy the spatial rule and so they can never engage in the kind of sensory penetration that takes place in the ventriloquist effect.

There is another trigger that leads to sensory penetration, which is known as cross-modal correspondence. There are different accounts of cross-modal correspondence, but we can understand it, for now, as an associative relation between two modalities. Consider how vision can influence flavor. Seeing the color of the food one is about to ingest can influence the flavor that one experiences. This takes place despite the fact that the spatial rule isn’t satisfied, as we concluded earlier.

So, sensory penetration can occur in some cases without the spatial rule being satisfied. Between flavor and vision, there are a few associations, but we will focus on the pervasive association between color and flavor. One then might argue the following. Look, an associative relation is symmetrical. If x is associated with y, then y is associated with x. So if colors are associated with flavors, and this association is sufficient to cause sensory penetration of flavors by vision, then the principle of symmetry says that flavors are associated with colors, and this association ought to be sufficient to allow for flavor to penetrate vision, in some cases.

I have yet to see any research that addresses how flavor can influence vision (something I will discuss briefly later), and so introspective, bolstered by ordinary language, arguments must suffice here. Remember the distinction between sensory penetration at the level of processing, and sensory penetration at the level of experience. Introspectively it doesn’t seem that flavors influence the colors that we see. It could be that the association between colors and flavors does allow for sensory penetration of the visual system at the level of processing, but introspectively this doesn’t register in one’s visual experiences, which is what I am arguing for.

Ordinary language seems to supplement the introspective argument well. In conversation many of us have noted that the rich colors of foods influence our flavor experience, but we don’t talk about the converse holding as well. We don’t talk about how eating something while looking at a painting influences the way it looks [12]

But still a keen introspector could raise the following objection: but wait, sometimes I see a food and it looks delicious, but to my surprise, it actually tastes terrible. Subsequently, I see that food as being gross, or repulsive. Doesn’t this count as a case of flavor sensorily penetrating vision?

Here it is important to draw a distinction between low-level and high-level perception. Let’s say you and I are looking at a blue circular object that is a blueberry. You are fortunate enough to have the concept of a blueberry, while I am not. When you and I look at the blue circular object, you see it as a blueberry, while I simply see a blue-circular object. The properties that I experience, color, shape, etc. are known as low-level properties. The property that you experience, being a blueberry, is a high-level property. You have deployed the concept of a blueberry within your experience [13].

In the case of eating something which looks good, experiencing its terrible flavor, and then seeing the food as repulsive, flavor is influencing the high-level properties that one visually experiences. When most researchers, and I, talk about encapsulation and penetration, they are referring to encapsulation at a low-level. This is because it is obvious that all sorts of things can influence high-level perception, cognition included. An oncologist who has learned a lot in pathology will see a scan and see something as a tumor, while you and I will only see certain low-level properties in the image. So the important question is, can flavor influence vision at a low-level? The aforementioned objection says nothing to support the conclusion that flavor can influence vision at a low-level.

Since the spatial-temporal rule can never be satisfied between flavor and vision, and associations aren’t sufficient to allow flavor to sensorily penetrate vision at the level of experience, the inference to the best explanation is that vision is encapsulated from flavor at the level of experience.

Concluding remarks

For a while perception researchers seemed to focus exclusively on understanding how the visual system works. This could have been because the visual system occupies an enormous part of the brain, because researchers assumed that the senses were encapsulated relative to one another and therefore could be studied in isolation, or perhaps because researchers assumed that the principles governing how the visual system works would extend seamlessly to the other modalities [14].

Unfortunately, all three reasons turned out to be problematic. That the visual system owns a large neural real estate doesn’t justify studying it exclusively, it turns out that our senses (for the most part) are not encapsulated relative to one another, and non-visual sense modalities abide by different principles when creating our experiences.

Cross-modal perception research has been blossoming ever since these things were discovered. However, I fear that another bias toward studying certain systems over others is happening again. Most research within the cross-modal perception field seems to focus on the relationship between the spatial senses (vision, audition, and touch). Fewer researchers investigate the relationship between the chemical senses (olfaction and flavor), and even fewer research the relationship between the chemical senses and the spatial senses.

Though a few people are considering the ways in which vision can influence flavor, I have yet to see a single group discussing how flavor influences vision. I suspect there is, actually, a good reason for this. Most of us have always recognized the conclusion that I argued for here, that vision is encapsulated from flavor. At the very least, we all recognize that this is probably true.

Daniel Tippens is co-founder of The Electric Agora and a research technician at New York University School of Medicine.

---

Endnotes

[1]Technically speaking, attention doesn’t select the inputs to the eyes, it selects representations that have have been created from those inputs. But for simplicity’s sake I will just talk as though attention selects inputs.

[2] C. Spence, B. Smith, M. Auvray, Confusing Tastes with flavours, Published in
Perception and Its Modalities, edited by Dustin Stokes, Mohan Matthen andStephen Briggs, Oxford: Oxford University Press 2014.

[3] M. Auvray, C. Spence, The multisensory perception of flavour, Consciousness and Cognition, 17 (2008), pp. 1016–1031.

[4] Aradhna Krishna and Maureen Morrin, “Does Touch Affect Taste? The Perceptual Transfer of Product Container Haptic Cues.” Journal of Consumer Research: April 2008.

[5] K. Knöferle, C. Spence, Crossmodal correspondences between sounds and tastes
Psychonomic Bulletin, DOI 10.3758/s13423-012-0321-z.

[6] Spence, Charles, On the psychological impact of food colour, flavour 2015, 4:21 doi:10.1186/s13411-015-0031-3.

[7] Small, D. M., Gerber, J. C., Mak, Y. E., and Hummel, T. (2005). Differential neural responses evoked by orthonasal versus retronasal odorant perception in humans. Neuron, 47:593–605.

[8] Smith, Barry, On the Nature of Sensory Experience: The Case of Taste and Tasting, Phenomenology and Mind Online Journal 2013, pp. 292-313 ISSN 2280-7853 (print), ISSN 2239-4028

[9] By Welch, Robert B.; Warren, David H. Immediate perceptual response to intersensory discrepency, Psychological Bulletin, Vol 88(3), Nov 1980, 638-667.

[10] O’Callaghan, Casey, Perception and Multimodality, Oxford Handbook to Philosophy and Cognitive Science, Eric Margolis, Richard Samuels, and Stephen Stich, Editors Oxford University Press, forthcoming.

[11] But for further argument against the importance of the spatial-temporal rule, see Spence, C. Just how important is spatial coincidence to multisensory integration? Evaluating the spatial rule, Ann N Y Acad Sci. 2013 Aug;1296:31-49. doi: 10.1111/nyas.12121. Epub 2013 May 24.

[12] It is certainly plausible that if one is eating a delicious food while looking at a painting could alter one’s visual experience in an indirect way. For example, if the food is delicious, it could cause a change in one’s mood, which could cognitively penetrate one’s visual experience. But that would only be a case of cognitive penetration, and not sensory penetration.

[13] It is still controversial whether or not experience contains high-level properties. For a discussion on this issue, see: Bayne, Tim, Perception and the Reach of Phenomenal Content, Forthcoming in The Philosophical Quarterly. To be reprinted in Macpherson, F. and Hawley, K. (forthcoming) (eds.) The Admissible Contents of Experience, Oxford: Blackwell.

[14] O’Callaghan, Casey, Perception and Multimodality, Oxford Handbook to Philosophy and Cognitive Science, Eric Margolis, Richard Samuels, and Stephen Stich, Editors Oxford University Press, forthcoming.