Are we wrong to believe we have separate senses?


A bird in flight must control its movement with reference to multiple simultaneous sources of information: air mass, the ground plane, changes in stimulation caused by its own movements (image)

Eyes are for seeing and noses are for smelling. Everyone knows this. You know this. But hold on, what’s this? Ecological psychologists think you’re wrong!

The problem goes like this. Animals have to move about in an environment that already has other stuff in it besides the animal itself. All kinds of stuff. Imagine you’re a fish. You want to get over there to swallow that clod of plankton. If this was a video game you’d just press buttons until the plankton clod got big and filled up the screen. And when you can’t make the plankton clod look any bigger, that’s when you know you’ve arrived. Good old-fashioned visually-guided locomotion. But things aren’t that simple. You’re a fish, and there’s water in the way. Now you also have to move in such a way as to cancel out the movement of the water. Also, since you’re a fish, that is, a living, active organism, the visual scene before your eyes is in part structured by your own movements. Now you have to work out which bits of the stuff you’re looking at are moving because stuff out there is moving, and which bits are moving because you’re a fish and you’re twitching your face from side to side and your paddling is creating weird ripples and shadows.

One way to respond to this is to shrug and say, “I dunno, being a fish is complicated, I guess. At least they don’t have to worry about table manners or heights.” This is the normal way to respond. If you’re an ecological psychologist, however, you might respond differently. You might come out with something like this: “We propose that the senses form a single, unitary, irreducible perceptual system whose sole function is to detect parameters of the global array.”

In making this proposal, Stoffregen, Mantel, and Bardy are doing us a service for they are forcing us to question an unexamined assumption. In this case, the assumption is that the senses constitute distinct and separate systems. As the authors note, James J. Gibson also liked to question unexamined assumptions. But Gibson himself accepted this particular assumption without examining it. According to SMB, this has led to problems. One such problem: research within the ecological tradition has asked questions of the form, “What visual parameter is implicated in controlling the movement of this seabird into this body of water?” But this reductively detaches the visual component of the behaviour from everything else that is going on in the bird–air–water–gravity–etc. system.

A more esoteric problem is this: specification of structure in the environment into structure in perception does not work if we consider the senses as distinct systems. If all the seabird has to go on is vision, then it has no way of distinguishing between changes in the appearance of the water surface that are caused by its movements towards the surface, and those that are caused by the movement of the air relative to the same surface. Without being able to distinguish between these, SMB claim, the bird cannot adjust its behaviour over time in the appropriate way. Specification only works if we consider all of the relevant structure at once. And we need specification. Ambiguity is no good. If specification fails, then direct perception is also impossible, and ecological psychology itself disintegrates into an incoherent puddle.

Now, it is tempting to respond to SMB’s proposal that the senses constitute a single, unitary system in this manner: “Nuh-uhh! The senses are obviously separate! Look! Eyeballs!” But SMB have been making this argument for a few years now and have apparently come across this response before (S&B originally published this argument in BBS in 2001; the 2017 paper is largely a reassertion of that one). The authors call for their critics to go beyond merely pointing out apparent instances of counter-evidence and to engage with their arguments “at the level of general principles”. So let’s try to do that. I will make three points.

First, while every assumption deserves to come under scrutiny once in a while, it’s impossible to evaluate any given assumption except in the light of remaining assumptions that we hold to be more fundamental or secure. Thus, SMB state:

If we accept that perception yields knowledge (i.e., that perception is epistemological), then we can ask what type of knowledge is obtained. Following Darwinian principles … Gibson reasoned that perception yields (and should yield) knowledge about the environment.

But wait, what? Why should we accept this assumption? I mean, not to be rude, but “perception yields knowledge”… that sounds a little, I dunno, Cartesian? SMB also talk about perception as having content: “Our focus is on the animal–environment interaction as the contents of perception”.

Maybe the real problematic assumption here is this one: that information is contentful, i.e. that it is about structure in the environment. This argument was made by Van Dijk, Withagen, & Bongers in a paper that I blogged about here.  If information is contentless, then the issue of ambiguity does not arise. Ambiguity is not in the information because information is not being transmitted. The animal is not in the business of constructing knowledge about what the world has in it. Rather, the animal is in the business of regulating the fit between itself and its environment. If this is the case then perception does not have to be infallible, it just has to be good enough to keep the animal alive for a certain while. The fit is either maintained, in which case the animal is still alive, or it isn’t, and it’s dead. But why say that this fit is the content of perception?

Second, in their enthusiasm to do violence to the Aristotelian dogma of separate senses, SMB sometimes risk doing damage to other theoretical distinctions which we presumably want to maintain. For instance, it seems useful to distinguish between a) action and b) structure in the environment with reference to which that action is being regulated. In their more effusive moments SMB seem to want to fold this all together:

The global array is not ambient to massless, geometric points; rather, it intrinsically, inescapably is ambient to and provides information about the full dynamics of the animal–environment interaction. The global array pulses with the life of the organism; it embodies animal–environment reciprocity.

SMB argue that Gibson’s notion of the ambient optic array as a set of points of potential observation is “disembodied” and “reductionist”, and we should instead think in terms of an “embodied point of observation”. We should recognize that visual information in real life is structured simultaneously by stuff in the environment and by the activities of the living, moving animal itself. When you walk, your eyeballs jiggle in your head, which is going up and down on top of your neck thump-te-thump. This is fine, but don’t we also still want to be able to talk about structure that’s actually out there in a straightforward way, without constantly having to talk about it in terms of the animal–environment fit?

Third, and more generally: suppose we all went along with SMB and just admitted that the notion of separate senses is mere superstitious hokum and baloney that has been holding us back from pursuing a properly scientific study of perception. Where would that actually get us? In what way is it useful to treat perception as a single system? It certainly seems useful to treat vision, say, as a distinct system. To do so is consistent with our everyday conception of our experience—our way of talking about it. Don’t we want to be able to explain, say, the navigability of the inside of a building in terms of what it looks like without having to also talk about what the air feels like on our faces or what smells are wafting in from a nearby cafeteria? How does it help us to flat out deny the distinctions we have traditionally made between the senses?

It’s hard to see the global array concept getting any real uptake based only on arguments that have been presented so far. What would make the case more convincing? SMB need to show either that the assumption of distinct senses is pernicious and has misled us in important ways, or that rejecting the assumption leads to useful outcomes or applications that would otherwise be inaccessible to us. They have done neither of these so far. Maybe they’re right, though. I’m going to ask a fish.


Stoffregen, T. A., Mantel, B., & Bardy, B. G. (2017) The senses considered as one perceptual system. Ecological Psychology, 29(3):165–197 doi: 10.1080/10407413.2017.1331116



Overcoming loser philosophy: Comments on Käufer and Chemero’s Phenomenology: An Introduction

If you are studying affordances, or studying the role of the body in perception and cognition, then you ‘are not merely influenced by phenomenology,’ you are ‘doing phenomenology, insofar as [you] are pursuing the basic ideas and insights this tradition was founded on.’ So claim Stephan Käufer and Tony Chemero (K&C) at the start of their new bookPhenomenology: An Introduction (2015).

Now, one would be forgiven for thinking it’s safe to ignore this book, on the grounds that a) it’s got the word ‘Phenomenology’ in the title, and b) it’s got the words ‘An Introduction’ in the title. But don’t let’s get hung up on those two details. K&C are pursuing a serious project here. They aim to show that the phenomenological tradition is alive, and that it provides the necessary foundation for embodied cognition research. The message of the book might be summarized thus: Nothing in embodied cognition makes sense except in the light of lived experience. It’s a good message, and I hope people take notice.

I am not going to summarize, in this post, all of the book’s discussion of the history of phenomenology. The book does a nice job of covering the major thinkers while also weaving in the story of how psychology emerges as a separate discipline seeking to differentiate itself from philosophy, without ever quite succeeding. There are chapters on Kant and Wundt, then Husserl, Heidegger, the Gestalt psychologists, Merleau-Ponty, Sartre and Beauvoir, then James Gibson, Hubert Dreyfus, and a final chapter giving a brief overview of the current state of play in embodied and enactive cognitive science. I will quickly go over a couple of the important ideas from the Merleau-Ponty chapter, then I’ll get into K&C’s discussion of what, in their view, are the most interesting current issues in the field.

Merleau-Ponty and the lived body

A hand holding the handle of a coffee cup

Can you really be sure that that coffee cup has a backside? (Source)

Husserl spends a lot of time fretting about this kind of question: How can it be that, when we look at a tree, we seem to experience the tree as having a backside, even though, right now, we can only see the front? Husserl’s solution is to posit that the tree that we experience is a mental entity, an intentional object: we see the tree as whole because we’re not directly experiencing the tree itself, we’re experiencing a copy of the tree that’s internal to our consciousness. Husserl sees this as a methodological move that allows him to ignore the question of whether the tree really does have a backside or not, or, for that matter, whether there’s any tree there at all.

Merleau-Ponty provides a different way out of the problem. He distinguishes between the lived body, which he also calls the habit body, and the objective body, or the body as an object. He uses this distinction to explain what’s going on in phantom limb cases where people who have lost a limb report still feeling pain or sensation at the location where the limb used to be. The way we normally experience the world, Merleau-Ponty says, is as a lived body that exhibits skills and habits. In phantom limb cases, the skills and habits have formed in the presence of the limb, and these skills and habits remain intact even though the objective limb itself is no longer present. The loss of the limb means that the habit body is no longer in harmony with the objective body. Now when the individual encounters a situation which used to involve the limb, the situation may still be perceived as one in which the old action is possible, but the objective body no longer supports this action.

To return to Husserl’s problem, Merleau-Ponty points out that when we encounter an object from a particular aspect, we do not do so as disembodied observers, but precisely as lived bodies. This allows us to say that when we encounter a tree, or a coffee cup, we experience these objects as having a backside to the extent that our bodies are disposed to act on the whole object. We see a tree that we can walk around, or a cup that we can wrap our fingers around. The backside is not in our perception, but in the complete act of engaging with the world.

Incidentally, this also provides the solution to a problem Chemero raised in his previous book (though K&C do not acknowledge this): how do we perceive that a beer can contains drinkable beer, if we are not representing the contents? Chemero (2009, 118) uses this example to motivate the claim that perception must be grounded in constraints and conventions. It is held that the label on the beer can is conventionally related to the presence of beer. We, as perceivers, have access both to the convention and to the label, and together these indicate the presence of beer inside the can. But Merleau-Ponty’s account of the lived body renders unnecessary any such appeal to conventions. We don’t need to perceive the beer itself, any more than we need to perceive the back of the coffee cup. Our bodily skills mean that we can just act in the presence of the beer can. And sure, it might turn out, after all, that the beer can contains sour milk, or that the coffee cup doesn’t actually have a backside. But such considerations do not normally enter into our activities. We do not live in a world where we have had to interact with milk-containing beer cans or backside-less coffee cups, so there’s no reason why our habit body would come to be shaped by such eventualities. In general, we get along just fine.

Now onto the current problems in the field. First: affordances.

Affordances, invitations, and the frame problem in ecological psychology

Affordances, it is generally said, are opportunities to act on the environment. But that raises a question: how is it that only some affordances come to be selected for action? It would seem to be the case that a detached observer is required to do the selecting—some mental controller that sits inside the organism and is able to assess the available options. This is a problem that Gibson was aware of, as shown in this passage that K&C quote twice (Gibson, 1979, 225): ‘The rules that govern behavior are not like laws enforced by an authority or decisions made by a commander; behavior is regular without being regulated. The question is how this can be.’ (Another way to put this is to say that the ecological approach provides a theory of the structure available to perception, but not a theory of behaviour as such; see this post for discussion.)

The way K&C present this is as a version of the frame problem—a problem in AI research that gave rise to much philosophical bellyaching in the 1980s, most notably from Dan Dennett. The problem is this: how can a robot ever be expected to choose its next action, given the infinite number of facts that the robot could, in principle, take into account before making its selection? As K&C see it, the only way out of this problem is to appeal to dynamical systems. An actor is a dynamical system. And the actor is nested inside a larger dynamical system—its environment. If the actor is a dynamical system then it does not need to mentally entertain any of the possible actions available to it; rather, it just acts, and it learns by changing over time. Engaging with the world not only changes the world, it also changes the internal dynamics of the actor. The outcomes of actions can have the effect of reconfiguring the attractor states within the actor’s nervous system. Here K&C appeal to Walter Freeman’s work on the neural dynamics of the rabbit olfactory bulb. If the rabbit experiences a particular scent and subsequently gets food, then this will have the effect of subtly re-configuring the rabbit’s nervous system, which will affect how the rabbit responds to the same scent the next time it encounters it (207). The frame problem dissolves because now everything is dynamic. (But I’ll have more to say about the frame problem below.)

In order to fix how we talk about affordances, K&C recommend that we follow Withagen et al (2012) in making a distinction between mere affordances and invitations. Affordances are the possibilities to act, which are always infinite in number; invitations (a term borrowed from Merleau-Ponty) are the subset of affordances that stand out at a given moment as live options. Withagen et al make this distinction in the context of a discussion of agency. K&C use it to appeal again to dynamical systems. K&C discuss the example of someone engaged in the activity of building a bookcase. The bookcase builder does not attend to an infinite world of possibilities, but only to aspects of her environment that are relevant for her ongoing project. The dynamics of this organism–environment system (builder plus bookcase-parts plus tools and so forth) have become self-organized into ‘a temporary, special-purpose dynamical system’ (203), one in which only certain affordances actually invite behaviour.

The future of phenomenological cognitive science

This is all well and good as a description of what it’s like to be a living being engaged in activities in the world. But where does it get us? How can we turn the insights of phenomenologists into a productive programme of empirical research?

How K&C propose to do this rests, in large part, on a phrase that they borrow from Shaun Gallagher and Dan Zahavi (2008), who say that, in their empirical studies of behaviour, researchers should front-load phenomenology. Unfortunately, it is not very clear exactly what work this phrase is supposed to be doing. K&C define front-loading phenomenology as ‘designing scientific experiments specifically in order to test posits concerning phenomenology’ (218). This suggests that phenomenology itself should be the subject matter of the research. But this seems at odds with what Gallagher and Zahavi themselves say (2008, 38; emphasis added): ‘The idea is to front-load phenomenological insights into the design of experiments, that is, to allow the insights developed in phenomenological analyses to inform the way experiments are set up.’ Here, it seems that phenomenology is not the subject matter of the research per se, but is part of the experimental methodology: thinking in terms of phenomenology is just part of the experimenter’s toolkit for setting up useful experiments. In their own recommendations for conducting future research, K&C seem to vacillate between two different types of strategy, neither of which seems to exactly match what Gallagher and Zahavi are after.

Strategy 1: Directly investigate stuff that phenomenologists have actually said. An example of research using this strategy is Chemero’s work with Dobri Dotov (Dotov et al, 2010), on how equipment can go from being ready-to-hand to being unready-to-hand. In the experiment, subjects are asked to play a computer game in which they have to move a mouse cursor about on a screen. At some point in the game, the experimenter interrupts the link between the mouse movement and the cursor movement. This is like what happens when you get a bit of water on your laptop trackpad, or in the old days when you got a bit of fluff taffled up in the mouse’s rollerball. As we know from having experienced this kind of thing, the disruption is apt to put us off what we had just been doing and may well cause us to flail about in an attempt to regain control of the situation. Dotov et al present this in explicitly Heideggerian terms, as a disruption that shatters our smooth coping with equipment in the environment. And they have a way of quantifying this: the disruption goes along with a disruption of the subject’s movement dynamics, which move from exhibiting pink noise (or 1/f noise, which is assumed to be characteristic of smooth coping) towards exhibiting white noise (characteristic of random movement, or flailing about). This is neat as a ‘demonstration’ of an idea from Heidegger’s phenomenology. But it is not clear how this is supposed to serve as a model for doing cognitive science in general. Are we to restrict our empirical efforts to attempting to prove Heidegger correct?

Strategy 2: Reduce everything to maths, but do so in a way that you hope is consistent with phenomenological theorizing. The second strategy is to subsume everything under the logic of dynamical systems modelling. This is defended on the grounds that complex systems are inclined to ‘self-organize and so have a tendency to behave like much simpler systems,’ and therefore ‘one will often be able to model these systems in terms of extremely simple functions, with only a few easily observable parameters’ (200). And the logic of these systems means that, in principle, it’s possible to study any phenomenon, from the level of the cell to the level of whole societies, all in terms of relatively simple dynamical equations. One might object that by reducing everything to maths, you have taken yourself outside of the realm of phenomenology altogether. Equations don’t have feelings. And there’s nothing that it’s like to be an equals sign. More to the point, there’s nothing that it’s like to be a cell or a basketball team either. Experience only operates at the level of individual organisms. By collapsing everything into mathematics, have we not lost sight of the very thing—lived experience—that caused us to so admire the phenomenologists in the first place? Surely we’re owed at least a hint of an explanation as to why it’s desirable to model experience-involving processes in exactly the same way as non-experience-involving ones. But according to K&C, there is no problem here: it’s perfectly possible to do dynamical systems modelling and still be in the business of successfully front-loading your phenomenology (201). What’s important is that the way you choose what phenomenon you are going to model in the first place should be informed by a suitably phenomenological worldview.

I have to say, I think K&C give a fair account of the current state of play within phenomenologically-informed cognitive science (that is, in enactivst and dynamical systems research). Still, neither of these two strategies strikes me as being particularly attractive.

The second strategy, in particular, seems to be too much in awe of the frame problem. I suspect that part of the motivation for trying to reduce everything to dynamical systems explanations is the worry that unless everything is expressed in the same language (mathematics) then we will never exorcise the creeping spectre of dualism and we’ll be forever stuck with some version of the frame problem. But there’s a reason why the frame problem originated where it did—in the field of artificial intelligence research. The old project in AI had the goal of building a robot with the capacity to act autonomously in a natural environment. In order to achieve that, AI researchers were going to have to be able to build a robot that could select its actions from a seemingly infinite set of possibilities. But why should we non-AI researchers take that as a model for our own psychological programme? I just don’t buy that building a working model of a behaving, living system is a realistic goal for psychology. Or a useful one. I’m much more interested in using psychology as a set of tools for intervening in real-world problems: how can we design cities that are better adapted to how we want to live, or educational resources that are better suited to how we learn, or clinical environments that better draw out the capacities of people we currently deem to be mentally ill? As such, I just don’t care about the frame problem. And neither should you!

What I’d like to see developed more is a strategy more closely aligned to that definition from Gallagher and Zahavi: to front-load phenomenology is to use phenomenology as a tool for designing experiments that can actually inform us about how we engage in specific activities—a tool that will allow us to pursue genuinely useful research. This is what Andrew was getting at in this post on dynamical mechanistic explanations. And I’ve argued (Baggs, 2014) that the appropriate way to conduct research is to take a task-oriented approach; that is, we can identify an activity that is of interest to us, from the perspective of outside analysts, but to do so we must identify a dynamic that is meaningful from the perspective of the actor. Taking this kind of approach allows researchers to study phenomena—such as steering a car round a corner, or learning to walk—that are phenomenological through and through. (See Wilson and Golonka, 2013, which sets out the general strategy for actually employing the task-oriented methodology.) The task-oriented approach seems much more worthy of the label ‘phenomenological’ than does the reduce-everything-to-maths approach.

But this is an argument for the future. For now, I’ll say that I very much enjoyed reading K&C’s book. What the book does well is it places current research in embodied and enactive cognitive science in a long-ish historical context. In doing so, the authors reveal that, far from being ‘radical’, in the sense of being outlandish and hopelessly outside the mainstream, these approaches are grounded in a solid phenomenological tradition. We inheret from Heidegger and from Merleau-Ponty a view of action as skilled engagement with the world, by a lived body. Indeed, taking this long view, it is the cognitivist AI programme that turns out to be the historical aberration. Hubert Dreyfus drily explains this in an excellent interview from 2005 (K&C also quote a different passage from this interview):

The people in the AI lab, with their “mental representations,” had taken over Descartes and Hume and Kant, who said concepts were rules, and so forth. And far from teaching us how we should think about the mind, AI researchers had taken over what we had just recently learned in philosophy was the wrong way to do it. The irony is that the year that AI (artificial intelligence) was named by John McCarthy was the very year that Wittgenstein’s Philosophical Investigations came out against mental representations. [Dreyfus may be a little muddled on the dates here—EB] (Heidegger had already done so in 1927 with Being and Time.) So, the AI researchers had inherited a lemon. They had taken over this loser philosophy! If they had known philosophy, they could’ve predicted, like me, that it was a hopeless research program, but they took Cartesian philosophy and turned it into a research program. Anybody who knew enough philosophy could’ve predicted it was going to fail. But nobody else paid any attention. That’s why I got this prize.

K&C have shown us the direction to take if we are to avoid falling back into loser philosophy. They have done us a service.

Further reading/listening

  • There’s a great Brain Science Podcast interview with Tony Chemero talking about the book, to be found here.


Dr. Paperclip defeats the wicked Gibsonians: a satire

‘It looks like you’re writing a letter,’ interjected Dr. Paperclip. ‘Can I help?’

‘Oh!’ responded the young Spandrel, startled. ‘I didn’t know anyone was here. How long have you been sitting on my desk?’

‘Don’t worry, I’m not really here. Now, about that letter…’

‘Er…’ Spandrel began, ‘…well, it’s nothing really. And it’s not a letter, more of a… tweet.’

‘Yes, yes, very good,’ replied the doctor, ‘how technology has come on.’ With this, the doctor rested one eyeball wistfully for a moment on one of his wire extremities, then began again, ‘I couldn’t help noticing that you were writing about… What are they calling them, now? Embodied cognitions?’

‘Um,’ replied Spandrel, ‘I’m not sure about that. I was just posting a link to this video of a robot dog. Have you seen it? Watch! There’s a guy comes along in a minute and kicks the thing and it doesn’t even fall down!’

‘Yes, yes, I’ve seen it,’ the doctor waved. ‘And I wasn’t in the least impressed. You see, you come to get a feel for this sort of thing after a while. Back when I was in graduate school there were all these old geezers around who liked to spend their days stuffing pigeons into boxes. “The pigeons can teach us how language works,” they said. Well, you don’t see too many of those guys around anymore.’

‘What a strange thing to do,’ Spandrel remarked.

‘Of course!’ declared the doctor. ‘I see you’re a fast learner! The point is, those guys died out. It’s evolution, my dear boy!’

‘Yes, I suppose that makes sense,’ replied Spandrel.

‘And now there are these new people,’ the doctor continued. ‘They call themselves the radically embodied cognitive science gang, if I’m not mistaken. I feel sorry for them, really.’

‘Why’s that?’

‘Ah!’ said the doctor, ‘Funny you should ask! You see, these new people like to make a big noise about these silly little things like your dog video here. “The dog doesn’t have a mind,” they say. And then do you know what they say next?’


‘They say that because the dog doesn’t have a mind, this means that you and I don’t have a mind either!’

‘What a strange the thing to say!’ Spandrel ejected.

‘Isn’t it!’ said the doctor. ‘Of course, they don’t admit to saying this, if you ask them. They’ll tell you some story about how we don’t need a “mind”—’ the doctor used both of his wire ends to gesture the scare quotes, ‘—we don’t need a mind because we “see the world directly”.’

‘What does that mean?’ Spandrel asked.

‘Well exactly, it’s absurd isn’t it!’ the doctor agreed. ‘But I can prove that they are dead wrong! And you know how?’


‘Physics!’ the doctor exclaimed. ‘You see, I’ve been hanging out at the physics department lately (I was pretending to hold some loose sheaves of paper together) and I’ve picked up the most amazing gossip.’

‘What’s that?’ gasped Spandrel.

‘Well… It turns out… The world you think you see… It’s a complete fiction. An illusion. There’s nothing there at all! The world is just one big quantum soup!’

‘I beg you pardon? Then how do I see it.’

‘Ah! Well here’s the clever bit. Your mind has been selected to see what it needs to see. Your mind, you see, is a particularly well-adapted quantum crouton. You never see the soup that you’re floating in because, as I told you, it’s not there. You only see the inside of your crouton, and that’s enough to keep you floating happily along.’

‘Wow, that’s mind-blowing,’ admitted Spandrel. ‘But I have one question. If I’m a crouton. And you’re a crouton. How do we know there’s any soup out there at all?’

‘We don’t!’ the doctor announced. ‘But you’d better hope I’m right. Because if I’m wrong, it would mean that everything we think we know about quantum physics is mistaken. And it would also mean that the world does not contain any croutons. Physics says!’

‘Wow, that’s fabulous. Thanks, Dr….’

‘Dr. Paperclip!’ said Dr. Paperclip. ‘Glad I could be of help!’

With this, Dr. Paperclip briefly folded himslef into a working hydrogen atom—one eyeball serving as the nucelus and the other eyeball flying around a wire orbit like a googly electron—before quickly shrinking into nothingness and disappearing into an inkpot.

Context: interface theory; WilsonHickok; Clippy

Minds do not extend, but activities do

I was intrigued to see that Jeff Wagman and Tony Chemero have a chapter out declaring “the end of the debate over extended cognition” (2014). “Oh, good,” I thought, “finally some sensible souls have stepped in to put an end to this nonsense.” But, alas, this chapter will not end the debate. It is merely the latest in a line of arguments declaring victory for one side.

The trouble is, the extended cognition debate is not one that can be ended by one side proving the other side wrong. The disagreement is not over empirical facts, but over a priori assumptions about what words like “mind” mean, or ought to mean. The problems here can be traced to the very first sentence in Clark and Chalmers’s (1998) paper introducing the concept of the extended mind, where they pose the question: “Where does the mind stop and the rest of the world begin?” This question commits a category mistake (one that Gilbert Ryle would have diagnosed): it assumes that minds are a kind of thing in the world that can be detected, when in fact “minds” are nothing more than labels we use to describe one another’s behaviour. The question is incoherent. To end the debate, we need to give up on this misguided search for the physical substrate of “minds”, and, instead, start to understand the world in terms of activities. I will here set out the beginnings of a case for this.

Otto’s notebook

The standard argument for the extended mind thesis is Clark and Chalmers’s thought experiment, the “Otto’s notebook” argument. Otto, who suffers from Alzheimer’s, has to write down the address for the Museum of Modern Art in his notebook and consult this note every time he wants to go there. Meanwhile, Inga, who does not suffer from Alzheimer’s, does not require this external memory aid but recalls the address using her own “biological memory”. Since Otto’s notebook is, we are assured, playing the same function as Inga’s brain, we have no reason to reject the conclusion that Otto’s notebook is literally part of his mind.

There are some severe issues with this thought experiment. Here is one. We are told that Otto has Alzheimer’s, which means he has to write things down in his notebook. Yet apparently he has no problem remembering to carry the notebook around with him at all times, or remembering how to consult it, or remembering what piece of information he is looking for, remembering what the task was that originally caused him to consult his notebook. We are to assume, I suppose, that, having once consulted his notebook and seen that MoMA is on 53rd St, Otto will happily set out from his starting point (where? we are not told, although it is stated that Inga is within walking distance), not once forgetting his purpose along the way. Otto, in short, is a perfectly functional, healthy individual, bar an unfortunate inability to remember certain facts. The form of Alzheimer’s at play here is, to say the least, unusual. This is hardly a realistic description of any actual condition.

Of course, it’s easy to be pedantic about thought experiments—to take issue with their simplifying assumptions. In this case, though, the objection is more than merely pedantic. The objection is that this particular thought experiment leaves out certain details that matter. The whole argument relies on Inga and Otto being identical except in one crucial respect: one keeps her memories on the inside (in her brain), the other keeps his memories on the outside (in his notebook). What is left out is any account of the processes by which Otto and Inga supposedly achieve their identical goals. It is merely asserted that both have “access” to their respective store of memory, and the fact that “accessing” a notebook can be described using the same verb as “accessing” one’s own memory is supposed to ensure that these are, in fact, one and the same phenomenon, and that Otto’s notebook is, therefore, part of his mind. Colour me underwhelmed.[1] The point here is: once we start to understand the processes properly, it simply won’t matter which components of the process are part of the “mind” and which components aren’t. Such labels will not be doing any explanatory work.

Jimmie’s Mass

We can contrast the vignette about Otto’s notebook with an account by Oliver Sacks of an actual neurological patient, Jimmie G., whom Sacks dubs “the lost mariner” (chapter 2 in The Man Who Mistook His Wife for a Hat; the chapter was earlier published in the New York Review of Books). Jimmie G. does not have Alzheimer’s. Sacks diagnoses him as having Korsakov’s syndrome, a memory condition caused by alcohol abuse. Jimmie has lost the ability to form new memories, and believes he is still living in 1945 following the end of the Second World War, although Sacks first meets him in the mid-1970s. Jimmie is unable to remember what happened moments before. He is trapped in a perpetual present, and is apparently, for the most part, not even aware of his condition. The turning point in Sacks’s account comes when Sacks asks whether Jimmie has, in some profound sense, lost his soul:

One tended to speak of him, instinctively, as a spiritual casualty—a “lost soul”: was it possible that he had really been “de-souled” by a disease? “Do you think he has a soul?” I once asked the Sisters. They were outraged by my question, but could see why I asked it. “Watch Jimmie in chapel,” they said, “and judge for yourself.”

I did, and I was moved, profoundly moved and impressed, because I saw here an intensity and steadiness of attention and concentration that I had never seen before in him or conceived him capable of. I watched him pray, I watched him at Mass, I watched him kneel and take the Sacrament on his tongue, and could not doubt the fullness and totality of Communion, the perfect alignment of his spirit with the spirit of the Mass. Fully, intensely, quietly, in the quietude of absolute concentration and attention, he entered and partook of the Holy Communion. He was wholly held, absorbed, by a feeling. There was no forgetting, no Korsakov’s then, nor did it seem possible or imaginable that there should be; for he was no longer at the mercy of a faulty and fallible mechanism—that of meaningless sequences and memory traces—but was absorbed in an act, an act of his whole being, which carried feeling and meaning in an organic continuity and unity, a continuity and unity so seamless it could not permit any break.

Sacks hints at an explanation of why it is that Jimmie appears to find his bearing in the taking of the Communion: ‘The same depth of absorption and attention was to be seen in relation to music and art: he had no difficulty, I noticed, “following” music or simple dramas, for every moment in music and art refers to, contains, other moments.’

What Sacks seems to suggest here is that persisting, temporally extend structures in the environment can support and give form to behaviour. The Mass has a standard structure, a structure that was present earlier in Jimmie’s life. And it was in the context of this structure that Jimmie’s behaviour came to be shaped. Jimmie learnt how to conduct himself appropriately in the context of the Mass, and this Mass-specific behaviour remains intact. Even if Jimmie cannot remember how he got into the chapel five minutes ago, he can see that he’s in a Mass, and knows what to do at any moment. But as soon as he leaves the chapel and steps outside he will find himself in a context that does not contain such a formal structure. He will find himself in a novel situation for which his behavioural repertoire is no longer appropriately shaped by past learning. He will, once again, be lost.

It would make little sense to talk about what’s going on here in terms of the extended mind (although this is precisely the kind of situation, one would think, that an extended mind concept should be applicable to). Is the Mass part of Jimmie’s extended cognitive system? No, the Mass predates Jimmie’s existence, and has had the same formal structure throughout Jimmie’s life. It is this very unchanging nature of the Mass which means that Jimmie’s behaviour can remain intact relative to it. To talk of an extended mind is to imply that activities necessarily emanate from within an individual actor. The reality is that activities constitute a setting into which animals find themselves deposited: we encounter ongoing activities from birth or even before. And it is only through exploratory action and engagement that we learn to take part in these activities, to take control of them, and to alter their course. (For a partial discussion of this focused on human emotional developmental, see Greenwood, 2013.)

Extended activities

Wagman and Chemero do not talk about Masses or notebooks. The activities they discuss are those involved in a particular type of perception research: work on dynamic touch. An important motivating insight for this work comes from Merleau-Ponty: a blind man using a cane does not perceive the cane itself but perceives the world at the end of the cane. A dynamic touch study might go like this: a subject comes into the lab, is blindfolded and laden down with a backpack and is handed a stick; the subject is then asked to use the stick to poke about at a sloped surface and is asked to give a verbal judgement about whether they will be able to stand on that surface. All good fun, and it turns out that we’re quite good at doing this sort of task. Wagman and Chemero want to use this kind of result to claim that minds do indeed extend. The results, they say, “suggest that both the weighted backpack and the handheld wooden dowel are experienced as part of the body.” But, of course, this is of no consequence to cognitivist opponents of the extended mind. It claims only that sticks can be experienced as parts of bodies. But one of the things that is at issue is whether it is reasonable to talk of minds as extending into bodies in the first place. Wagman and Chemero simply assume that it is reasonable, and want their opponents to agree with their assumption.

What these kinds of results do show is that the act of perceiving is one that necessarily spans animal and environment. It is activities that are distributed, not minds. And activities are the kinds of things that we can investigate empirically. We can take away or disrupt some component of the overall system and observe whether or not this affects the outcome of the activity. Indeed, if a given disruption has an effect on the behaviour, this is a good reason to deduce that the thing that was disturbed was an integral part of the activity, and instrumental in the individual’s control of their behaviour. But it is simply not helpful to insist that whatever is integral to an activity is therefore part of an “extended mind” or “extended cognitive system”.

The extended mind metaphor does hit on an essential truth: classical cognitivism’s view of “mind” is inadequate because it simply assumes that all behaviour (or all interesting behaviour) is the output of symbol-manipulation processes. But the fact that cognitivism is too narrow is no reason to replace one problematic use of “mind” with another that only creates new problems. The extended mind metaphor has outlived its usefulness.


  1. Beware philosophers bearing thought experiments. Especially when those thought experiments rely on tenuous claims about real-life neurological conditions. Ask this: Does my philosopher have the appropriate clinical experience to be making pronouncements on these matters?
  2. If we really want to end the debate over extended cognition, we should simply stop arguing about whether “minds” or “cognition” might “extend” into objects. This is a language game that can have no winners because the participants refuse to agree on the rules (see also Sabrina’s old post on this). Instead, we should agree that behaviour—acting in the world, engaging socially with others, and all the rest—involves activities that extend across resources both internal to and external to organisms. The task for psychologists, then, is to identify the structures that are involved in putting these activities together.


[1] For further entertaining discussion of Otto’s notebook, I recommend Jerry Fodor’s (2009) review of Andy Clark’s book.


  • Clark, A. & Chalmers, D. (1998) The extended mindAnalysis 58(1):7–19.
  • Fodor, J. (2009) Where is my mind? London Review of Books 31(3):13–15.
  • Greenwood, J. (2013) Contingent transcranialism and deep functional cognitive integration: The case of human emotional ontogenesis. Philosophical Psychology 26(3):420–436. doi: 10.1080/09515089.2011.633752
  • Sacks, O. (1985) The Man Who Mistook His Wife for a Hat. London: Duckworth.
  • Wagman, J. B. & Chemero, A. (2014) The end of the debate over extended cognition, in Solymosi, T. & Shook, J. R. (Eds.) Neuroscience, Neurophilosophy and Pragmatism: Brains at Work with the World. New York: Palgrave Macmillan. doi: 10.1057/9781137376077.0012

Speaking is a type of acting, but being spoken to is not a type of perceiving

I have a paper in the newly published issue of the journal Ecological Psychology. The paper sets out the case that language-use should be understood as fundamentally a form of acting. I contrast this with a view of language-use as fundamentally a form of perceiving information via conventional patterns.

The argument may come across in the paper less clearly than I had hoped. The paper is part of a special issue on the topic of “Language as a public activity”, which is introduced by the editors, Bert Hodges and Carol Fowler (Hodges and Fowler, 2015). The editors valiantly attempt to provide one-paragraph summaries of all the papers in the issue. And they do an excellent job of summarizing mine (they’ve even highlighted a couple of things about it which I wasn’t aware of, for which I’m grateful). But there are a couple of statements in their summary which I would not endorse, which tells me that I wasn’t clear enough about those claims. So in this post I will try to try to do a couple of things: (a) summarize the argument I think I tried to make in this paper, and (b) set things out in a more intuitive way.

To understand speaking we need to understand how populated environments are structured

I tried to argue against the view that language-use necessarily involves conventional patterns. Conventions are to be understood here as mappings between invariant patterns (e.g., the word “dandelion”) and events (e.g., the perceiving of a dandelion). I don’t think we need to appeal to any such notion of conventions to explain how speaking works.

The key is to adopt a developmental perspective on language-use.

Children, when they are learning to speak, do so in an environment that contains other actors. And although none of these actors will be constantly present at every moment throughout the child’s growing up, there will nevertheless be a handful of actors who form part of the furniture of the child’s surroundings.

Now, the actors in a child’s environment themselves create structure through their own activities. For instance, sometimes an adult will point to some object and say something like, “Ooh, look at that giraffe!” One of the advantages about being a child (as opposed to being, say, a dictaphone) is that you don’t have to wait for other actors to produce these structures: if you want to explore the structure of your surroundings, you can perform an action that disturbs things and that thus creates potentially meaningful new patterns.

This is something that children do all the time. It happens even in situations where the child’s behaviour seems at first glance to be little more than a series of responses to an adult’s prompting, e.g. in ritualized games of the type where an adult repeatedly asks the child “Where’s your nose?”, “Where’s your mouth?”, etc. See this video of a child interacting with his grandmother.

When the child cannot immediately produce the appropriate response to the prompt “Where’s your mouth?” (a little after 1:30), he does not disengage but continues to attend to what the adult is saying. Notice that this behaviour from the child functions as a means of eliciting further behaviour from the adult. What looks like a failure to respond is in fact a form of engagement. At any moment, the child could choose to wriggle out of the adult’s grip and find some other activity to get involved in. But he doesn’t. In reality, the child is driving the game as much as the adult is. Even an apparent absence of response can in fact be an active attempt by the child to disturb the structure of his surroundings.

It is not always easy to notice that learning to speak is such an active process. (Notice how, a couple of times in the video, the grandmother interprets the child’s attentive non-responding as the child’s way of saying “this is boring”.)

When viewed in the wild, learning to speak does not look like a process of inferring mappings over a set of forms and a set of meanings. It looks like a series of explorations by the child of the environment it finds itself surrounded by—an environment that contains other actors who respond in meaningful ways to the child’s own activity. It is not the case that in learning to speak the child must learn some abstract set of patterns that we call a language (e.g. English). The child’s environment contains mostly just a small set of actors who are intermittently present. The child’s immediate task is to learn how to influence and manage the behaviour of these particular actors. There’s no need to learn an abstract set of conventions that can be used to communicate with an ideal conversation partner: what matters is what’s in the child’s surroundings.

Speaking is acting to alter the layout of a populated environment

A young child points out of a train window

A young speaker manages her environment (image source)

If my claim is that we should not view (early-childhood) speaking as involving conventional mappings, then I need to be able to provide an alternative account of what speaking, in fact, is. This is hard to do but I think it’s worth a try.

Again, let’s start with the infant at the early stages of learning to talk. The infant is born with the ability to make sounds. Different types of verbal action will result in different kinds of responses from the infant’s environment. From the day the infant is born, it can act on its environment by crying. Crying is a type of acting that not only disturbs the ambient sound patterns in the environment; it also perturbs the activity of other actors in the infant’s surroundings (see Thompson et al, 1996). But crying is not a very accurate means of eliciting a response: caregivers respond to crying but don’t immediately know exactly what to do to bring the crying to an end.

As the child’s repertoire of sound-making skills expands, the child will be able to elicit increasingly specific responses from its caregiver-containing environment. The crucial developmental shift occurs when the child learns to control its verbal actions with reference to a specific type of relation that exists in its environment: relations that stand between other actors and other objects and events in the environment (this occurs at around nine months of age; Tomasello calls it the “nine-month revolution”). At this stage, the child starts to point to, and to name, objects and events.

How does this work? Well, it works because of the caregiver’s own history of learning. If the infant produces some sort of sound that approximates “doggie”, this orients the caregiver’s attention towards dog-shaped objects in the caregiver’s environment. How does this work? I confess, I have no real idea. But I don’t think it needs to involve conventions. The caregiver does not need to recognize the conventional link between this particular sound pattern and this particular object. The sound orients the caregiver’s attention because the caregiver’s nervous system is configured in such a way that the one immediately triggers the other. The caregiver’s nervous system got that way because of activities the caregiver engaged in in the past. What matters is that this link within the caregiver’s nervous system (between sound pattern and object) is a persisting piece of structure in the infant’s environment. The infant cannot directly perceive the caregiver’s nervous system, but can perceive the behavioural patterns that it generates. The infant can repeatedly produce the sound “doggie” at irregular intervals over weeks and months, and the sound will reliably go along with the caregiver’s attending to the object. This is the kind of structure that the child needs to be able to attend to in order to learn to speak—in order to learn to elicit meaningful responses from its environment.

One difficulty here is that, while it seems reasonable to view speaking as a kind of acting, it is not very clear how we should view the status of the addressee.

Being spoken to is something we do; it is also something that we have done to us

I liked this sentence from Sabrina’s paper, which is also in the special issue (Golonka, 2015): “A tacit assumption in ecological psychology seems to be that anything that has an effect on behavior must be grounded in the perception of an affordance and, therefore, must be guided by law-based information.” I think that’s true: ecological psychology is disposed to overlook anything that’s not information-based, first-person control of an activity.

But suppose I punch you in the leg. This will certainly have an effect on your behaviour. You’ll probably hobble around for a bit and complain. Your behaviour here is not grounded in the perception of an affordance. It is not a response to informational input. It is a response to an internal change in you as a system which I, rather rudely, brought about.

Being spoken to is a bit like being punched in the leg, sometimes. When you are spoken to, your attention is directed towards things that you yourself are not in control of selecting. It is the speaker who selects what you are to attend to, and the speaker achieves this by performing an action that perturbs your nervous system in an appropriate way.

To get things a little clearer here, being spoken to is two things:

  1. Being spoken to is acting: in order to engage with someone in the first place it is necessary to seek that person out and to not run away from them, and in order to keep the situation moving it is necessary to actively attend to the situation you are in and to make appropriate adjustments to your own behaviour (as the kid in the video does)
  2. Being spoken to is something you undergo: it is having your nervous system perturbed by the actions of another actor—the speaker

How these two processes fit together is a central problem for any psychology concerned with the problem of acting in a populated environment, or with collective activity.

What should be evident is that the second item here—having your nervous system acted upon (like being punched in the leg)—cannot really be described as a case either of acting or perceiving. It is not a straightforward case of perception-action because what is going on here is not under the control of the person being spoken to.

As a field, we have not yet come up with a satisfying way to capture this non-voluntary aspect of public language as part of our theorizing. My hunch is that a big reason for this is that we have been trying to extend James Gibson’s account of visual perception to deal with something which it is not equipped to deal with. Gibson’s description scheme treats perception as the activity of a single actor exploring its environment, which, for the purposes of much of his work he treats as if it were inanimate: the environment is described as a set of objects and surfaces whose structure is projected into energy arrays. The problems of explaining behaviour in a populated environment are, quite reasonably, largely set aside in Gibson’s books. But as a consequence, the description scheme he ends up with is one that cannot straightforwardly be extended to describe what’s going on in multi-actor settings. If we want to understand how speaking works we’re going to have to look beyond the perceiving-acting behaviour of the individual; we’re going to have to try to understand the larger context within which speaking behaviour is situated.


The editors of this special issue are searching for a way of describing language that is compatible with the anti-representational tenets of the ecological approach. This is clearly a difficult project. There are several traps that must be avoided.

One of these is the temptation to simply map the terms “speaking” and “listening” onto the terms “acting” and “perceiving”. Doing so obscures the nature of speaking as a form of action directed at an environment that is structured by the presence of other actors.

Another trap is the temptation to see linguistic meaning as fundamentally a relation between a word form and an event in the world. This is a disembodied view of meaning. In reality, meaning does not need to be disembodied in this way because meaning is distributed throughout the entire system that a child finds itself in. In the paper I wrote that: “The act of speaking is not meaningful because of the words it contains but because of the effect it has on the meaningfully structured environment into which it is directed.”

Clearly there is much more work to do before we can claim that we have a useful description scheme for talking about language in a properly ecological or enactive way. Are we moving towards one? Hard to say. I’d like to see a lot more attention paid by anti-representational researchers to developmental processes and to the peculiarities of the structure of settings that contain multiple actors. Until we have an account of speaking that is anti-representational from the ground up we risk falling back into the familiar conception of speaking as the sending of messages in a speech code. I think we can do better than that.


How direct is social perception? A comment on Abramova and Slors (2015)

Students constructing a staircase, each working separately on a particular part.

Students constructing a staircase, each working separately on a particular part; Abramova and Slors (2015) label such activity as “distributive action coordination”. (Image source)

It has become trendy recently to talk about social perception as a direct process. Or, at least, I gather it has become trendy, because the journal Consciousness and Cognition has a special issue coming out on the subject. Most of the articles appear to be playing the game of taking a novel phrase, in this case “direct social perception”, and trying to cram that phrase into the author’s preferred psychological description scheme, be that Bayesianism, theory-of-mind cognitivism, or some other approach. Each to their own. I don’t have anything much to say about these articles. There was one paper that caught my eye, though. It is written by Ekaterina Abramova and Marc Slors (A&S), and it attempts to give an account of “simple action coordination” in explicitly Gibsonian terms—in terms of individual actors, or agents, who, through their actions, “shape the field of affordances for another agent”.

This strikes me as a move in a useful direction, and one to be encouraged. It is also one that immediately raises a whole set of conceptual issues which demand careful thought.

I do not want to critique the paper itself, as such. The proposals presented are quite preliminary in nature (the authors themselves claim only that they “have tentatively made an indirect case in favor of [direct social perception] in simple action coordination”). Instead, I want to take issue with the way the authors seem to be interpreting the concept of affordances. This will, I hope, be illustrative of some of the issues that arise when we appeal to affordances in trying to explain social phenomena; these issues are general and go well beyond this particular paper. I want to suggest that A&S’s reading of the concept of affordances is unduly narrow. As a consequence of this, they end up maintaining certain notions which I believe we’d be better off simply abandoning. Specifically, they maintain that coordinated activities involving multiple actors must be supported by a “shared intention”; and they appear to be committed to the notion that acting with others necessarily involves “understanding” others in some sense—e.g., “recognizing” that certain objects afford certain actions for others. I want to suggest that there is a way to formulate the notion of affordances which allows an account of coordinated multi-actor activity in which these issues simply do not arise.

It is, however, worth noting a couple of things about the paper up front. Firstly, the phrase “direct social perception” is being appealed to here in a promissory manner. It is being invoked as part of a rejection of the mainstream, cognitivist account of acting with others as involving the making of inferences about others’ intentions. On the mainstream account it is assumed that intentions are a matter of private mental activity and that therefore they are inherently inaccessible to anyone but the individual whose intentions they are. A&S reckon that their affordance-based description scheme allows them to avoid having to appeal to inference-making, although it is not always clear from what they write that they have completely escaped thinking about the actions they describe in terms of inferences. Thus, in their conclusion they claim that acting with another agent requires “perceiving affordances for another person [which] involves seeing that person as an agent”, and that this “does involve attributing some basic kind of intentional relation between a person and her environment”. It is hard to interpret that “attributing” as anything other than a form of inferring.

Secondly, the authors make a distinction between two types of acting with others:

    1. distributive action coordination, in which actors work on their own tasks while avoiding getting in each other’s way, as in the photograph at the top of this post
    2. contributive action coordination, in which multiple actors use one another to complete a task which none would be able to perform independently, as in the ubiquitous sofa-carrying case; A&S write: “In such cases coordinated joint action is usually required to start simultaneously”


Four men carry a sofa together.

A rare four-man sofa-lift; according to A&S this is a case of “contributive action coordination”. (Image source)

The authors think that their affordance-based scheme can straightforwardly explain what’s going on in the case of distributive action coordination (and that this can already be modelled using dynamic field theory), but that some work is required before an account of coordination of the contributive type can be attempted. This distinction is, intuitively and descriptively, a reasonable one to make. However, I think the distinction breaks down quite rapidly when we consider the situation in terms of affordances relative to an individual actor (see this previous post in which I argue against analysing the situation primarily in terms of interpersonal relations; I claim that we should instead focus on the relations between the actor and its populated environment).

Perceiving affordances for others is not like perceiving affordances for oneself

In their attempt to give an account of contributive, sofa-carrying-type action coordination, A&S suggest that it is necessary that the actor perceive their environment not only in terms of what it affords for them, but also in terms of what it affords for others. They cite the following passage from James Gibson’s chapter introducing the “theory of affordances” (Gibson, 1979, p. 141):

The child begins, no doubt, by perceiving the affordances of things for her, (. . .). But she must learn to perceive the affordances of things for other observers as well as [for] herself.

As with a number of the remarks that Gibson makes in that chapter, this one can be read in a couple of different ways:

  1. Perceiving the affordances of things for others requires one to perceive that, from where the other person is standing, that person’s environment affords action X
  2. Perceiving the affordances of things for others requires one to attend to and discriminate a correspondence, or fit, between two pieces of structure in one’s own (the perceiver’s) environment: a correspondence between the object and the other actor

The first reading appeals to what in the cognitive tradition is called other-modelling. A&S appear to be leaning towards the first reading: “in the plank-lifting case, the field of affordances of a person standing at one end of a plank is changed when she sees another person at the other end of the plank. The plank becomes ‘jointly liftable’ for her because she recognizes the availability of exactly the same affordance for the other person as well.” This seems to sit uneasily with any claim to be pursuing a direct perception account of social activity, because other-modelling adds an inferential step between what is seen (what is specified in the optic array) and what is “perceived”.

The second reading is more interesting, I think. Here there is no requirement to put oneself in the place of the other. On this reading, perceiving affordances for others is nothing like perceiving affordances for oneself. And how could it be? If affordances are animal-relative, as we are told is the case, then we can only directly perceive our own environment in terms of what it affords to us as individuals. Any notions I might have about how the world looks to you are a matter of conjecture on my part. What I perceive is an environment (my surroundings) which happens to be populated with other actors, whom I perceive in relation to all the other objects in my environment. I might perceive you and the sofa as objects that I can potentially assemble in a way that affords sofa-carrying. But this is no different in degree from my perceiving the relation between a sofa and, say, one of those flat trolleys you have to push around when you get to the warehouse bit in Ikea. The trolley and sofa can likewise be assembled into a structure for sofa-transportation (over a flat surface, at least). The point is that the other person, like the trolley, is, for the purposes of the task at hand (shifting this sofa to some place else), merely usable structure in the environment.

To be clear, the claim is that perceiving is only ever of an environment that affords actions to the animal doing the perceiving. Consider an example from soccer. A striker bearing down on the opponents’ goal must pick out a particular place within the goalmouth into which to aim the ball such that the goalkeeper will not be able reach it. We might say that this involves the striker perceiving the affordances for the goalkeeper. And in a sense it does. But this does not mean that the striker must know what it is like to be the goalkeeper. The striker need have had no experience of being a goalkeeper and need have no appreciation of what the situation looks like from the goalkeeper’s point of view. The striker perceives the situation in terms of what kinds of action will most likely result in a goal—in terms of what kinds of shot the goalkeeper is unlikely to reach. The goalkeeper is a mere obstacle, to be negotiated, evaded, got the better of.

Note that on this way of thinking about affordances it does not make sense to talk of what the environment affords for the collective, or to talk of “fields of affordances for dyads or groups”. Groups do not experience their surroundings or perceive affordances, only actors do.

Affordances make “shared intentions” unnecessary

An American beaver in Alaska lifts its head above the dam.

Beavers don’t waste their time worrying about whether or not the other beavers in the colony are committed to a shared intention for dam-maintenance. (Image source)

In the abstract for their paper, A&S write: “Coordination ensues, we argue, when, given a shared intention, the actions of and/or affordances for one agent shape the field of affordances for another agent” (my emphasis). They ask us to simply accept that coordination requires a pre-existing shared intention. The problem here is that, if you share an intention with someone (whatever that means), then you must already be coordinated with that person in some sense. Coordination does not “ensue”; it has already been presupposed. The question of how that coordination came about has been begged.

Given how opaque this notion of “shared intentions” is, there is good reason to want to avoid invoking it in the first place. Fortunately, the affordance concept is powerful enough to allow us to do this. The crucial thing is to note that the field of affordances surrounding an animal is subject to change as the internal structure of the animal changes.

Consider the beaver. Beaver dams are maintained by members of the beaver colony. This might be taken as a case of what A&S call “distributive action coordination”: each individual beaver takes part in building and repairing a structure that is beneficial to all. But there is no need to appeal to shared intentions here. (And since we are talking about a non-human animal here, we are naturally less inclined to do so.) The beaver’s motivation to repair the dam is explained in this delightful Attenborough clip. Beavers are apparently averse to the sound of running water, so when they hear the sound of water rushing out of a damaged section of the dam they are drawn towards that location and immediately start work on the repairs. Put another way, the sound of running water triggers an internal change in the beaver that reorients the animal’s activity.

Or consider the sofa-carrying case again. It is not the case that the actors involved must initiate their actions “simultaneously”. In fact, there is no natural start point to the action. To understand how the actors arrived at the position of being about to lift this particular sofa it is necessary to look backwards and ask: what happened before now that led to the present situation? Perhaps I am moving house and you agreed a couple of weeks ago to help move the furniture. Why not take that as the starting point of the action, then? When you agreed to help, you became oriented towards the furniture in a particular way: there was a change in your internal structure such that you went from being neutral with reference to this furniture to being action-oriented relative to it. The furniture became a set of objects to be moved. And as we now stand at either end of the sofa, the idea that we might need to establish a shared intention before we can get this bulky object off the ground simply does not enter into consideration. We both became object-moving devices some time earlier, and this is manifest in the fact that we are both here, and in the fact that we have probably already shifted a bunch of objects out of the room.

Or we could take the explanation further back and ask how it was that we learned to lift bulky objects in coordination with others in the first place.

The point is that, appropriately understood, the concept of affordances simply obviates the need to think about these situations in terms of shared intentions. It is perhaps useful here to make a distinction between “action” and “task”. Action has no start point. Actors do not simply engage in routine performances, or scripts, that have a clearly defined start and end point. Action is continuous: everything the actor does is part of an ongoing process in which the actor changes its environment and in doing so changes its own internal structure. I have argued (Baggs, 2014) that the task, by contrast, is an analytical device that allows us to artificially break up this stream of action into tractable chunks that can be studied; these chunks need to have a clearly defined start and end point and a characteristic mode of transition between the two. The outfielder problem is a good example: we can define the start (the ball being launched), the end (the catch), and the transition (the parabolic trajectory of the ball through the air), and this is very useful as a guide for what we should measure. But we should not make the mistake of thinking that the task as we have defined it is the same as the action from the perspective of the actor. In the context of action in multi-actor settings, such a narrow definition creates unnecessary problems. If we assume that the action itself starts at the point that we happen to start observing or measuring, i.e. at the point when we are already standing at either end of the sofa, then it looks like our behaviour must be structured by some hidden “shared intention” which is not itself perceivable. What is overlooked is the fact that our internal structure has been shaped by previous action. If the affordances for lifting the sofa are directly perceivable it’s because we perceive the situation we are in relative to the type of animal we have become.

“Direct social perception”: a conceptual blind alley?

A note of caution is in order about this phrase, “direct social perception”. The existing phrase, “direct perception”, at least as Gibson uses the term, is meant to denote a relationship between an actor and its environment. Here is a description from Turvey, Shaw, Reed and Mace (1981, p. 239):

Gibson argued that the proper “objects” of perceiving are the same as those of activity. Standing still, walking, and running are all relations between an animal and its supporting surface. Though not always explicitly recognized … the supporting surface is just as much an essential constituent of these activities as, for instance, legs; and useful perceiving involved in controlling posture and locomotion must be directed toward the same surface. Thus it would seem that a two-term relation involving the same surface or ground can exist in both cases: an animal *runs* on the ground and an animal *sees* the ground. This much should be common sense. There is no thing *between* the animal and the ground in the relation. This is what Gibson has always meant by direct perception and it is the same as what one would mean by direct action if one were discussing activity.

Even if one may be inclined to take issue with some of this, it is at least clear that direct perception means something: it refers to a particular way of thinking about what’s going on between the animal and its environment.

This is not the case with the phrase “direct social perception”. This is because “social” is a descriptive category, introduced by a third-party observer or an analyst. A situation is described as “social” if it involves multiple actors. But just because we can describe a situation as “social”, we are not necessarily licensed to assume that the “socialness” of the situation must play a causal role in our eventual explanation of the actors’ behaviour. (See the parallel argument here for the phrase “joint action”.) We might describe the beaver as a social animal, yet the beaver does not need to appreciate its own status as a social animal in order to fix its dam.

It may be prudent to avoid the phrase “direct social perception” altogether, on the grounds that the inherent mixing of perspectives—descriptive and explanatory—is bound to lead to confusion further along the line. (I will however point out Eric Charles’s claim that there’s a way of reading ecological psychology according to which other minds are perceivable in the sense that the actions of others are perceived in the context of larger, ongoing patterns of behaviour.)

Some devices for avoiding conceptual traps

In lieu of a summary, I conclude with some recommendations for avoiding some of the ever-present conceptual traps surrounding the analyst of multi-actor activity.

  • Shared intentions only seem necessary as a function of the language we use to describe situations. To avoid this, consider non-human animals: the beaver, or, better, the ant. Ants don’t do shared intentions. When proposing an account of multi-actor coordinated activity, ask: how would this apply to ants.
  • Describing something as “social” or “joint” is all well and good as a description, but do not make the mistake of thinking that just because you (the analyst) can describe something as social, that the socialness of the activity must necessarily play a causal explanatory role in an account of the behaviour of any given individual actor.
  • Actions do not begin at the point that you start measuring them. Tasks, on the other hand, do. Tasks are a methodological device that establishes some more or less arbitrary start and end point (Baggs, 2014). Do not confuse the action an actor is performing with the task as you have defined it.


Is ecological psychology compatible with enactivism? Is information always animal-relative? A comment on Van Dijk, Withagen, and Bongers (2015)

One question I have had for a while is why it is that there seems to be so little overlap between ecological psychology and enactivism. Both approaches are explicitly anti-representational, both seek to treat the organism and its environment as constituting a system, and both draw their inspiration and metaphors from biology and ecology rather than, as is the case with cognitivism, from the fields of computer science and logic. And yet believers in the enactivist approach seem to have trouble communicating their ideas to believers in the ecological approach (among whom I count myself), and vice versa. It’s like there are two separate churches and the congregation inside each simply cannot comprehend the experience that led others to join the rival church.

So I was interested to read this paper by my fellow ecological psychologists Ludger Van Dijk, Rob Withagen, and Raoul Bongers (2015) who attempt to reach out to the enactivists and allay some of their concerns about the ecological approach. (Disclosure: I’ve had the pleasure to spend some time chatting with Ludger at a couple of conferences and have found we are generally in agreement on the deep issues in the field. I had nothing to do with the creation of this paper.) The paper has gone some way to clearing up for me just what the enactivists’ objection is to the Gibsonian programme. But it also raises a new issue. The authors ask ecological psychologists to abandon the concept of information about the environment in favour purely of talking about information for affordances.

I am sympathetic to this proposal. However, I do not have a very clear idea about how it will be received by the rest of the field. I am curious to find out. I suspect Van Dijk et al have understated just how much of a challenge this is to ecological psychology as it is currently practiced. I will lay out Van Dijk et al’s argument here in an attempt to set out as clearly as possible just what the dilemma is that ecological psychologists are faced with.

A content-free conception of information?

Enactivists have long worried that the ecological programme does not go far enough in rejecting the traditional dualisms of psychology, specifically the dualism that asserts that content or meaning exists externally to the animal and independently of the activity of that animal. As Van Dijk et al explain:

[Ecological psychologists] typically talk of information a lot, and it is not always clear whether this information is of a content-less kind. Indeed, the enactivist tradition seems worried about this. Varela et al. (1991), for example, felt that Gibsonians were building their “theory of perception entirely from the side of the environment” (Varela et al., 1991, p. 204). They worried that the ecological notion of information that grounded information in the correspondence between the structure of ambient light and the environment, required too little active participation of the animal.

In short, Varela et al accuse Gibsonians of merely appearing to talk the anti-representational talk while smuggling in content by the back door. They read Gibson’s description of what’s-out-there-to-be-perceived (his ecological optics) as focusing only on how light comes to contain structure from the surrounding world without explaining how the animal makes use of that structure. This leaves an explanatory gap between the structure in light and the biological and phenomenological processes internal to the animal. Or, put another way, Varela et al suspect us Gibsonians of being closet dualists, because for all our talk of the animal in its environment as a perceiving-acting system, we do not go far enough in recognizing that perceiving and acting are in fact one and the same process. Simply “picking up” information that already exists in the array is not acting; it is registering or representing. It is a passive process. Only if the target of perceiving (the objects and surfaces and so on) and the act of perceiving are truly flattened into the same process can we remove any whiff of mind-body dualism.

The solution that Van Dijk et al propose is to abandon all talk of information as being “about” something in the environment. Information-about implies that meaning exists externally to the animal and that it exists prior to the animal’s trying to act on the environment. It implies a notion of information that “carries content” from the objects in an animal’s environment to the animal’s sense receptors, to be passively detected. Instead, we should think of information as content-less: before the animal tries to act, there is no “information”, as such. Information arises in the animal’s acting on and exploring of its surroundings. The information that arises is information-for affordances; that is, it reveals to the animal possibilities for subsequent action:

“Information for” calls attention to the fact that ecological information need not be about anything – has no “aboutness” – prior to use, but it is for something to an active animal. It is for perceiving the environment, for acting on affordances and the likes.

As I stated above, I’m sympathetic to this proposal. But what Van Dijk et al do not really talk about (although they allude to it) is just how at odds this proposal is with the way information is generally talked about by ecological psychologists today.

Can ecological psychology make do without information-about?

Van Dijk et al argue that in his later work Gibson was moving away from talk of information-about in favour of purely talking about information-for. Abandoning information-about should, then, be no big deal. Except that there’s a problem: whatever direction Gibson himself might have been moving in, his followers did not necessarily continue to move in that direction. Indeed, the acceptance of both information-for and information-about is core to modern ecological psychology. Michaels and Carello make this point quite explicitly (1981, p. 37):

A description of the relation between the environment and the structure of light and sound provides one facet of the concept of information—that is, *information-about*. These patterns of energy describe objects, places, and events in the environment. However, there is a second and equally important facet of information that information-about does not touch upon. This second facet of information is *information-for* and the object of the preposition *for* is the animal. Information is the bridge between an animal and its environment and cannot be usefully described without a specification of *both*.

And, strikingly (Michaels and Carello, 1981, p. 38):

To summarize, a failure to consider both parts of information—information *about* an environment *for* an animal—is to miss the very essence of the concept of information.

In the formulation of ecological psychology that followed Gibson, then, information was explicitly defined as having two parts, and this two-part nature was seen as a feature of the concept of information, not a bug. This dual property is formalized in the symmetry principle which was put forward by Turvey, Shaw, and Mace, and which is discussed here by Andrew. The symmetry principle is presented as an account of how it is that perception can be direct. The environment is said to specify patterns in light (information) which in turn specify what is perceived. Conversely, what is perceived is said to specify what the pattern is that is in the light which in turn specifies what is in the environment (hence “symmetry”). According to this formulation, it is not a problem if light is said to carry information about the environment. On the contrary, this is seen as a good thing because it explains how it is that we perceive what is actually in our environment. “Information” is merely the central term in a set of 1:1 correspondences that point in two directions at the same time: to the environment and to the perceiver. (But note that while the symmetry principle makes reference to perception it does not make reference to the activity of the animal, which lends support to the enactivists’ suspicion that perception is being covertly treated as the passive inputting of content.)

Other formulations of information by ecological psychologists also rely on information-about. Both Chemero (2009, p. 116 and following pages) and Wilson and Golonka (2013) attempt to develop a version of information that draws on Barwise and Perry’s situation semantics. Here, the symmetry principle is abandoned and it is allowed that information can be “about” something that is not directly specified in the pattern in the energy array. So a spoken sentence can be said to be about something more than just the sound pattern and the vocal equipment that produced that pattern: it can be said to be about what the sentence is in fact about. Instead of appealing only to specification, these authors appeal to constraints that have to be learned, and that link the pattern to its meaning. But, crucially, the pattern is still said to have a meaning, independently of the activity of the actor. This is what Van Dijk et al are asking us to give up: the idea that meaning exists independently of the animal and its actions.

The dilemma for Gibsonians

A confession: not so long ago I used to think of enactivism as merely a form of old-fashioned idealism masquerading as modern biologically-inspired scientific psychology. Enactivism is easy to dismiss because the book that’s presented as foundational to the field, Varela et al’s (1991) The Embodied Mind, is also a self-indulgent foray into Buddhist scholarship. I used to think that the only reason that enactivists weren’t Gibsonians was because they simply hadn’t appreciated the enormity of Gibson’s achievement. Now I am not so sure. I am now persuaded that at least some of the problems that enactivists see are real, and that those problems are not merely an artefact of how the ideas have been communicated to outsiders but are internal to the ecological approach. Van Dijk et al’s paper has helped crystallize that for me. It is not enough to just insist that we are right; we need to do more work if we wish to persuade others that that is the case. So it will be useful to try to be clear about just what the challenge is that we are faced with.

Part of the problem may be that Gibson’s approach to perception is in large part really a theory of structure. Ecological optics is about the correspondence between structure in light and the structure of the world. It provides an account of what’s out there to be perceived, but not of how we learn to make use of that structure, still less of how our use of that structure comes to shape the things we can do when we’re no longer looking at the structure directly (e.g. how it is that I can remember scoffing at Varela et al’s arguments even though I no longer have their book in front of me). It’s been pointed out that the way James J. Gibson and Eleanor J. Gibson divided up their study interests—the former focusing on the senses while the latter focussed on perceptual learning—has introduced a long-standing divide into the entire field. If we really want to persuade the enactivists that Gibson is worth following then we are going to have to take on the hard work of putting the active learner back into its environment. Van Dijk et al argue that this requires a concept of information that is truly animal-relative and content-free. (They also argue that James J. Gibson was himself engaged in pursuing this conception as he attempted to develop his theory of affordances.)

The dilemma appears to be thus:

  • We can concede that much of ecological psychology is not really a theory of psychology but a theory of structure, a theory of what’s-out-there-for-an-animal-to-perceive; this need not be seen as a defeat: we can insist that Gibson’s account is the best account of structure that we have and that no anti-representational approach to psychology can succeed that doesn’t acknowledge Gibson’s great achievement here
  • Or we can insist that Gibson’s approach really does constitute a theory of psychology, and we can continue to conduct empirical research in the traditional ecological manner: taking some arbitrary task (such as stair-climbing or aperture-passing) and attempting to identify the invariant that is implicated in the control of that task; this option has worked well up to now and it has certainly been productive; but this approach may preclude ecological psychologists from having anything much to say about phenomena that go beyond the immediate perception of structure, things like language and other social phenomena (i.e., the concerns of this blog), or indeed about any phenomenon that appears to involve creativity or spontaneity on the part of the actor

The strange thing is that Van Dijk et al wrote their paper in an attempt to ride to Gibson’s defence, against enactivist attacks. In doing so they have revealed to us just how much work we still have to do. I suppose this may seem like a rather negative conclusion but I don’t think it is. I think it’s quite exciting that some of this hard work is still there to be done. I’m curious to find out if others feel the same way.