Visual Population Codes: Toward a Common Multivariate Framework for Cell Recording and Functional Imaging

651 Pages·2011·11.937 MB·English

by  Nikolaus Kriegeskorte, Gabriel Kreiman

Contents
Series Foreword 1x
Preface xi
Introduction: A Guided Tour through the Book 1
THEORY AND EXPERIMENT 21
1 Grandmother Cells and Distributed Representations 23
Simon J. Thorpe
2 Strategies for Finding Neural Codes 53
Sheila Nirenberg
3 Multineuron Representations of Visual Attention 71
Jasper Poort, Arezoo Pooresmaeili, and Pieter R. Roelfsema
4 Decoding Early Visual Representations from fMRI Ensemble
Responses 101
Yukiyasu Kamitani
5 Understanding Visual Representation by Developing Receptive-Field
Models 133
Kendrick N. Kay
6 System Identification, Encoding Models, and Decoding Models: A Powerful
New Approach to fMRI Research 163
Jack L. Gallant, Shinji Nishimoto, Thomas Naselaris, and Michael C. K. Wu
7 Population Coding of Object Contour Shape in V4 and Posterior
Inferotemporal Cortex 189
Anitha Pasupathy and Scott L. Brincat
v1 Con~n~
8 Measuring Representational Distances: The Spike-Train Metrics
Approach 213
Conor Houghton and Jonathan D. Victor
9 The Role of Categories, Features, and Learning for the Representation of
Visual Object Similarity in the Human Brain 245
Hans P. Op de Beeck
10 Ultrafast Decoding from Cells in the Macaque Monkey 275
Chou P. Hung and James J. DiCarlo
11 Representational Similarity Analysis of Object Population Codes in
Humans, Monkeys, and Models 307
Nikolaus Kriegeskorte and Marieke Mur
12 Three Virtues of Similarity-Based Multivariate Pattern Analysis: An
Example from the Human Object Vision Pathway 335
Andrew C. Connolly, M. Ida Gobbini, and James Y. Haxby
13 Investigating High-Level Visual Representations: Objects, Bodies, and
Scenes 357
Dwight J. Kravitz, Annie W-Y. Chan and Chris I. Baker
14 To Err Is Human: Correlating fMRI Decoding and Behavioral
Errors to Probe the Neural Representation of Natural Scene
Categories 391
Dirk B. Walther, Diane M. Beck, and Li Fei-Fei
15 Decoding Visual Consciousness from Human Brain Signals 417
John-Dylan Haynes
16 Probabilistic Codes and Hierarchical Inference in the Brain 441
Karl Friston
II BACKGROUND AND METHODS 475
17 Introduction to the Anatomy and Function of Visual Cortex 477
Kendra S. Burbank and Gabriel Kreiman
18 Introduction to Statistical Learning and Pattern Classification 497
Jed Singer and Gabriel Kreiman
19 Tutorial on Pattern Classification in Cell Recording 517
Ethan Meyers and Gabriel Kreiman
20 Tutorial on Pattern Classification in Functional Imaging 539
Marieke Mur and Nikolaus Kriegeskorte
Contents vii
21 Information-Theoretic Approaches to Pattern Analysis 565
Stefano Panzeri and Robin A. A. Ince
22 Local Field Potentials, BOLD, and Spiking Activity: Relationships and
Physiological Mechanisms 599
Philipp Berens, Nikos K. Logothetis and Andreas S. Tolias
Contributors 625
Index 629
Series Foreword
C omputational neuroscience is an approach to understanding the development and
function of nervous systems at many different structural scales, including the bio-
physical, the circuit, and the systems levels. Methods include theoretical analysis and
modeling of neurons, networks, and brain systems and are complementary to empiri-
cal techniques in neuroscience. Areas and topics of particular interest to this book
series include computational mechanisms in neurons, analysis of signal processing
in neural circuits, representation of sensory information, systems models of senso-
rimotor integration, computational approaches to biological motor control, and
models of learning and memory. Further topics of interest include the intersection
of computational neuroscience with engineering, from representation and dynamics
to observation and control.

Terrence J. Sejnowski
Tomaso Poggio
Preface
T his is a book about visual information processing in primate brains. As in other
biological networks, the function of the visual system emerges from the interaction
of the system’ s components. Such inherently interactive phenomena cannot be
understood by analyzing the components in isolation. In neuronal coding, the idea
that the whole is more than the sum of its parts is exemplifi ed by the concept of
“ p opulation code,” the idea that visual content is represented, at each stage of the
visual hierarchy, by the pattern of activity across the local population of neurons.
Although this concept appeared decades ago in neurophysiological studies of brain
function, the dominant approach to measurement and analysis has been to focus on
one cell at a time and to characterize its selectivity, receptive fi eld, and other proper-
ties. A similar approach has been followed in the context of functional imaging,
albeit at a much coarser spatial scale. Although functional imaging measures complex
spatiotemporal activity patterns, most studies have focused on regional-average
activation.
T he theoretical concept of the population code motivates multichannel measure-
ment and multivariate analysis of activity patterns. Population analyses have a long
history in vision and other fi elds. Notable examples include the decoding of motor
commands from population activity in motor cortex and parietal cortex, the decod-
ing of a rodent ’ s position from the population activity of hippocampal neurons, and
analyses of the population coding of olfactory information. Despite these examples,
the dominant approach to understanding neuronal representations has been uni-
variate analysis.
O ver the past decade, the multivariate approach has gained signifi cant momen-
tum, especially in the fi eld of vision. Many researchers now analyze the information
in complex activity patterns across many measurement channels. Functional imaging
and cell recording measure brain activity in fundamentally different ways, but they
now use similar theoretical concepts and mathematical tools in their modeling and
analysis. Results indicate that the interactions between sites do matter to neuronal
coding.
xii Preface
At the micro-scale, the study of single-neuron responses continues to produce
valuable insights. And at the macro-scale, classical brain mapping with its focus on
regional-average activation continues to defi ne the big picture of brain function. But
cell recording and functional imaging are beginning to close the gap of spatial scales
between them and invade the meso-scale, where regional population codes reside.
In terms of measurement, high-resolution imaging is invading this intermediate
scale by providing sub-millimeter resolution, and multi-electrode neuronal record-
ings promise to give us a richer picture of regional single-cell activity. In terms of
analysis, a common multivariate framework for analyzing population codes is
emerging. This framework promises to help bridge the divide between cell recording
and functional imaging and between animal and human studies. Moreover, it prom-
ises to allow us to test computational network models by integrating them in the
analysis of brain-activity data.
The purpose of this book is to present recent advances in understanding of visual
population codes afforded by the multivariate framework, to describe the current
state of the art in multivariate pattern-information analysis, and to take a step
toward a unifi ed perspective for cell recording and functional imaging. The book
should serve as an introduction, overview, and reference for scientists and graduate
students across disciplines who are interested in human and primate vision and,
more generally, in understanding how the brain represents information.
The fi rst part of the book, “ T heory and Experiment,” is coarsely organized accord-
ing to the fl ow of visual information from the retina to the highest stages of ventral-
stream processing. Most of the chapters combine a review of theory and published
empirical fi ndings with methodological considerations. The second part, “ B ack-
ground and Methods,” is intended to provide readers from different disciplines with
essential background on vision, different techniques for measuring brain activity
(and their relationships), and mathematical analysis methods. This preface is fol-
lowed by an introduction ( “ A Guided Tour through the Book ” ), which explains
some basic concepts, summarizes each chapter, and clarifi es the chapters’ relation-
ships. Chapter abstracts provide a further level of detail to allow a quick grasp of
the information.
W e have roughly organized the book according to the stages of visual processing,
interspersing animal cell recording and human imaging studies, so as to emphasize
the commonality of subject matter between these still somewhat separated fi elds.
Readers may discover that the perspective of functional imaging has a lot to con-
tribute to that of cell recording, and vice versa. They may also fi nd that the way
questions are framed for early visual areas may help rethink the challenges of
understanding higher-level representations, and vice versa. As mentioned above,
multivariate analyses have provided important insights in other domains beyond
vision. We encourage the reader to examine this rich literature, and we hope that
Preface xiii
the multivariate framework for analyzing population codes will benefi t from an
exchange between vision and other fi elds.
I n order to understand brain function, we need to develop theory, experiment,
and analysis conjointly, in a way that embraces the parallel and interactive nature
of cortical computation. The emerging multivariate framework is an important step
in that direction, helping us make sense of ever richer spatiotemporal brain-activity
data and enabling us to see the forest, too, not just the trees.
Several people have helped throughout this effort. We would like to thank
Christina Chung and Jane Tingley, who helped with several aspects of the book. We
also acknowledge the patience and wisdom from the people at MIT Press while we
learnt our initial steps in making this work a reality. The work towards this book
was made possible through funding from the National Science Foundation (0954570;
BCS-1010109), National Institute of Health (R21 EY019710; X02 OD005393;
R21NS070250), the Whitehall Foundation, the Klingenstein Fund, the Massachusetts
Lions Eye Research Foundation, and the UK Medical Research Council.

Introduction: A Guided Tour through the Book
This chapter gives an overview of the content of the book. We follow the chapters
in the sequence in which they appear, summarize key fi ndings and theoretical argu-
ments, and clarify the relationships between the chapters. Along the way, we explain
some basic issues of overarching importance.
T he book is divided into two parts: “ T heory and Experiment” and “ B ackground
and Methods. ” The fi rst part describes recent primary research fi ndings about the
visual system, along with cutting-edge theory and methodological considerations.
The second part provides some of the more general neuroscientifi c and mathemati-
cal background needed for understanding the fi rst part.
A lthough each chapter is independent, the fi rst part, “ Theory and Experiment, ”
is designed to be read in sequence. The sequence roughly follows the stages of
ventral-stream visual processing, which forms the focus of the book. Within this
rough order, we placed closely related chapters together. We purposely interspersed
theoretical and experimental chapters, and, within the latter, animal electrode
recording and human fMRI studies. An overview of the chapters is given in fi gure
I.1 and table I.1 .
Localist and Distributed Codes
In chapter 1, Simon J. Thorpe reviews the debate about localist versus distributed
neuronal coding in the context of recent experimental evidence. Early fi ndings of
neuronal selectivity to simple features at low levels of the visual hierarchy and to
more complex features at higher levels suggested, by extrapolation, that there might
be neurons that respond selectively to particular objects, such as one’ s grandmother.
On a continuum of possible coding schemes from localist to distributed, this “ g rand-
mother cell” theory forms the localist pole. A code of grandmother cells could still
have multiple neurons devoted to each object; the key feature is the high selectivity
of the neurons. A grandmother-cell code is explicit in that no further processing is
2 Introduction
Figure I.1
Chapter overview. Along the vertical axis (arrow on the left), the chapters have been arranged roughly
according to the stage of processing they focus on. Horizontally, chapters with a stronger focus on a
particular stage of processing are closer to the axis on the left. Where possible, chapters related by other
criteria are grouped together. For example, chapters 5 and 6 use the method of voxel-receptive-fi eld
modeling, while chapters 9 and 11 – 14 use the method of representational similarity analysis. Neuron and
voxel icons label chapters using neuronal recordings and fMRI, respectively. Chapters focusing on theory,
experiment, or methods have been visually indicated (see legend), with methods chapters marked by a
gray underlay and experimental chapters with a strong methodological component marked by a partial
gray underlay.
required to read out the code and conclude that a particular object is present. At
the other end of the continuum is a distributed code, in which each neuron will
respond to many different objects; thus, there is no single neuron that unequivocally
indicates the presence of a particular object. In a distributed code, the information
is in the combination of active neurons.
F or a population of n neurons, a localist single-neuron code can represent no
more than n distinct objects, one for each neuron— a nd less if multiple neurons