86-375/675 (15-387) Computational Perception

Carnegie Mellon University

Fall 2013. MWF 3:30-4:20 p.m. Mellon Institute 130.

Course Description

The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, or sheer perceptual power, perception in biology just works, and works in complex, ever changing environments, and can pick up the most subtle sensory patterns. Is it the neural hardware? Does biology solve fundamentally different problems? What can we learn from biological systems and human perception?
In this course, we will first study the biological and psychological data of biological perceptual systems in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. The course will explore four major themes in computational perception this year: 1) scene statistics, sensory and cortical representation, 2) probabilistic models and mechanisms of perception, 3) neural decoding, mental representation and perceptual synthesis,
4) perceptual science, computation and artistic expression.
You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course welcomes students from
neuroscience, psychology, arts, architecture, computer science and engineerings who are interested
in learning about how computations in the brain allow us to interpret and perceive the world,
and how the science and engineering of vision, biology and art can interact and inform one another
to foster artistic expression, engineering innovation and scientific understanding.

Classroom Etiquette

Please turn OFF your laptop, cell phones or any other electronic devices in the classroom.

Grading Scheme

Evaluation

% of Grade

Homework/Quizzes

50

Programming Problem sets

50

Term Project (685)

30

Grading scheme: A: > 88%, B: > 75%. C: > 65%.

This year's Syllabus

Date

Lecture Topic

Relevant Readings

Assignments

BIOLOGY OF PERCEPTION

MWF 8/26 week

Perception and Illusion

FS. Ch 1

MWF 9/2

Philosophy, History and the Senses

FS. Ch 1, Marr, Ch 1

MWF 9/9

Retinal computation and tunings

FS. Ch 6, Meister, Masland reviews

MWF 9/16

Linear System, Fourier Transform, Pyramid

FS ch 5, 6 , Abbott and Dayan Ch 1 and 2

MWF 9/23

Striate and Extrastriate cortex

ch 9,10

M 9/30

Simple cell and image representation

Abbott and Dayan chapter 1 and 2

F 10/4

Guest Lecture: Art and Perception

handout

&nbsp

MODELS OF PERCEPTION

MWF 10/7

Efficient and sparse codes

handouts (Olshausen)

M 10/14

Midterm Evaluation

WF 10/16

Mid-Semester Break

MW 10/21

Models Lightness perception (retinex,intrinsic)

FS ch 16, Land's papers

F 10/25

Guest Lecture: Electric Fish perception.

Papers

M 11/4

Shape from shading

Papers

< tr>

W 11/6

Binocular Stereo

FS ch 18,19

< tr>

M 11/11

Perceptual organization

FS ch 8, 13

< tr>

WF 11/13

Texture Perception

FS ch 2

< tr>

MWF 11/18

Motion recognition

FS ch 14,15

< tr>

MWF 11/25

Object recognition

FS ch 8

< tr>

MWF 12/2

Scene, Context and Attention

Papers

F 12/13

FINAL EXAM (1:00-4:00)

Some additional attention will be allocated to the following topics:

Topic 1: Scene statistics, sensory and cortical representation

To undersatnd perception, we must understand the natural environments which shape our brain and our perceptual computational machinery. Central to
to understanding the neural basis of perceptual inference from a Bayesian perspective
is understanding how the statistical
regularities in natural scenes are encoded in cortical representation to serve as priors in
the inference process. Natural images however are enormously complex and maybe best expressed
in hierarchical forms. Thus, a major challenge in computational vision is to understand the
basic vocabulary of images, and the computational rules with which elementary components can be
composed to form successive compositional structures to encode the hierarchical priors of natural
scenes. We will explore statistical models of images, as well as compositional models such as
DBN (Deep belief net) and RCM (Recursive compositional models) for learning the hierarchical
language of vision. We will explore
how these hierarchical scene priors are encoded in neural tunings and neural
connectivities to faciliate perceptual inference.

Topic 2: Probabilistic models and algorithms of perception

While perception has been popularly formulated in terms of Bayesian inference in the theoretical
level, little is known about the computational algorithms and implementation of perceptual
inference.
We will explore mechanistic and normative models for motion, binocular stereo, texture, surface and
contour perception, perceptual organization
and hierarchical models for object recognition, drawing knowledge from
works in computer vision and computational neural models.
We will study a number of algorithms that have been effective in computer vision
for performing learning and inference, including gradient descent, particle and Kalman filtering,
MCMC sampling and mean field approximation, and explore the links between observed neural dynamics
and these inference algorithms. We will explore various theoretical frameworks on
how perceptual representations are encoded
and represented in neuronal ensembles, including the issue of population codes, synchrony and
binding.

With an understanding of cortical representation and neural mechanisms for perceptual inference,
we can begin to explore how neural decoding and neural simulation technology can be
coupled with large-scale multi-electrode array to decode mental images in our brain
as well as to generate perceptual representation in
the brain by electrical stimulation.
There are over 40 million blind individuals in the world. A variety of invasive and
noninvasive procedures have emerged
over the years to use electrical stimulation to "restore" or create vision, ranging
from retinal implant to electrical stimulation
in LGN and stimulation of the visual cortex. We will investigate how V1 and the extrastriate
cortex can represent mental images and precepts individually and together, both in terms
of theories, models and neural evidence.
We will study literature of artificial vision in human and animal models and explore
paradigms for the development of visual prosthesis by integrating computer
vision, electrical recording and stimulation technology.

Topic 4: Perception, computation and art

Visual perception and artistic expression are deeply connected at many levels. In fact, visual
perception in the brain might involves both analysis and synthesis. That is, our perception
is not simply analyzing what is out there, but an active synthesis of an internal mental
representation of what is out there, sometimes leading to illusion and hallucination. We
will explore this synthesis process and how it might be tied to aesthetics and art making.
The integration of
visual art and the experimental study of vision has its roots in formal analysis of paintings.
Advances in our understanding of how our brain or perception works have lead to resurgence of interests in
linking art with vision science. Here, we will explore some of the new links between neuroscience,
computational vision and the art, with a view to enrich our understanding and making of arts --
how artistic expression is rooted in
perceptual computation and how scientific understanding of vision have transformed arts
over the centuries.