Introduction to Deep Learning - Neural Engineering

54 Pages·2014·8.7 MB·English

by  Anil Bharath

A
 Shallow
 Introduc0on
 to
 Deep

Learning

Anil
 Anthony
 Bharath

[email protected]

This
 Version:
 23-­‐Jan-­‐2014

Imperial
 College
 London
 &
 Cortexica
 Vision
 Systems
Disclaimer

•  These
 slides
 were
 created
 with
 the
 purpose
 of
 explaining

biological
 recep0ve
 fields
 and
 the
 manner
 in
 which
 similar

spa0al
 paOerns
 can
 be
 “learned”
 in
 a
 DL
 context.

 These

slides
 are
 not
 intended
 to
 be
 a
 full
 review
 or
 exposi0on
 on

DL.

•  These
 are
 first-­‐genera0on
 notes;
 although
 many
 elements

have
 been
 presented
 previously,
 the
 order
 of
 presenta0on

is
 new.

•  I
 welcome
 feedback
 on
 how
 this
 lecture
 might
 be

improved:
 I
 may
 not
 agree,
 but
 I
 do
 not
 bite.

•  An
 accompanying
 bibliography
 in
 DL
 is
 available
 from
 Woo-­‐
Sup
 Han
 (BICV
 Group).
Deep
 Learning:
 Defini0on

•  An
 unambiguous
 defini0on
 is
 debatable,
 but

some
 characteris0cs
 are
 clearly
 iden0fiable:

–  A
 Deep
 Learning
 (DL)
 system
 contains
 mul0ple
 hidden

layers
 of
 network
 between
 input
 and
 output
 layers;

–  DL
 systems
 are
 reminiscent
 of
 mul0-­‐layer

perceptrons,
 BUT:

•  DL
 systems
 are
 oZen
 trained
 by
 exploi0ng
 both
 supervised

and
 unsupervised
 training
 in
 the
 same
 network;

•  DL
 systems
 may
 mixed
 architectures
 from
 layer
 to
 layer,
 that

oZen
 conform
 to
 hybrid
 probabilis0c
 models;

•  Training
 regimes
 may
 differ
 depending
 on
 layer,
 reflec0ng
 a

balance
 between
 the
 need
 to
 represent
 and
 to
 classify
 data.
Acronyms

•  DLN:
 Deep
 Learning
 Network

•  DL:
 Deep
 Learning
 (refers
 to
 the
 field
 of
 study)

•  DBN:
 Deep
 Belief
 Network

•  DBM:
 Deep
 Boltzmann
 Machine

•  RBM:
 Restricted
 Boltzmann
 Machine

•  CNN:
 Convolu0onal
 Neural
 Network

•  HTM:
 Hierarchical
 Temporal
 Memory

OZen
 not

Included

•  MSC:
 Map
 Seeking
 Circuit
  Architecture
 is

In
 DL
 literature

mul0-­‐layered,

but
 training
 is

not
 in
 the

usual
 manner
In
 order
 to
 appreciate
 DLNs…

•  We’ll
 look
 first
 at
 two
 examples
 of
 sensory

processing
 networks;

•  One
 is
 technological,
 the
 other
 is
 biological;

•  Both
 handle
 visual
 informa0on.
Ramón
 y
 Cajal’s
 Re0na

Mammalian
 re0nal
 structure
 –
 note

the
 apparent
 layers
 that
 are

very
 clearly
 depicted.

 Different
 anatomical

Layers
 are
 associated
 with
 different

func’onal
 proper0es
 of
 cells.

The
 layered
 structure
 is
 typical
 of
 neural

circuitry
 involved
 in
 sensory
 percep0on.

c.1900
 By
 San’ago
 Ramón
 y
 Cajal.

c.

1896
The
 Complexity
 of
 Vision

Some
 “sta0s0cs”
 on
 the
 human
 re0na:

–  120
 million
 photoreceptors
 in
 each
 human
 re0na.

–  Approximately
 5
 dis0nct
 layers
 of
 cells.

–  Various
 areas
 of
 the
 re0na:

•  Central
 Fovea

•  Fovea

•  Periphery

–  Varia0on
 of
 proper0es/ra0os
 of
 photoreceptors/ganglion

cells
 across
 the
 re0na.

7
A
 “Handle”
 to
 Complexity

The
 Recep0ve
 Field:

–  A
 “nearly
 definable”*
 concept
 to
 describe
 the
 effect
 that
 a

visual
 s0mulus
 has
 on
 a
 ganglion
 cell
 exhibi0ng
 centre

surround
 organisa0on.

•  Change
 from
 sustained
 to
 transient
 centre
 response
 as

 s’muli
 are

displaced
 from
 centre.

•  Excitatory
 responses
 evoked
 to
 opposite
 s’mulus
 phase
 with

surround
 s’mula’on.

•  Ac’ve
 inhibi’on
 of
 the
 centre
 response
 with
 surround
 s’mula’on.

•  A
 maximal
 response
 can
 be
 evoked
 only
 with
 an
 op’mally
 sized

spot.

*The
 term
 “nearly
 definable”
 is
 aOributed
 to
 Hartline.

8
Recep0ve
 fields

•  Spike-­‐triggered
 averaging
 (STA)
 –
 also
 known

as
 reverse
 “correla0on”:
 a
 probabilis0c

characterisa0on
 of
 recep0ve
 fields.

•  Run
 through
 a
 random
 series
 of
 spa0al

paOerns,
 and
 empirically
 es0mate:

–  P(spike
 |
 spa0al
 paOern
 s0mulus;
 t)

•  Provides
 a
 systema0c
 way
 of
 capturing

response
 characteris0cs
 of
 cells.
The
 “Op0mal”
 S0mulus

•  A
 visual
 Interpreta0on
 of
 a
 re0nal
 recep0ve
 Field

–  The
 “op0mal
 visual
 s0muli”:
 probability
 of
 firing
 of

two
 classes
 of
 re0nal
 ganglion
 cells
 might
 be
 highest

for
 these
 s0muli*.

–  Examples:

OFF
 Centre
  ON
 Centre

*This
 is
 a
 simplified,
 but
 “on
 average”
 response.

10