Arti cial Neural Networks

31 Pages·2005·0.39 MB·English

Arti(cid:12)cial Neural Networks
[Read Ch. 4]
[Recommended exercises 4.1, 4.2, 4.5, 4.9, 4.11]
(cid:15) Threshold units
(cid:15) Gradient descent
(cid:15) Multilayer networks
(cid:15) Backpropagation
(cid:15) Hidden layer representations
(cid:15) Example: Face Recognition
(cid:15) Advanced topics
74 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
Connectionist Models
Consider humans:
(cid:15) Neuron switching time ~ :001 second
10
(cid:15) Number of neurons ~ 10
4(cid:0)5
(cid:15) Connections per neuron ~ 10
(cid:15) Scene recognition time ~ :1 second
(cid:15) 100 inference steps doesn't seem like enough
! much parallel computation
Properties of arti(cid:12)cial neural nets (ANN's):
(cid:15) Many neuron-like threshold switching units
(cid:15) Many weighted interconnections among units
(cid:15) Highly parallel, distributed process
(cid:15) Emphasis on tuning weights automatically
75 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
When to Consider Neural Networks
(cid:15) Input is high-dimensional discrete or real-valued
(e.g. raw sensor input)
(cid:15) Output is discrete or real valued
(cid:15) Output is a vector of values
(cid:15) Possibly noisy data
(cid:15) Form of target function is unknown
(cid:15) Human readability of result is unimportant
Examples:
(cid:15) Speech phoneme recognition [Waibel]
(cid:15) Image classi(cid:12)cation [Kanade, Baluja, Rowley]
(cid:15) Financial prediction
76 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
ALVINN drives 70 mph on highways
Sharp Straight Sharp
Left Ahead Right
30 Output
Units
4 Hidden
Units
30x32 Sensor
Input Retina
77 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
Perceptron
x1 w1 x0=1
w
x w 0
2 2
Σ
.
AA
. n
Σ n
w x { Σ
. i i 1 if w x > 0
w i=0 o = i=0 i i
n
-1 otherwise
x
n
8
>>
< 1 if w + w x + (cid:1) (cid:1) (cid:1) + w x > 0
0 1 1 n n
o(x ; : : : ; x ) =
1 n >>: (cid:0)1 otherwise.
Sometimes we'll use simpler vector notation:
8
>>
< 1 if w~ (cid:1) ~x > 0
o(~x) =
>>: (cid:0)1 otherwise.
78 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
Decision Surface of a Perceptron
x x
2 2
+
+
- + -
+
x x
1 1
- - +
-
(a) (b)
Represents some useful functions
(cid:15) What weights represent
g(x ; x ) = AND(x ; x )?
1 2 1 2
But some functions not representable
(cid:15) e.g., not linearly separable
(cid:15) Therefore, we'll want networks of these...
79 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
Perceptron training rule
w w + (cid:1)w
i i i
where
(cid:1)w = (cid:17)(t (cid:0) o)x
i i
Where:
(cid:15) t = c(~x) is target value
(cid:15) o is perceptron output
(cid:15) (cid:17) is small constant (e.g., .1) called learning rate
80 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
Perceptron training rule
Can prove it will converge
(cid:15) If training data is linearly separable
(cid:15) and (cid:17) su(cid:14)ciently small
81 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
Gradient Descent
To understand, consider simpler linear unit, where
o = w0 + w1x1 + (cid:1) (cid:1) (cid:1) + wnxn
Let's learn w 's that minimize the squared error
i
1 X
2
E[w~] (cid:17) (td (cid:0) od)
2 d2D
Where D is set of training examples
82 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997
Gradient Descent
25
20
15
w]
E[
10
5
0
2
1
-2
-1
0 0
1
2
-1 3
w0 w1
Gradient
2 3
@E @E @E
6 7
rE[w~] (cid:17) 4 ; ; (cid:1) (cid:1) (cid:1) 5
@w @w @w
0 1 n
Training rule:
(cid:1)w~ = (cid:0)(cid:17)rE[w~]
i.e.,
@E
(cid:1)w = (cid:0)(cid:17)
i
@w
i
83 lecture slides for textbook Machine Learning,T. Mitchell, McGraw Hill, 1997