Table Of Content"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an1d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
SOLUTIONS FOR CHAPTER 1
Q.1.1 DMS with source probabilities : {0.30 0.25 0.20 0.15 0.10}
1
Entropy H(X) = ∑ p log
i p
i i
= 0.30 log 1/0.30 + 0.25 log 1/0.25 + ……………
= 2.228 bits
q
Q.1.2 Define D(p ⎜⎜q) = ∑ p log i (1)
i p
i i
p, q – probability distributions of discrete source X.
i i
q ⎛ q ⎞
D(p ⎜⎜q) = ∑ pi log pi ≤ ∑Pi ⎜⎜ pi −1⎟⎟ [using identity ln x ≤ x – 1]
i i i ⎝ i ⎠
= ∑(q − p ) = 0
i i
i
∴ D(p ⎜⎜q) ≥ 0
Put q = 1/n in (1) where n = cardinality of the distance source.
i
D(p ⎜⎜q) = ∑ p log p + ∑ p logn
i i i
i i
∑ p log p + logn = −H(X) + logn ≥ 0
i i
= i
−H(X) ≤ logn
H(X) = log n for uniform probability distribution. Hence proved that entropy of a
discrete source is maximum when output symbols are equally probable. The
quantity D(p ⎜⎜q) is called the Kullback-Leibler Distance.
Q. 1.3 The plots are given below:
1
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an2d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
3
2
y = x - 1
1
y = ln(x)
0
-1
-2
-3
0 0.5 1 1.5 2 2.5 3 3.5
Q 1.4 Consider two probability distributions: {p , p , , p } and {q , q , , q }.
0 1 K-1 0 1 K-1
K−1 ⎛ q ⎞ 1 K−1 ⎛ q ⎞
We have ∑ p log ⎜ k ⎟= ∑ p ln⎜ k ⎟. Use ln x ≤ 1 – x,
k 2⎜ p ⎟ ln2 k ⎜ p ⎟
k=0 ⎝ k ⎠ k=0 ⎝ k ⎠
K−1 ⎛ q ⎞ 1 K−1 ⎛ q ⎞
∑ p log ⎜ k ⎟≤ ∑ p ⎜ k −1⎟
k 2⎜ p ⎟ ln2 k⎜ p ⎟
k=0 ⎝ k ⎠ k=0 ⎝ k ⎠
1 K−1
≤ ∑(q − p )
ln2 k k
k=0
1 ⎛K−1 K−1 ⎞
≤ ⎜∑q −∑ p ⎟=0
ln2⎝ k k⎠
k=0 k=0
K−1 ⎛ q ⎞
Thus, ∑ p log ⎜ k ⎟≤0. (1)
k 2⎜ p ⎟
k=0 ⎝ k ⎠
n m P(x ,y )
Now, I(X; Y) =∑∑P(x ,y )log i j (2)
i j P(x )P(y )
i=1 j=1 i j
From (1) and (2) we can conclude (after basic manipulations) that I(X;Y) ≥ 0. The
equality holds if and only if P(x ,x )= P(x )P(x ), i.e., when the input and output
i j i j
symbols of the channel are statistically independent.
2
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an3d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
Q.1.5 Source X has infinitely large set of outputs P(x) = 2-i, i = 1, 2, 3, ………
i
∞ 1 ∞
H(X) = ∑ p(x ) log = ∑2−i log 2−i
i p(x )
i=1 i i=1
∞
= ∑i.2−i = 2bits
i=1
Q.1.6 Given: P(x) = p(1-p)i-1 i = 1, 2, 3, ……..
i
{ }
H(X) = −∑ p(1− p)i−1 log p(1− p)i−1
i
= −∑ p(1− p)i−1 {log p + (i−1) log(1− p)}
i
= − plog p∑ p(1− p)i−1− plog(1− p) ∑(i−1) (1− p)i−1
i=1 i
1 1− p
= − plog p × − plog(1− p)×
p p2
⎛ 1− p⎞ plog p − (1− p)log(1− p) 1
= −log p −⎜ ⎟ log(1− p) = = H(p) bits
⎜ ⎟
⎝ p ⎠ p p
Q 1.7 Hint: Same approach as the previous two problems.
Q 1.8 Yes it is uniquely decodable code because each symbol is coded uniquely.
Q 1.9 The relative entropy or Kullback Leibler distance between two probability mass
functions p(x) and q(x) is defined as
⎛ p(x)⎞
D(p||q)= ∑p(x)log⎜ ⎟. (1.76)
⎜ ⎟
⎝ q(x)⎠
x∈X
(i) Show that D(p||q) is non negative.
⎛ p(x)⎞ ⎛ q(x)⎞
Solution: −D(p||q)=−∑ p(x)log⎜ ⎟ = ∑ p(x)log⎜ ⎟
⎜ ⎟ ⎜ ⎟
⎝ q(x)⎠ ⎝ p(x)⎠
x∈X x∈X
q(x)
≤log∑ p(x) (from Jensen’s Inequality: Ef(X)≥ f(EX))
p(x)
x∈X
=log∑q(x) = log(1) = 0.
x∈X
Thus, −D(p||q)≤0 or D(p||q)≥0.
3
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an4d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
p(x)
(ii) D(p||q) = ∑ p(x)log
q(x)
Symmetry Property:
D(p||q) =D(q|| p)
p(x) q(x)
∑ p(x)log = ∑q(x)log
q(x) p(x)
∑ p(x)log p(x)−∑ p(x)logq(x) ≠ ∑q(x)logq(x)−∑q(x)log p(x)
Therefore Kullback Leibler distance does not follow symmetry property.
Triangle Inequality:
D(p||q)+D(q||r)≥D(p||r)
p(x) q(x) p(x)
∑ p(x)log +∑q(x)log ≥ ∑ p(x)log
q(x) r(x) r(x)
On solving this we get
q(x)
∑(−p(x)+q(x))log ≥ 0
r(x)
This relation does not hold if p(x) > q(x).
Therefore Kullback Leibler distance does not follow triangle inequality property.
(iii) I(X;Y) = ∑ p(x,y)I(x;y)
log(x,y)
= ∑ p(x,y)
log p(x)p(y)
= D(p(x,y)|| p(x)p(y))
Q.1.10
4
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an5d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
Q.1.11 The codeword lengths are possible if and only if they satisfy Kraft-McMillan
inequality, which in this case is
1 1 1 d
+ + + ≤1
23 23 23 28
d 5
≤
256 8
d ≤160
5
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an6d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
Q. 1.12 First note that it is a discrete random variable with a valid probability
∞ 1
distribution, since ∑P =∑ =1.
n Anlog2 n
n n=2
However, H(X) = ∑−P logP =+∞(the series does not converge!).
n n
n
Thus the entropy of a discrete random variable can also be infinite.
⎧a−1 0≤x≤a
Q.1.13 P(x) = ⎨
⎩ 0 otherwise
a
Differential entropy = −∫ p(x)log p(x)dx
0
a 1
= −∫ logadx = log a
a 2
0
The plot is given below. Note that the differential entropy can be negative.
H(X)
3
2
1 log a
2
0
-1
-2
-3
0 2 4 6 8 10
Q.1.14 DMS with source probabilities {0.35, 0.25, 0.20, 0.15, 0.05}
(i) Huffman code
6
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an7d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
(ii) R = ∑ p l = 0.35 × 2 + 0.25 × 2 + 0.20 × 2 + 0.15 × 3 + 0.05 × 3 = 2.2
i i
i
bits.
H(X) 1
(iii) η= H(X) = ∑ p log = 2.121
R i p
i
2.121
η= = 0.964 = 96.4%.
2.2
Q.1.15 (i)
1
2 1.00
S1 0.35 Codes
S1 2
3
S1 0.25 S2 3
S3 11
1
S1 0.20 S4 12
S5 13
2 0.40
S1 0.15
3
S1 0.05
(ii) R = ∑ p l = 0.35 + 0.25 + 0.20 × 2 + 0.15 × 2 + 0.05 × 2 = 1.40 ternary
i i
i
digits/ symbol
Q.1.16 (i) DMS with source probabilities :
Symbols Prob. Efficient fixed length code
S 0.20 000
1
S 0.20 001
2
S 0.15 010
3
S 0.15 011
4
S 0.10 100
5
S 0.10 101
6
S 0.05 110
7
S 0.05 111
8
7
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an8d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
L
Average code length = R = ∑n(x )P(x )= 3
k k
k=1
(ii) Huffman code
L
R = ∑n(x )P(x ) = 2.9 bits.
k k
k=1
n
(iii) The entropy H(X) = −∑P(x )logP(x ) = 2.8464 bits
i i
i=1
Huffman code gives shorter code length (η = 98.15%)
Q.1.17 Symbol Probability Self information Code
0
x 0.5 0.5 0
1
x 0.4 0.528 10
2 0 1
x 0.1 1 0.332 11
3
H(X) = 1.36 bits / symbol pair
R = 1.5 bits / symbol pair
1
η = 90.66 %
(ii)
8
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding an9d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
H(X) = 2.72 bits / symbol pair
R = 2.78 bits / symbol pair
2
η = 97.84 %
Symbol Self Huffman Probability
Triple Information Codes
x x x 0.375 000 0.125
1 1 1
x x x 0.3322 110 0.1
1 1 2
x x x 0.3322 100 0.1
1 2 1
x x x 0.3322 101 0.1
2 1 1
x x x 0.2915 0100 0.08
1 2 2
x x x 0.2915 0110 0.08
2 1 2
x x x 0.2915 0111 0.08
2 2 1
x x x 0.2538 0010 0.064
2 2 2
x x x 0.1330 11100 0.025
1 1 3
x x x 0.1330 11101 0.025
1 3 1
x x x 0.1330 010100 0.025
3 1 1
x x x 0.11287 010101 0.02
1 2 3
x x x 0.11287 010110 0.02
1 3 2
x x x 0.11287 010111 0.02
2 3 1
x x x 0.11287 001100 0.02
2 1 3
x x x 0.11287 001101 0.02
3 1 2
x x x 0.11287 001110 0.02
3 2 1
x x x 0.09545 111100 0.016
2 2 3
x x x 0.09545 111101 0.016
2 3 2
x x x 0.09545 111110 0.016
3 2 2
x x x 0.0382 1111110 0.005
1 3 3
x x x 0.0382 1111111 0.005
3 1 3
x x x 0.0382 00111110 0.005
3 3 1
x x x 0.03186 00111100 0.004
2 3 3
x x x 0.03186 00111101 0.004
3 2 3
x x x 0.03186 001111110 0.004
3 3 2
x x x 0.00996 001111111 0.001
3 3 3
9
"Copyrighted Material" -"Additional resource material supplied with the book Information, Theory, Coding a1n0d
Cryptography, Second Edition written by Ranjan Bose & published by McGraw-Hill Education (India) Pvt. Ltd. This
resource material is for Instructor's use only."
H(X) = 4.0826 bits / triple
R = 4.118 bits / symbol triple
2
H(X)
η = = 99.14%
R
3
Q.1.18 For a B-symbol block x x …… x ,
1 2 B
n n n
H (x x …… x ) = − ∑ ∑ − ∑ p(x x ..... x ) log p(x x ......x )
1 2 B j j j j j j
1 2 B 1 2 B
j=1 j =1 j =1
1 2 B
P(x x ….. x ) = p(x ) p(x ⎜x ) p(x ⎜x x ) …… p(x |x x …. x )
1 2 B 1 2 1 3 1 2 B 1 2 B-1
Assuming the B r, vs to be statistically independent
∴ H (x x …… x ) = H(x ) + H(x ) + ….. + H(x )
1 2 B 1 2 B
= BH(X)
Q 1.19 Hint: Apply equation 1.27.
Q.1.20 Lempel Ziv Code for
01001111100101000001010101100110000
Parsing 0,1,00,11,111,001,01,000,0010,10,101,100,110
gives
Dictionary Dictionary Location Contents Codeword
1 0001 0 00000
2 0010 1 00001
3 0011 00 00010
4 0100 11 00101
5 0101 111 01001
6 0110 001 00111
7 0111 01 00011
8 1000 000 00110
9 1001 0010 01100
10 1010 10 00100
11 1011 101 10101
12 1100 100 10100
13 1101 110 01000
Encoded stream
00000 00001 00010 00101 01001 00111 00011 00110 01100 00100
10101 10100 01000 00110
Q.1.21 (i) Lempel Ziv Code for
133002021113000022122233
Parsing 13,30,02,021,11,300,00,22,12,223
gives
Dictionary Dictionary Location Contents Codeword
10
