Mathematics For Machine Learning (MML) Official Solutions (Instructor's Solution Manual)

74 Pages·2020·0.819 MB·English

by  C. S. (Cheng Soon) Ong, M. P. (Marc Peter) Deisenroth, A. Aldo Faisal

2
Linear Algebra
Exercises
2.1 Weconsider(R\{−1},(cid:63)),where
a(cid:63)b:=ab+a+b, a,b∈R\{−1} (2.1)
a. Showthat(R\{−1},(cid:63))isanAbeliangroup.
b. Solve
3(cid:63)x(cid:63)x=15
intheAbeliangroup(R\{−1},(cid:63)),where(cid:63)isdefinedin(2.1).
a. First,weshowthatR\{−1}isclosedunder(cid:63):Foralla,b∈R\{−1}:
a(cid:63)b=ab+a+b+1−1=(a+1)(b+1)−1(cid:54)=−1
(cid:124) (cid:123)(cid:122) (cid:125)(cid:124) (cid:123)(cid:122) (cid:125)
(cid:54)=0 (cid:54)=0
⇒a(cid:63)b∈R\{−1}
Next,weshowthegroupaxioms
Associativity:Foralla,b,c∈R\{−1}:
(a(cid:63)b)(cid:63)c=(ab+a+b)(cid:63)c
=(ab+a+b)c+(ab+a+b)+c
=abc+ac+bc+ab+a+b+c
=a(bc+b+c)+a+(bc+b+c)
=a(cid:63)(bc+b+c)
=a(cid:63)(b(cid:63)c)
Commutativity:
∀a,b∈R\{−1}:a(cid:63)b=ab+a+b=ba+b+a=b(cid:63)a
NeutralElement:n=0istheneutralelementsince
∀a∈R\{−1}:a(cid:63)0=a=0(cid:63)a
InverseElement:Weneedtofinda¯,suchthata(cid:63)a¯=0=a¯(cid:63)a.
a¯(cid:63)a=0 ⇐⇒ a¯a+a+a¯=0
⇐⇒ a¯(a+1)=−a
a⇐(cid:54)=⇒−1 a¯=− a =−1+ 1 (cid:54)=−1∈R\{−1}
a+1 a+1
468
ThismaterialwillbepublishedbyCambridgeUniversityPressasMathematicsforMachineLearn-
ingbyMarcPeterDeisenroth,A.AldoFaisal,andChengSoonOng.Thispre-publicationversionis
freetoviewanddownloadforpersonaluseonly.Notforre-distribution,re-saleoruseinderiva-
tiveworks.(cid:13)cbyM.P.Deisenroth,A.A.Faisal,andC.S.Ong,2020.https://mml-book.com.
Exercises 469
b.
3(cid:63)x(cid:63)x=15 ⇐⇒ 3(cid:63)(x2+x+x)=15
⇐⇒ 3x2+6x+3+x2+2x=15
⇐⇒ 4x2+8x−12=0
⇐⇒ (x−1)(x+3)=0
⇐⇒ x∈{−3,1}
2.2 LetnbeinN\{0}.Letk,xbeinZ.Wedefinethecongruenceclassk¯ofthe
integerkastheset
k={x∈Z|x−k=0 (modn)}
={x∈Z|∃a∈Z: (x−k=n·a)}.
We now define Z/nZ (sometimes written Zn) as the set of all congruence
classesmodulon.Euclideandivisionimpliesthatthissetisafinitesetcon-
tainingnelements:
Zn ={0,1,...,n−1}
Foralla,b∈Zn,wedefine
a⊕b:=a+b
a. Showthat(Zn,⊕)isagroup.IsitAbelian?
b. Wenowdefineanotheroperation⊗forallaandbinZn as
a⊗b=a×b, (2.2)
wherea×brepresentstheusualmultiplicationinZ.
Letn=5.DrawthetimestableoftheelementsofZ \{0}under⊗,i.e.,
5
calculatetheproductsa⊗bforallaandbinZ \{0}.
5
Hence, show that Z \{0} is closed under ⊗ and possesses a neutral
5
element for ⊗. Display the inverse of all elements in Z \{0} under ⊗.
5
Concludethat(Z \{0},⊗)isanAbeliangroup.
5
c. Showthat(Z \{0},⊗)isnotagroup.
8
d. We recall that the B´ezout theorem states that two integers a and b are
relativelyprime(i.e.,gcd(a,b)=1)ifandonlyifthereexisttwointegers
uandv suchthatau+bv =1.Showthat(Zn\{0},⊗)isagroupifand
onlyifn∈N\{0}isprime.
a. Weshowthatthegroupaxiomsaresatisfied:
Closure:Leta,bbeinZn.Wehave:
a⊕b=a+b
=(a+b) mod n
by definition of the congruence class, and since [(a + b)modn] ∈
{0,...,n−1},itfollowsthata⊕b∈Zn.Thus,Zn isclosedunder⊕.
(cid:13)c2020M.P.Deisenroth,A.A.Faisal,C.S.Ong.TobepublishedbyCambridgeUniversityPress.
470 LinearAlgebra
Associativity:LetcbeinZn.Wehave:
(a⊕b)⊕c=(a+b)⊕c=(a+b)+c=a+(b+c)
=a⊕(b+c)=a⊕(b⊕c)
sothat⊕isassociative.
Neutralelement:Wehave
a+0=a+0=a=0+a
so0istheneutralelementfor⊕.
Inverseelement:Wehave
a+(−a)=a−a=0=(−a)+a
andweknowthat(−a)isequalto(−a)modnwhichbelongstoZn
andisthustheinverseofa.
Commutativity: Finally, the commutativity of (Zn,⊕) follows from
thatof(Z,+)sincewehave
a⊕b=a+b=b+a=b⊕a,
whichshowsthat(Zn,⊕)isanAbeliangroup.
b. LetuscalculatethetimestableofZ \{0}under⊗:
5
⊗ 1 2 3 4
1 1 2 3 4
2 2 4 1 3
3 3 1 4 2
4 4 3 2 1
WecannoticethatalltheproductsareinZ \{0},andthatinparticular,
5
noneofthemisequalto0.Thus,Z \{0}isclosedunder⊗.Theneutral
5
elementis1andwehave(1)−1 =1,(2)−1 =3,(3)−1 =2,and(4)−1 =4.
Associativityandcommutativityarestraightforwardand(Z \{0},⊗)is
5
anAbeliangroup.
c. Theelements2and4belongtoZ \{0},buttheirproduct2⊗4=8=0
8
doesnot.Thus,thissetisnotclosedunder⊗andisnotagroup.
d. Letusassumethatnisnotprimeandcanthusbewrittenasaproduct
n = a×b of two integers a and b in {2,...,n−1}. Both elements a
and b belong to Zn\{0} but their product a⊗b = n = 0 does not.
Thus,thissetisnotclosedunder⊗and(Zn\{0},⊗)isnotagroup.
Let n be a prime number. Let a and b be in Zn\{0} with a and b in
{1,...,n−1}.Asnisprime,weknowthataisrelativelyprimeton,
andsoisb.Letusthentakefourintegersu,v,u(cid:48) andv(cid:48),suchthat
au+nv=1
bu(cid:48)+nv(cid:48) =1.
Wethushavethat(au+nv)(bu(cid:48)+nv(cid:48))=1,whichwecanrewriteas
ab(uu(cid:48))+n(auv(cid:48)+vbu(cid:48)+nvv(cid:48))=1
Draft(2020-02-23)of“MathematicsforMachineLearning”.Feedback:https://mml-book.com.
Exercises 471
By virtue of the B´ezout theorem, this implies that ab and n are rela-
tively prime, which ensures that the product a⊗b is not equal to 0
andbelongstoZn\{0},whichisthusclosedunder⊗.
Theassociativityandcommutativityof⊗arestraightforward,butwe
need to show that every element has an inverse. First, the neutral
elementis1.LetusagainconsideranelementainZn\{0}withain
{1,...,n−1}. As a and n are coprime, the B´ezout theorem enables
ustodefinetwointegersuandvsuchthat
au+nv=1, (2.3)
whichimpliesthatau=1−nvandthus
au=1 mod n, (2.4)
whichmeansthata⊗u=au=1,orthatuistheinverseofa.Over-
all, (Zn\{0},⊗) is an Abelian group. Note that the B´ezout theorem
ensurestheexistenceofaninversewithoutyieldingitsexplicitvalue,
whichisthepurposeoftheextendedEuclideanalgorithm.
2.3 ConsiderthesetG of3×3matricesdefinedasfollows:
  (cid:12) 
G =10 x1 yz∈R3×3(cid:12)(cid:12)(cid:12)x,y,z∈R
 0 0 1 (cid:12)(cid:12) 
Wedefine·asthestandardmatrixmultiplication.
Is(G,·)agroup?Ifyes,isitAbelian?Justifyyouranswer.
Closure:Leta,b,c,x,yandz beinRandletusdefineAandB inG as
   
1 x z 1 a c
A=0 1 y , B=0 1 b .
0 0 1 0 0 1
Then,
    
1 x z 1 a c 1 a+x c+xb+z
A·B=0 1 y0 1 b=0 1 b+y  .
0 0 1 0 0 1 0 0 1
Sincea+x,b+y andc+xb+z areinRwehaveA·B ∈G.Thus,G is
closedundermatrixmultiplication.
Associativity:Letα,β andγ beinRandletC inG bedefinedas
 
1 α γ
C =0 1 β .
0 0 1
Itholdsthat
  
1 a+x c+xb+z 1 α γ
(A·B)·C =0 1 b+y 0 1 β
0 0 1 0 0 1
 
1 α+a+x γ+αβ+xβ+c+xb+z
=0 1 β+b+y  .
0 0 1
(cid:13)c2020M.P.Deisenroth,A.A.Faisal,C.S.Ong.TobepublishedbyCambridgeUniversityPress.
472 LinearAlgebra
Similarly,
   
1 x z 1 α+a γ+αβ+c
A·(B·C)=0 1 y·0 1 β+b 
0 0 1 0 0 1
 
1 α+a+x γ+αβ+xβ+c+xb+z
=0 1 β+b+y =(A·B)·C.
0 0 1
Therefore,·isassociative.
Neutralelement:ForallAinG,wehave:I ·A = A = A·I andthus
3 3
I istheneutralelement.
3
Non-commutativity: We show that · is not commutative. Consider the
matricesX,Y ∈G,where
   
1 1 0 1 0 0
X =0 1 0 , Y =0 1 1 . (2.5)
0 0 1 0 0 1
Then,
 
1 1 1
X·Y =0 1 1 ,
0 0 1
 
1 1 0
Y ·X =0 1 1(cid:54)=X·Y .
0 0 1
Therefore,·isnotcommutative.
Inverseelement:LetuslookforarightinverseA−1 ofA.Suchamatrix
r
should satisfy AA−1 = I . We thus solve the linear system [A|I ] that
r 3 3
wetransforminto[I |A−1]:
3 r
 
1 x z 1 0 0 −zR
3
 0 1 y 0 1 0  −yR3
0 0 1 0 0 1
 
1 x 0 1 0 −z −xR
2
(cid:32)  0 1 0 0 1 −y 
0 0 1 0 0 1
 
1 0 0 1 −x xy−z
(cid:32)  0 1 0 0 1 −y .
0 0 1 0 0 1
Therefore,weobtaintherightinverse
 
1 −x xy−z
A−r1 =0 1 −y ∈G
0 0 1
Because of the uniqueness of the inverse element, if a left inverse A−1
l
exists, then it is equal to the right inverse. But as · is not commutative,
Draft(2020-02-23)of“MathematicsforMachineLearning”.Feedback:https://mml-book.com.
Exercises 473
weneedtocheckmanuallythatwealsohaveAA−1 =I ,whichwedo
r 3
next:
   
1 −x −z+xy 1 x z
A−r1A=0 1 −y ·0 1 y
0 0 1 0 0 1
 
1 x−x z−xy−z+xy
=0 1 y+y =I3.
0 0 1
Thus, every element of G has an inverse. Overall, (G,·) is a non-Abelian
group.
2.4 Computethefollowingmatrixproducts,ifpossible:
a.
  
1 2 1 1 0
4 50 1 1
7 8 1 0 1
This matrix product is not defined. Highlight that the neighboring di-
mensionshavetofit(i.e.,m×nmatricesneedtobemultipliedbyn×p
(fromtheright)ork×mmatrices(fromtheleft).)
b.
    
1 2 3 1 1 0 4 3 5
4 5 60 1 1=10 9 11
7 8 9 1 0 1 16 15 17
c.
    
1 1 0 1 2 3 5 7 9
0 1 14 5 6=11 13 15
1 0 1 7 8 9 8 10 12
d.
 
0 3
(cid:20) (cid:21) (cid:20) (cid:21)
1 2 1 2 1 −1 14 6
 =
4 1 −1 −4 2 1  −21 2
5 2
e.
   
0 3 12 3 −3 −12
(cid:20) (cid:21)
1 −1 1 2 1 2 −3 1 2 6 
  = 
2 1  4 1 −1 −4  6 5 1 0 
5 2 13 12 3 2
2.5 Find the set S of all solutions in x of the following inhomogeneous linear
systemsAx=b,whereAandbaredefinedasfollows:
a.
   
1 1 −1 −1 1
2 5 −7 −5 −2
A=  , b= 
2 −1 1 3   4 
5 2 −4 2 6
(cid:13)c2020M.P.Deisenroth,A.A.Faisal,C.S.Ong.TobepublishedbyCambridgeUniversityPress.
474 LinearAlgebra
WeapplyGaussianeliminationtotheaugmentedmatrix
 
1 1 −1 −1 1
 2 5 −7 −5 −2 −2R1
 
 2 −1 1 3 4  −2R1
5 2 −4 2 6 −5R
1
 1 1 −1 −1 1  −1R
3 2
(cid:32)  0 3 −5 −3 −4 |· 13
 0 −3 3 5 2  +R2
0 −3 1 7 1 +R
2
 1 0 2 0 7   1 0 2 0 7 
3 3 3 3
(cid:32)  0 1 −53 −1 −43 (cid:32)  0 1 −53 −1 −43
 0 0 −2 2 −2  0 0 −2 2 −2
0 0 −4 4 −3 −2R 0 0 0 0 1
3
The last row of the final linear system shows that the equation system
hasnosolutionandthusS =∅.
b.
   
1 −1 0 0 1 3
 1 1 0 −3 0   6 
A=  , b= 
 2 −1 0 1 −1  5 
−1 2 0 −2 −1 −1
WestartagainbywritingdowntheaugmentedmatrixandapplyGaus-
sianeliminationtoobtainthereducedrowechelonform:
 
1 −1 0 0 1 3
 1 1 0 −3 0 6  −R1
 
 2 −1 0 1 −1 5  −2R1
−1 2 0 −2 −1 −1 +R
1
 
1 −1 0 0 1 3 +R
3
(cid:32)  0 2 0 −3 −1 3  −2R3
 
 0 1 0 1 −3 −1 swapwithR2
0 1 0 −2 0 2 −R
3
 
1 0 0 1 −2 2
(cid:32)  0 1 0 1 −3 −1
 
 0 0 0 −5 5 5  ·(−1)
5
0 0 0 −3 3 3 +5R
3 3
 
1 0 0 1 −2 2 −R
3
(cid:32)  0 1 0 1 −3 −1 −R3
 
 0 0 0 1 −1 −1
0 0 0 0 0 0
 
1 0 0 0 −1 3
(cid:32)  0 1 0 0 −2 0 
0 0 0 1 −1 −1
From the reduced row echelon form we apply the “Minus-1 Trick” in
ordertogetthefollowingsystem:
Draft(2020-02-23)of“MathematicsforMachineLearning”.Feedback:https://mml-book.com.
Exercises 475
 
1 0 0 0 −1 3
 0 1 0 0 −2 0 
 
 0 0 −1 0 0 0 
 
 0 0 0 1 −1 −1
0 0 0 0 −1 0
The right-hand side of the system yields us a particular solution while
the columns corresponding to the −1 pivots are the directions of the
solutionspace.Wethenobtainthesetofallpossiblesolutionsas
      (cid:12) 
S :=x∈R6 :x= 300 +λ1−001+λ2−−012(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)λ1,λ2 ∈R .
 −01  00  −−11(cid:12)(cid:12)(cid:12)(cid:12) 
2.6 Using Gaussian elimination, find all solutions of the inhomogeneous equa-
tionsystemAx=bwith
   
0 1 0 0 1 0 2
A=0 0 0 1 1 0 , b=−1 .
0 1 0 0 0 1 1
We start by determining the reduced row echelon form of the augmented
matrix[A|b].
 
0 1 0 0 1 0 2
 0 0 0 1 1 0 −1
0 1 0 0 0 1 1 −R
1
 
0 1 0 0 1 0 2 +R
3
(cid:32)  0 0 0 1 1 0 −1 +R3
0 0 0 0 −1 1 −1 ·(−1)
 
0 1 0 0 0 1 1
(cid:32)  0 0 0 1 0 1 −2
0 0 0 0 1 −1 1
This augmented matrix is now in reduced row echelon form. Applying the
“Minus-1trick”givesusthefollowingaugmentedmatrix:
−1 0 0 0 0 0 0 
 0 1 0 0 0 1 1 
 
 0 0 −1 0 0 0 0 
 
 0 0 0 1 0 1 −2
 
 0 0 0 0 1 −1 1 
0 0 0 0 0 −1 0
Theright-handsideofthisaugmentedmatrixgivesusaparticularsolution
whilethecolumnscorrespondingtothe−1pivotsarethespanthesolution
(cid:13)c2020M.P.Deisenroth,A.A.Faisal,C.S.Ong.TobepublishedbyCambridgeUniversityPress.
476 LinearAlgebra
spaceS.Therefore,weobtainthegeneralsolution
 (cid:12) 
S =−010102+(cid:124)λ1−000001+λ2(cid:123)−(cid:122)000001+λ3−−010111(cid:125)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)λ1,λ2,λ3 ∈R
solvesAx=0
Now,wefindaparticularsolutionthatsolvestheinhomogeneousequation
system. From the RREF, we see that x = 1,x = −2,x = 1+x and
2 4 5 6
x ,x ,x ∈ R are free variables (they correspond to variables that belong
1 3 6
tothenon-pivotcolumnsoftheaugmentedmatrix).Therefore,aparticular
solutionis
 0 
 1 
 
xp = 0 ∈R6.
−2
 
 1 
0
Thegeneralsolutionaddssolutionsfromthehomogeneousequationsystem
Ax=0.WecanusetheRREFoftheaugmentedsystemtoreadouttheseso-
lutionsbyusingtheMinus-1Trick.PaddingtheRREFwithrowscontaining
-1onthediagonalgivesus
−1 0 0 0 0 0 
 0 1 0 0 0 1 
 
 0 0 −1 0 0 0 
  .
 0 0 0 1 0 1 
 
 0 0 0 0 1 −1
0 0 0 0 0 −1
Thisyieldsthegeneralsolution
−1  0   0 
 0   0   1 
     
xp+λ1 00 +λ2−01+λ3 01 , λ1,λ2,λ3 ∈R.
     
 0   0  −1
0 0 −1
(cid:124) (cid:123)(cid:122) (cid:125)
solvesAx=0
 
x
1
2.7 Find all solutions in x = x2 ∈ R3 of the equation system Ax = 12x,
x
3
Draft(2020-02-23)of“MathematicsforMachineLearning”.Feedback:https://mml-book.com.
Exercises 477
where
 
6 4 3
A=6 0 9
0 8 0
and(cid:80)3 x =1.
i=1 i
We start by rephrasing the problem into solving a homogeneous system of
linear equations. Let x be in R3. We notice that Ax = 12x is equivalent
to (A−12I)x = 0, which can be rewritten as the homogeneous system
A˜x=0,wherewedefine
 
−6 4 3
A˜ = 6 −12 9  .
0 8 −12
Theconstraint(cid:80)3 x = 1canbetranscribedasafourthequation,which
i=1 i
leadsustoconsiderthefollowinglinearsystem,whichwebringtoreduced
rowechelonform:
 
−6 4 3 0 +R
2
 
 6 −12 9 0  ·1
  3
 
 0 8 −12 0  ·1
  4
1 1 1 1
 
0 −8 12 0 +4R
3
 
 2 −4 3 0  +2R
(cid:32)   3
 
 0 2 −3 0 
 
1 1 1 1
 
0 0 0 0
 
 2 0 −3 0  ·1
(cid:32)   2
 
 0 2 −3 0  ·1
  2
1 1 1 1
 
1 0 −3 0
2
(cid:32)  0 1 −3 0 
 2 
1 1 1 1 −R −R
1 2
   
1 0 −3 0 +(3)R 1 0 0 3
2 8 3 8
(cid:32)  0 1 −3 0  +(3)R (cid:32)  0 1 0 3 
 2  8 3  8 
0 0 4 1 ·1 0 0 1 1
4 4
Therefore,weobtaintheuniquesolution
3 3
8 1
x=3= 3 .
8 8
1 2
4
(cid:13)c2020M.P.Deisenroth,A.A.Faisal,C.S.Ong.TobepublishedbyCambridgeUniversityPress.