Table Of ContentMatrix Algebra From a
Statistician's Perspective
David A. Harville
Springer
Preface
Matrix algebra plays a very important role in statistics and in many other disci-
plines.Inmanyareasofstatistics,ithasbecomeroutinetousematrixalgebrain
thepresentationandthederivationorverificationofresults.Onesuchareaislinear
statisticalmodels;anotherismultivariateanalysis.Intheseareas,aknowledgeof
matrixalgebraisneededinapplyingimportantconcepts,aswellasinstudyingthe
underlying theory, and is even needed to use various software packages (if they
aretobeusedwithconfidenceandcompetence).
On many occasions, I have taught graduate-level courses in linear statistical
models.Typically,theprerequisitesforsuchcoursesincludeanintroductory(un-
dergraduate) course in matrix (or linear) algebra. Also typically, the preparation
providedbythisprerequisitecourseisnotfullyadequate.Thereareseveralrea-
sonsforthis.Thelevelofabstractionorgeneralityinthematrix(orlinear)algebra
coursemayhavebeensohighthatitdidnotleadtoa“workingknowledge”ofthe
subject,or,attheotherextreme,thecoursemayhaveemphasizedcomputationsat
theexpenseoffundamentalconcepts.Further,thecontentofintroductorycourses
onmatrix(orlinear)algebravarieswidelyfrominstitutiontoinstitutionandfrom
instructortoinstructor.Topicssuchasquadraticforms,partitionedmatrices,and
generalized inverses that play an important role in the study of linear statistical
modelsmaybecoveredinadequatelyifatall.Anadditionaldifficultyisthatsev-
eral years may have elapsed between the completion of the prerequisite course
onmatrix(orlinear)algebraandthebeginningofthecourseonlinearstatistical
models.
Thisbookisaboutmatrixalgebra.Adistinguishingfeatureisthatthecontent,
theorderingoftopics,andthelevelofgeneralityareonesthatIconsiderappro-
priate for someone with an interest in linear statistical models and perhaps also
vi
forsomeonewithaninterestinanotherareaofstatisticsorinarelateddiscipline.
I have tried to keep the presentation at a level that is suitable for anyone who
has had an introductory course in matrix (or linear) algebra. In fact, the book is
essentiallyself-contained,anditishopedthatmuch,ifnotall,ofthematerialmay
becomprehensibletoadeterminedreaderwithrelativelylittlepreviousexposure
tomatrixalgebra.Tomakethematerialreadableforasbroadanaudienceaspos-
sible,Ihaveavoidedtheuseofabbreviationsandacronymsandhavesometimes
adoptedterminologyandnotationthatmayseemmoremeaningfulandfamiliarto
thenon-mathematicianthanthosefavoredbymathematicians.Proofsareprovided
foressentiallyalloftheresultsinthebook.Thebookincludesanumberofresults
and proofs that are not readily available from standard sources and many others
thatcanbefoundonlyinrelativelyhigh-levelbooksorinjournalarticles.
The book can be used as a companion to the textbook in a course on linear
statistical models or on a related topic through- it can be used to supplement
whateverresultsonmatricesmaybeincludedinthetextbookandasasourceof
proofs.And,itcanbeusedasaprimaryorsupplementarytextinasecondcourse
onmatrices,includingacoursedesignedtoenhancethepreparationofthestudents
foracourseorcoursesonlinearstatisticalmodelsand/orrelatedtopics.Aboveall,
itcanserveasaconvenientreferencebookforstatisticiansandforvariousother
professionals.
Whilethemotivationforthewritingofthebookcamefromthestatisticalapplica-
tionsofmatrixalgebra,thebookitselfdoesnotincludeanyappreciablediscussion
of statistical applications. It is assumed that the book is being read because the
readerisawareoftheapplications(oratleastofthepotentialforapplications)or
becausethematerialisofintrinsicinterestthrough-thisassumptionisconsistent
withtheusesdiscussedinthepreviousparagraph.(Inanycase,Ihavefoundthat
thediscussionsofapplicationsthataresometimesinterjectedintotreatisesonma-
trixalgebratendtobemeaningfulonlytothosewhoarealreadyknowledgeable
abouttheapplicationsandcanbemoreofadistractionthanahelp.)
Thebookhasanumberoffeaturesthatcombinetosetitapartfromthemore
traditionalbooksonmatrixalgebrathrough-italsodiffersinsignificantrespects
from those matrix-algebra books that share its (statistical) orientation, such as
the books of Searle (1982), Graybill (1983), and Basilevsky (1983). The cover-
ageisrestrictedtorealmatrices(i.e.,matriceswhoseelementsarerealnumbers)
through-complexmatrices(i.e.,matriceswhoseelementsarecomplexnumbers)
aretypicallynotencounteredinstatisticalapplications,andtheirexclusionleads
tosimplificationsinterminology,notation,andresults.Thecoverageincludeslin-
ear spaces, but only linear spaces whose members are (real) matrices through-
theinclusionoflinearspacesfacilitatesadeeperunderstandingofvariousmatrix
concepts(e.g.,rank)thatareveryrelevantinapplicationstolinearstatisticalmod-
els,whiletherestrictiontolinearspaceswhosemembersarematricesmakesthe
presentationmoreappropriatefortheintendedaudience.
The book features an extensive discussion of generalized inverses and makes
heavyuseofgeneralizedinversesinthediscussionofsuchstandardtopicsasthe
solutionoflinearsystemsandtherankofamatrix.Thediscussionofeigenvalues
vii
andeigenvectorsisdeferreduntilthenext-to-lastchapterofthebookthrough-I
havefounditunnecessarytouseresultsoneigenvaluesandeigenvectorsinteaching
a first course on linear statistical models and, in any case, find it aesthetically
displeasingtouseresultsoneigenvaluesandeigenvectorstoprovemoreelementary
matrixresults.And,thediscussionoflineartransformationsisdeferreduntilthe
verylastchapterthrough-inmoreadvancedpresentations,matricesareregarded
assubservienttolineartransformations.
The book provides rather extensive coverage of some nonstandard topics that
haveimportantapplicationsinstatisticsandinmanyotherdisciplines.Thesein-
cludematrixdifferentiation(Chapter15),thevecandvechoperators(Chapter16),
theminimizationofasecond-degreepolynomial(innvariables)subjecttolinear
constraints(Chapter19),andtheranks,determinants,andordinaryandgeneral-
izedinversesofpartitionedmatricesandofsumsofmatrices(Chapter18andparts
ofChapters8,9,1316,17,and19).Anattempthasbeenmadetowritethebook
insuchawaythatthepresentationiscoherentandnonredundantbut,atthesame
time,isconducivetousingthevariouspartsofthebookselectively.
With the obvious exception of certain of their parts, Chapters 12 through 22
(whichcompriseapproximatelythree-quartersofthebook’spages)canbereadin
arbitraryorder.TheorderingofChapters1through11(bothrelativetoeachother
andrelativetoChapters12through22)ismuchmorecritical.Nevertheless,even
Chapters 1 through 11 include sections or subsections that are prerequisites for
onlyasmallpartofthesubsequentmaterial.Moreoftenthannot,thelessessential
sectionsorsubsectionsaredeferreduntiltheendofthechapterorsection.
Thebookdoesnotaddressthecomputationalaspectsofmatrixalgebrainany
systematic way, however it does include descriptions and discussion of certain
computationalstrategiesandcoversanumberofresultsthatcanbeusefulindealing
withcomputationalissues.Matrixnormsarediscussed,butonlytoalimitedextent.
In particular, the coverage of matrix norms is restricted to those norms that are
definedintermsofinnerproducts.
Inwritingthebook,IwasinfluencedtoaconsiderableextentbyHalmos’s(1958)
bookonfinite-dimensionalvectorspaces,byMarsagliaandStyan’s(1974)paper
on ranks, by Henderson and Searle’s (1979, 1981b) papers on the vec and vech
operators,byMagnusandNeudecker’s(1988)bookonmatrixdifferentialcalculus,
andbyRaoandMitra’s(1971)bookongeneralizedinverses.And,Ibenefitedfrom
conversations with Oscar Kempthorne and from reading some notes (on linear
systems,determinants,matrices,andquadraticforms)thathehadpreparedfora
course(onlinearstatisticalmodels)atIowaStateUniversity.Ialsobenefitedfrom
reading the first two chapters (pertaining to linear algebra) of notes prepared by
JustusF.Seelyforacourse(onlinearstatisticalmodels)atOregonStateUniversity.
Thebookcontainsmanynumberedexercises.Theexercisesarelocatedat(or
near)theendsofthechaptersandaregroupedbysectionthrough-someexercises
mayrequiretheuseofresultscoveredinprevioussections,chapters,orexercises.
Manyoftheexercisesconsistofverifyingresultssupplementarytothoseincluded
inthebodyofthechapter.Bybreakingsomeofthemoredifficultexercisesinto
partsand/orbyprovidinghints,Ihaveattemptedtomakealloftheexercisesappro-
viii
priatefortheintendedaudience.Ihavepreparedsolutionstoalloftheexercises,
anditismyintentiontomakethemavailableonatleastalimitedbasis.
Theoriginandhistoricaldevelopmentofmanyoftheresultscoveredinthebook
are difficult (if not impossible) to discern, and I have not made any systematic
attempt to do so. However, each of Chapters 15 through 21 ends with a short
sectionentitledBibliographicandSupplementaryNotes.SourcesthatIhavedrawn
onmore-or-lessdirectlyforanextensiveamountofmaterialareidentifiedinthat
section.Sourcesthattracethehistoricaldevelopmentofvariousideas,results,and
terminologyarealsoidentified.And,forcertainofthesectionsinachapter,some
indicationmaybegivenofwhetherthatsectionisaprerequisiteforvariousother
sections(orviceversa).
Thebookisdividedinto(22)numberedchapters,thechaptersintonumbered
sections, and (in some cases) the sections into lettered subsections. Sections are
identified by two numbers (chapter and section within chapter) separated by a
decimalpointthrough-thus,thethirdsectionofChapter9isreferredtoasSection
9.3.Withinasection,asubsectionisreferredtobyletteralone.Asubsectionin
a different chapter or in a different section of the same chapter is referred to by
referringtothesectionandbyappendingalettertothesectionnumberthrough-
forexample,inSection9.3,SubsectionbofSection9.1isreferredtoasSection
9.1b.Anexerciseinadifferentchapterisreferredtobythenumberobtainedby
insertingthechapternumber(andadecimalpoint)infrontoftheexercisenumber.
Certainofthedisplayed“equations”arenumbered.Anequationnumbercom-
prises two parts (corresponding to section within chapter and equation within
section)separatedbyadecimalpoint(andisenclosedinparentheses).Anequa-
tion in a different chapter is referred to by the “number” obtained by starting
withthechapternumberandappendingadecimalpointandtheequationnumber
through-forexample,inChapter6,result(2.5)ofChapter5isreferredtoasresult
(5.2.5).Forpurposesofnumbering(andreferringto)equationsintheexercises,the
exercisesineachchapteraretoberegardedasformingSectionEofthatchapter.
Preliminaryworkonthebookdatesbacktothe1982–1983academicyear,which
IspentasavisitingprofessorintheDepartmentofMathematicsattheUniversity
of Texas at Austin (on a faculty improvement leave from my then position as a
professorofstatisticsatIowaStateUniversity).Theactualwritingbeganaftermy
returntoIowaStateandcontinuedonasporadicbasis(astimepermitted)untilmy
departureinDecember1995.Theworkwascompletedduringthefirstpartofmy
tenure in the Mathematical Sciences Department of the IBM Thomas J. Watson
ResearchCenter.
IamindebtedtoBettyFlehinger,EmmanuelYashchin,ClaudeGreengard,and
BillPulleyblank(allofwhomareorweremanagersattheResearchCenter)forthe
timeandsupporttheyprovidedforthisactivity.Themostvaluableofthatsupport
(byfar)cameintheformofthesecretarialhelpofPeggyCargiulo,whoentered
a
thelastsixchaptersofthebookinLT Xandwasofimmensehelpingettingthe
E
manuscript into final form. I am also indebted to Darlene Wicks (of Iowa State
a
University),whoenteredChapters1through16inLT X.
E
IwishtothankJohnKimmel,whohasbeenmyeditoratSpringer-Verlag.He
ix
has been everything an author could hope for. I also wish to thank Paul Nikolai
(formerlyoftheAirForceFlightDynamicsLaboratoryofWright-PattersonAir
Force Base, Ohio) and Dale Zimmerman (of the Department of Statistics and
ActuarialScienceoftheUniversityofIowa),whosecarefulreadingandmarking
ofthemanuscriptledtoanumberofcorrectionsandimprovements.Thesechanges
were in addition to ones stimulated by the earlier comments of two anonymous
reviewers(andbythecommentsoftheeditor).
