Lifelong Machine Learning [2nd ed.]

190 Pages·2018·1.353 MB·english

by  Zhiyuan Chen, Bing Liu

Lifelong Machine Learning
Second Edition
Zhiyuan Chen
Google,Inc.
Bing Liu
UniversityofIllinoisatChicago
SYNTHESISLECTURESONARTIFICIALINTELLIGENCE
AND MACHINELEARNING#38
M
&C Morgan&cLaypool publishers
Copyright©2018byMorgan&Claypool
LifelongMachineLearning,SecondEdition
ZhiyuanChenandBingLiu
www.morganclaypool.com
ISBN:9781681733029 paperback
ISBN:9781681733036 ebook
ISBN:9781681733999 epub
ISBN:9781681733043 hardcover
DOI10.2200/S00832ED1V01Y201802AIM037
APublicationintheMorgan&ClaypoolPublishersseries
SYNTHESISLECTURESONARTIFICIALINTELLIGENCE
ANDMACHINELEARNING
Lecture#38
SeriesEditors:RonaldJ.Brachman,JacobsTechnion-CornellInstituteatCornellTech
PeterStone,UniversityofTexasatAustin
SeriesISSN
Print1939-4608 Electronic1939-4616
ABSTRACT
LifelongMachineLearning,SecondEdition is an introduction to an advanced machine learning
paradigm that continuously learns by accumulating past knowledge that it then uses in future
learning and problem solving. In contrast, the current dominant machine learning paradigm
learnsinisolation:givenatrainingdataset,itrunsamachinelearningalgorithmonthedataset
toproduceamodelthatisthenusedinitsintendedapplication.Itmakesnoattempttoretainthe
learnedknowledgeanduseitinsubsequentlearning.Unlikethisisolatedsystem,humanslearn
effectively with only a few examples precisely because our learning is very knowledge-driven:
the knowledge learned in the past helps us learn new things with little data or effort. Lifelong
learningaimstoemulatethiscapability,becausewithoutit,anAIsystemcannotbeconsidered
trulyintelligent.
Researchinlifelonglearninghasdevelopedsignificantlyintherelativelyshorttimesince
thefirsteditionofthisbookwaspublished.Thepurposeofthissecondeditionistoexpandthe
definition of lifelong learning, update the content of several chapters, and add a new chapter
aboutcontinuallearningindeepneuralnetworks—whichhasbeenactivelyresearchedoverthe
pasttwoorthreeyears.Afewchaptershavealsobeenreorganizedtomakeeachofthemmore
coherent for the reader. Moreover, the authors want to propose a unified framework for the
research area. Currently, there are several research topics in machine learning that are closely
related to lifelong learning—most notably, multi-task learning, transfer learning, and meta-
learning—becausetheyalsoemploytheideaofknowledgesharingandtransfer.Thisbookbrings
allthesetopicsunderoneroofanddiscussestheirsimilaritiesanddifferences.Itsgoalistointro-
ducethisemergingmachinelearningparadigmandpresentacomprehensivesurveyandreview
oftheimportantresearchresultsandlatestideasinthearea.Thisbookisthussuitableforstu-
dents,researchers,andpractitionerswhoareinterestedinmachinelearning,datamining,natural
language processing, or pattern recognition. Lecturers can readily use the book for courses in
anyoftheserelatedfields.
KEYWORDS
lifelong machine learning; lifelong learning; continuous learning; continual learn-
ing;meta-learning,never-endinglearning;multi-tasklearning;transferlearning
Contents
Preface ..........................................................xvii
1 Introduction .......................................................1
1.1 ClassicMachineLearningParadigm.................................. 1
1.2 MotivatingExamples.............................................. 3
1.3 ABriefHistoryofLifelongLearning ................................. 6
1.4 DefinitionofLifelongLearning ..................................... 9
1.5 TypesofKnowledgeandKeyChallenges ............................. 14
1.6 EvaluationMethodologyandRoleofBigData ........................ 16
1.7 OutlineoftheBook .............................................. 18
2 RelatedLearningParadigms .........................................21
2.1 TransferLearning................................................ 21
2.1.1 StructuralCorrespondenceLearning ........................... 22
2.1.2 NaïveBayesTransferClassifier................................ 23
2.1.3 DeepLearninginTransferLearning ........................... 24
2.1.4 DifferencefromLifelongLearning ............................ 25
2.2 Multi-TaskLearning ............................................. 26
2.2.1 TaskRelatednessinMulti-TaskLearning ....................... 26
2.2.2 GO-MTL:Multi-TaskLearningusingLatentBasis .............. 27
2.2.3 DeepLearninginMulti-TaskLearning......................... 29
2.2.4 DifferencefromLifelongLearning ............................ 30
2.3 OnlineLearning................................................. 31
2.3.1 DifferencefromLifelongLearning ............................ 31
2.4 ReinforcementLearning .......................................... 32
2.4.1 DifferencefromLifelongLearning ............................ 33
2.5 MetaLearning .................................................. 33
2.5.1 DifferencefromLifelongLearning ............................ 34
2.6 Summary ...................................................... 34
3 LifelongSupervisedLearning........................................35
3.1 DefinitionandOverview .......................................... 36
3.2 LifelongMemory-BasedLearning .................................. 37
3.2.1 TwoMemory-BasedLearningMethods ........................ 37
3.2.2 LearningaNewRepresentationforLifelongLearning ............ 37
3.3 LifelongNeuralNetworks ......................................... 38
3.3.1 MTLNet ................................................ 38
3.3.2 LifelongEBNN ........................................... 39
3.4 ELLA:AnEfficientLifelongLearningAlgorithm ..................... 40
3.4.1 ProblemSetting............................................ 41
3.4.2 ObjectiveFunction ......................................... 41
3.4.3 DealingwiththeFirstInefficiency ............................. 42
3.4.4 DealingwiththeSecondInefficiency........................... 44
3.4.5 ActiveTaskSelection ....................................... 45
3.5 LifelongNaiveBayesianClassification ............................... 46
3.5.1 NaïveBayesianTextClassification ............................. 46
3.5.2 BasicIdeasofLSC ......................................... 48
3.5.3 LSCTechnique............................................ 49
3.5.4 Discussions ............................................... 50
3.6 DomainWordEmbeddingviaMeta-Learning ........................ 51
3.7 SummaryandEvaluationDatasets .................................. 53
4 ContinualLearningandCatastrophicForgetting........................55
4.1 CatastrophicForgetting ........................................... 55
4.2 ContinualLearninginNeuralNetworks.............................. 57
4.3 LearningwithoutForgetting ....................................... 59
4.4 ProgressiveNeuralNetworks ....................................... 61
4.5 ElasticWeightConsolidation ...................................... 62
4.6 iCaRL:IncrementalClassifierandRepresentationLearning.............. 64
4.6.1 IncrementalTraining........................................ 64
4.6.2 UpdatingRepresentation .................................... 65
4.6.3 ConstructingExemplarSetsforNewClasses .................... 66
4.6.4 PerformingClassificationiniCaRL............................ 67
4.7 ExpertGate .................................................... 67
4.7.1 AutoencoderGate.......................................... 68
4.7.2 MeasuringTaskRelatednessforTraining ....................... 69
4.7.3 SelectingtheMostRelevantExpertforTesting .................. 69
4.7.4 Encoder-BasedLifelongLearning............................. 70
4.8 ContinualLearningwithGenerativeReplay .......................... 70
4.8.1 GenerativeAdversarialNetworks.............................. 70
4.8.2 GenerativeReplay .......................................... 71
4.9 EvaluatingCatastrophicForgetting.................................. 72
4.10 SummaryandEvaluationDatasets .................................. 73
5 Open-WorldLearning..............................................77
5.1 ProblemDefinitionandApplications ................................ 78
5.2 Center-BasedSimilaritySpaceLearning ............................. 79
5.2.1 IncrementallyUpdatingaCBSLearningModel.................. 79
5.2.2 TestingaCBSLearningModel ............................... 81
5.2.3 CBSLearningforUnseenClassDetection ...................... 82
5.3 DOC:DeepOpenClassification ................................... 85
5.3.1 Feed-ForwardLayersandthe1-vs.-RestLayer................... 85
5.3.2 ReducingOpen-SpaceRisk .................................. 86
5.3.3 DOCforImageClassification ................................ 88
5.3.4 UnseenClassDiscovery ..................................... 88
5.4 SummaryandEvaluationDatasets .................................. 89
6 LifelongTopicModeling............................................91
6.1 MainIdeasofLifelongTopicModeling .............................. 91
6.2 LTM:ALifelongTopicModel .................................... 94
6.2.1 LTMModel .............................................. 95
6.2.2 TopicKnowledgeMining.................................... 96
6.2.3 IncorporatingPastKnowledge ................................ 97
6.2.4 ConditionalDistributionofGibbsSampler...................... 99
6.3 AMC:ALifelongTopicModelforSmallData....................... 100
6.3.1 OverallAlgorithmofAMC ................................. 100
6.3.2 MiningMust-linkKnowledge ............................... 101
6.3.3 MiningCannot-linkKnowledge ............................. 103
6.3.4 ExtendedPólyaUrnModel ................................. 104
6.3.5 SamplingDistributionsinGibbsSampler ...................... 106
6.4 SummaryandEvaluationDatasets ................................. 108
7 LifelongInformationExtraction ....................................111
7.1 NELL:ANever-EndingLanguageLearner ......................... 111
7.1.1 NELLArchitecture........................................ 114
7.1.2 ExtractorsandLearninginNELL............................ 114
7.1.3 CouplingConstraintsinNELL .............................. 117
7.2 LifelongOpinionTargetExtraction ................................ 117
7.2.1 LifelongLearningthroughRecommendation ................... 118
7.2.2 AERAlgorithm .......................................... 119
7.2.3 KnowledgeLearning....................................... 120
7.2.4 RecommendationusingPastKnowledge ....................... 121
7.3 LearningontheJob ............................................. 123
7.3.1 ConditionalRandomFields ................................. 123
7.3.2 GeneralDependencyFeature ................................ 124
7.3.3 TheL-CRFAlgorithm ..................................... 126
7.4 Lifelong-RL:LifelongRelaxationLabeling .......................... 127
7.4.1 RelaxationLabeling ....................................... 127
7.4.2 LifelongRelaxationLabeling ................................ 128
7.5 SummaryandEvaluationDatasets ................................. 129
8 ContinuousKnowledgeLearninginChatbots .........................131
8.1 LiLi:LifelongInteractiveLearningandInference ..................... 132
8.2 BasicIdeasofLiLi .............................................. 134
8.3 ComponentsofLiLi ............................................ 136
8.4 ARunningExample ............................................ 137
8.5 SummaryandEvaluationDatasets ................................. 138
9 LifelongReinforcementLearning ...................................139
9.1 LifelongReinforcementLearningthroughMultipleEnvironments ....... 141
9.1.1 AcquiringandIncorporatingBias ............................ 141
9.2 HierarchicalBayesianLifelongReinforcementLearning................ 142
9.2.1 Motivation............................................... 142
9.2.2 HierarchicalBayesianApproach.............................. 143
9.2.3 MTRLAlgorithm......................................... 143
9.2.4 UpdatingHierarchicalModelParameters ...................... 144
9.2.5 SamplinganMDP ........................................ 146
9.3 PG-ELLA:LifelongPolicyGradientReinforcementLearning .......... 146
9.3.1 PolicyGradientReinforcementLearning ...................... 147
9.3.2 PolicyGradientLifelongLearningSetting ..................... 148
9.3.3 ObjectiveFunctionandOptimization ......................... 149
9.3.4 SafePolicySearchforLifelongLearning....................... 150
9.3.5 Cross-domainLifelongReinforcementLearning ................ 151
9.4 SummaryandEvaluationDatasets ................................. 152
10 ConclusionandFutureDirections ...................................153
Bibliography .....................................................159
Preface
The purpose of writing this second edition is to extend the definition of lifelong learning, to
update the content of several chapters, and to add a new chapter about continual learning in
deepneuralnetworks, which has been actively researched for the past two to three years. A few
chaptersarealsoreorganizedtomakeeachofthemmorecoherent.
The project of writing this book started with a tutorial on lifelong machine learning that
we gave at the 24th International Joint Conference on Artificial Intelligence (IJCAI) in 2015.
At that time, we had worked on the topic for a while and published several papers in ICML,
KDD, and ACL. When Morgan & Claypool Publishers contacted us about the possibility of
developing a book on the topic, we were excited. We strongly believe that lifelong machine
learning (or simply lifelong learning) is very important for the future of machine learning and
artificialintelligence(AI).Notethatlifelonglearningissometimesalsocalledcontinuallearning
orcontinuouslearningintheliterature.Ouroriginalresearchinterestinthetopicstemmedfrom
extensiveapplicationexperiencesinsentimentanalysis(SA)inastart-upcompanyseveralyears
ago.AtypicalSAprojectstartswithaclientwhoisinterestedinconsumeropinionsexpressed
in social media about their products or services and those of their competitors. There are two
main analysis tasks that an SA system needs to do: (1) discover the entities (e.g., iPhone) and
entityattributes/features(e.g.,batterylife)thatpeopletalkedaboutinopiniondocumentssuch
asonlinereviewsand(2)determinewhethertheopinionabouteachentityorentityattributeis
positive,negative,orneutral[Liu,2012,2015].Forexample,fromthesentence“iPhoneisreally
cool,butitsbatterylifesucks,”anSAsystemshoulddiscoverthattheauthoris(1)positiveabout
iPhoneand(2)negativeaboutiPhone’sbatterylife.
Afterworkingonmanyprojectsinmanydomains(whicharetypesofproductsorservices)
for clients, we realized that there is a great deal of sharing of information across domains and
projects.Asweseemoreandmore,newthingsgetfewerandfewer.Itiseasytoseethatsentiment
words and expressions (such as good, bad, poor, terrible, and cost an arm and a leg) are shared
across domains. There is also a great deal of sharing of entities and attributes. For example,
everyproducthastheattributeofprice,mostelectronicproductshavebattery,andmanyofthem
alsohavescreen.ItissillynottoexploitsuchsharingtosignificantlyimproveSAtomakeitmuch
more accurate than without using such sharing but only working on each project and its data
inisolation.Theclassicmachinelearningparadigmlearnsexactlyinisolation.Givenadataset,
a learning algorithm runs on the data to produce a model. The algorithm has no memory and
thus is unable to use the previously learned knowledge. In order to exploit knowledge sharing,
anSAsystemhastoretainandaccumulatetheknowledgelearnedinthepastanduseittohelp
futurelearningandproblemsolving,whichisexactlywhatlifelonglearningaimstodo.
It is not hard to imagine that this sharing of information or knowledge across domains
andtasksisgenerallytrueineveryfield.Itisparticularlyobviousinnaturallanguageprocessing
becausethemeaningsofwordsandphrasesarebasicallythesameacrossdomainsandtasksand
soisthesentencesyntax.Nomatterwhatsubjectmatterwetalkabout,weusethesamelanguage,
althougheachsubjectmayuseonlyasmallsubsetofthewordsandphrasesinalanguage.Ifthat
isnotthecase,itisdoubtfulthatanaturallanguagewouldhaveeverbeendevelopedbyhumans.
Thus,lifelonglearningisgenerallyapplicable,notjustrestrictedtosentimentanalysis.
The goal of this book is to introduce this emerging machine learning paradigm and to
presentacomprehensivesurveyandreviewoftheimportantresearchresultsandlatestideasin
the area. We also want to propose a unified framework for the research area. Currently, there
areseveralresearchtopicsinmachinelearningthatarecloselyrelatedtolifelonglearning,most
notably,multi-tasklearningandtransferlearning,becausetheyalsoemploytheideaofknowl-
edge sharing and transfer. This book brings all these topics under one roof and discusses their
similaritiesanddifferences.Weseelifelonglearningasanextensiontotheserelatedparadigms.
Through this book, we would also like to motivate and encourage researchers to work on life-
longlearning.Webelieveitrepresentsamajorresearchdirectionforbothmachinelearningand
artificial intelligence for years to come. Without the capability of retaining and accumulating
knowledge learned in the past, making inferences about it, and using the knowledge to help
futurelearningandproblemsolving,achievingartificialgeneralintelligence(AGI)isunlikely.
Twomainprincipleshaveguidedthewritingofthisbook.First,itshouldcontainstrong
motivationsforconductingresearchinlifelonglearninginordertoencouragegraduatestudents
andresearcherstoworkonlifelonglearningproblems.Second,thewritingshouldbeaccessible
topractitionersandupper-levelundergraduatestudentswhohavebasicknowledgeofmachine
learninganddatamining.Yetthereshouldbesufficientin-depthmaterialsforgraduatestudents
whoplantopursuePh.D.degreesinthemachinelearningand/ordataminingfields.
Thisbookisthussuitableforstudents,researchers,andpractitionerswhoareinterestedin
machine learning, data mining, natural language processing, or pattern recognition. Lecturers
canreadilyusethebookinclassforcoursesinanyoftheserelatedfields.
ZhiyuanChenandBingLiu
August2018