Relational Transfer in Reinforcement Learning - Alchemy

170 Pages·2009·0.88 MB·English

RELATIONAL TRANSFER INREINFORCEMENTLEARNING
by
LisaTorrey
Adissertationsubmittedinpartialfulfillment of
therequirementsforthedegreeof
DoctorofPhilosophy
(ComputerSciences)
atthe
UNIVERSITYOFWISCONSIN–MADISON
May2009
i
ACKNOWLEDGMENTS
This research was partially supported by DARPA grant HR0011-04-1-0007, United States
NavalResearchLaboratorygrantN00173-06-1-G002, andDARPAgrantFA8650-06-C-7606.
Special thanks to my advisor, Jude Shavlik, to my collaborators Trevor Walker and Rich
Maclin, to my committee members, and to the Machine Learning Group at the University of
Wisconsin-Madison.
We offer a public distribution of our RoboCup players and server code. They are available at
http://www.biostat.wisc.edu/ ml-group/RoboCup.
∼
DISCARDTHISPAGE
ii
TABLE OF CONTENTS
Page
LISTOFTABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
LISTOFFIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 ThesisTopic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 ThesisStatement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 ThesisContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1 ReinforcementLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 Implementing RLwithSupportVectorRegression . . . . . . . . . . . . . 8
2.1.2 RoboCup: AChallengingReinforcementLearningDomain . . . . . . . . 9
2.2 LearningCurvesandStatisticalComparisons . . . . . . . . . . . . . . . . . . . . 12
2.3 InductiveLogicProgramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.1 WhatILP Learns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.2 HowILP Learns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 MarkovLogicNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3 SurveyofResearchonTransferLearning . . . . . . . . . . . . . . . . . . . . . . . . 22
3.1 TransferinGeneral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2 TransferinInductiveLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2.1 Inductive Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2.2 BayesianTransfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.3 HierarchicalTransfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2.4 TransferwithMissingDataorClassLabels . . . . . . . . . . . . . . . . . 29
3.3 TransferinReinforcementLearning . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3.1 Starting-Point Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
iii
Page
3.3.2 Imitation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.3.3 HierarchicalMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3.4 Alteration Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3.5 NewRLAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4 AvoidingNegativeTransfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.1 RejectingBadInformation . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4.2 ChoosingaSourceTask . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.3 ModelingTaskSimilarity . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5 Automatically MappingTasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.5.1 EqualizingTaskRepresentations . . . . . . . . . . . . . . . . . . . . . . . 42
3.5.2 TryingMultipleMappings . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.5.3 MappingbyAnalogy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.6 TheoryofTransferLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.7 RecapofContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4 Advice-BasedTransferMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.1 AdviceTaking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.2 PolicyTransferviaAdvice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.2.1 TranslatingQ-Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.2 GeneratingAdvice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.3 Experimental ResultsforPolicyTransfer . . . . . . . . . . . . . . . . . . 53
4.3 SkillTransferviaAdvice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3.1 LearningSkills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3.2 TranslatingSkillsandAddingHuman-Provided Advice . . . . . . . . . . 59
4.3.3 Experimental ResultsforSkillTransfer . . . . . . . . . . . . . . . . . . . 60
4.4 TestingtheBoundariesofSkillTransfer . . . . . . . . . . . . . . . . . . . . . . . 62
4.5 SummaryofAdvice-BasedTransfer . . . . . . . . . . . . . . . . . . . . . . . . . 64
5 Macro-OperatorTransfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.1 RelationalMacro-OperatorsandDemonstration . . . . . . . . . . . . . . . . . . . 67
5.2 Single-MacroTransferviaDemonstration . . . . . . . . . . . . . . . . . . . . . . 69
5.2.1 Single-MacroStructureLearning . . . . . . . . . . . . . . . . . . . . . . 69
5.2.2 Single-MacroRulesetLearning . . . . . . . . . . . . . . . . . . . . . . . 74
5.2.3 Experimental ResultsforSingle-MacroTransfer . . . . . . . . . . . . . . 76
5.3 Multiple-MacroTransferviaDemonstration . . . . . . . . . . . . . . . . . . . . . 79
5.3.1 Multiple-Macro StructureandRulesetLearning . . . . . . . . . . . . . . . 80
5.3.2 Experimental ResultsforMultiple-MacroTransfer . . . . . . . . . . . . . 84
5.4 TestingtheBoundariesofMacroTransfer . . . . . . . . . . . . . . . . . . . . . . 87
iv
Page
5.5 SummaryofMacroTransfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
6 TransferviaMarkovLogicNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.1 MLNRelationalQ-FunctionTransfer . . . . . . . . . . . . . . . . . . . . . . . . 95
6.1.1 LearninganMLNQ-functionfrom aSourceTask . . . . . . . . . . . . . . 97
6.1.2 Applying anMLNQ-function inaTargetTask . . . . . . . . . . . . . . . 98
6.1.3 Experimental ResultsforMLNQ-function Transfer . . . . . . . . . . . . . 100
6.2 MLNRelationalPolicyTransfer . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6.2.1 LearningandUsinganMLNPolicy . . . . . . . . . . . . . . . . . . . . . 104
6.2.2 Experimental ResultsforMLNPolicyTransfer . . . . . . . . . . . . . . . 105
6.3 TestingtheBoundariesofMLNPolicyTransfer . . . . . . . . . . . . . . . . . . . 107
6.4 SummaryofMLNTransfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7 ConclusionsandFutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.1 Algorithm Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
7.2 PotentialExtensionsofMyResearch . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.3 TheFutureofTransferLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
LISTOFREFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
NOMENCLATURE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
APPENDIX A.RoboCupFeatureandActionSpaces . . . . . . . . . . . . . . . . . 137
APPENDIX B.PropositionalMappings . . . . . . . . . . . . . . . . . . . . . . . . 139
APPENDIX C.ObjectMappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
APPENDIX D.ConditionalIndependenceProof . . . . . . . . . . . . . . . . . . . 147
DISCARDTHISPAGE
v
LIST OF TABLES
Table Page
1.1 Alistofmajoralgorithms contributed bythisthesis. . . . . . . . . . . . . . . . . . . 4
2.1 A randomization test to judge whether one group of numbers is significantly higher
thananother,basedonCohen[13]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Arandomizationtesttoestimatea95%confidenceintervalforthedifferencebetween
twogroupsofnumbers,basedonCohen[13]. . . . . . . . . . . . . . . . . . . . . . . 15
4.1 TransferAlgorithm 1: PolicyTransferviaAdvice . . . . . . . . . . . . . . . . . . . . 51
4.2 Results of statistical tests on the differences between areas under the curve in policy
transfer vs. RL-SVR for several source/target pairs. For p < 0.05, the difference
is significant and the winning algorithm is shown; otherwise, the difference is not
statisticallysignificant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3 TransferAlgorithm 2: SkillTransferviaAdvice . . . . . . . . . . . . . . . . . . . . . 57
4.4 Results of statistical tests on the differences between areas under the curve in skill
transfervs. policytransferforseveralsource/targetpairs. Forp < 0.05,thedifference
is significant and the winning algorithm is shown; otherwise, the difference is not
statisticallysignificant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.1 TransferAlgorithm 3: Single-MacroTransferviaDemonstration . . . . . . . . . . . . 70
5.2 Algorithmforlearningoneormoremacrostructures,givengamesGfromthesource
task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.3 Algorithm forlearning rules to control the transition between a pairof nodes n ,n
i i+1
in a macro Macro, given games G from the source task. Data for this task can come
from any game that contains the sequence represented by Macro or some prefix or
suffix of that sequence. For example, in a macro with nodes (n ,n ,n ,n ), prefixes
1 2 3 4
includegamesmatching(n ,n )andsuffixesincludegamesmatching(n ,n ,n ). . . 71
1 2 2 3 4
vi
Table Page
5.4 Algorithm for learning rules to control the action choice in a node n of a macro that
represents the action Action with the variable argument ActionArg, given games G
from the source task. Data for this task can come from any game, and most comes
from those that contain the sequence represented by Macro or some prefix or suffix
ofthatsequence. Forexample,inamacrowithnodes(n ,n ,n ,n ),prefixesinclude
1 2 3 4
gamesmatching(n ,n )andsuffixesincludegamesmatching(n ,n ,n ). . . . . . . 71
1 2 2 3 4
5.5 Algorithmforselectingandscoringthefinalrulesetfor onetransitionoraction. Rules
areaddedtothefinalsetiftheyincreasetheoverall F(10)measure. . . . . . . . . . . 77
5.6 Results of statistical tests on the differences between areas under the curve in single-
macro transfer vs. skill transfer for several source/target pairs. For p < 0.05, the
differenceissignificantandthewinningalgorithmisshown;otherwise,thedifference
isnotstatisticallysignificant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.7 TransferAlgorithm 4: Multiple-Macro TransferviaDemonstration . . . . . . . . . . 81
5.8 Algorithm for learning rules to control the entry arc of a node n of a macro, given
games G from the source task. Data for this task can come from any game, and most
comes from those that contain the sequence represented by Macro or some prefix of
that sequence. For example, in a macro with nodes (n ,n ,n ,n ), prefixes include
1 2 3 4
gamesmatching(n ,n ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
1 2
5.9 Algorithm for learning rules to control the loop arc of a node n of a macro, given
games G from the source task. Data for this task can come from any game, and most
comes from those that contain the sequence represented by Macro or some prefix of
that sequence. For example, in a macro with nodes (n ,n ,n ,n ), prefixes include
1 2 3 4
gamesmatching(n ,n ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
1 2
5.10 Resultsofstatisticaltestsonthedifferencesbetweenareasunderthecurveinmultiple-
macro transfer vs. single-macro transfer for several source/target pairs. For p <
0.05, the difference is significant and the winning algorithm is shown; otherwise, the
differenceisnotstatisticallysignificant. . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.11 Results of statistical tests on the differences between areas under the curve in single-
macro transfer with MLNs vs. regular single-macro transfer for several source/target
pairs. For p < 0.05, the difference is significant and the winning algorithm is shown;
otherwise,thedifferenceisnotstatisticallysignificant. . . . . . . . . . . . . . . . . . 92
6.1 TransferAlgorithm 5: MLNRelational Q-FunctionTransfer . . . . . . . . . . . . . . 96
vii
Table Page
6.2 Algorithm fordividing the Q-values of an action a into bins, given training data from
gamesGandaparameter ǫdetermining distancebetweenbins. . . . . . . . . . . . . . 97
6.3 Algorithmforestimatingthe Q-valueofactionaintarget-taskstate susingtheMLN
Q-function. This is a weighted sum of bin expected values, where the expected value
ofabinisestimatedfromthetraining dataforthatbin. . . . . . . . . . . . . . . . . . 99
6.4 Results of statistical tests on the differences between areas under the curve in MLN
Q-function transfer vs. multiple-macro transfer for several source/target pairs. For
p < 0.05,thedifferenceissignificantandthewinningalgorithmisshown;otherwise,
thedifferenceisnotstatisticallysignificant. . . . . . . . . . . . . . . . . . . . . . . . 102
6.5 TransferAlgorithm 6: MLNRelationalPolicyTransfer . . . . . . . . . . . . . . . . . 103
6.6 Results of statistical tests on the differences between areas under the curve in MLN
policy transfer vs. MLN Q-function transfer for several source/target pairs. For p <
0.05, the difference is significant and the winning algorithm is shown; otherwise, the
differenceisnotstatisticallysignificant. . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.7 Results of statistical tests on the differences between areas under the curve in ruleset
policy transfer vs. regular MLN policy transfer for several source/target pairs. For
p < 0.05,thedifferenceissignificantandthewinningalgorithmisshown;otherwise,
thedifferenceisnotstatisticallysignificant. . . . . . . . . . . . . . . . . . . . . . . . 109
6.8 Results of statistical tests on the differences between areas under the curve in multi-
step MLN policy transfer vs. regular MLN policy transfer for several source/target
pairs. For p < 0.05, the difference is significant and the winning algorithm is shown;
otherwise,thedifferenceisnotstatisticallysignificant. . . . . . . . . . . . . . . . . . 110
A.1 RoboCuptaskactionspaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
A.2 RoboCuptaskfeaturespaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
B.1 Propositionalmappingsfrom2-on-1BreakAwayto3-on-2BreakAway,usedinpolicy
transfer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
B.2 Propositionalmappingsfrom3-on-2MoveDownfieldto4-on-3MoveDownfield,used
inpolicytransfer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
B.3 Propositional mappings from 3-on-2 KeepAway to 4-on-3 KeepAway, used in policy
transfer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142