Multi-class Generalized Binary Search for Active Inverse Reinforcement Learning

2.2 MB·English

by  Francisco Melo

Multi-class Generalized Binary Search for Active Inverse Reinforcement
Learning
Francisco S. Melo Manuel Lopes
INESC-ID/Instituto Superior Técnico INRIA Bordeaux Sud-Ouest
Portugal France
[email protected] [email protected]
Abstract ing from demonstration. This particular form of social
3 learning is commonly associated with imitation and
1
This paper addresses the problem of learning a task emulation behaviors in nature (Lopes et al, 2009a). It
0
2 from demonstration. We adopt the framework of in- isalsopossibletofindnumeroussuccessfulexamplesof
n verse reinforcement learning, where tasks are repre- robot systems that learn from demonstration (see the
a sented in the form of a reward function. Our contribu- survey works of Argall et al, 2009; Lopes et al, 2010).
J
tion is a novel active learning algorithm that enables In the simplest form of interaction, the demonstration
3
2 the learning agent to query the expert for more infor- may consist of examples of the right action to take in
mative demonstrations, thus leading to more sample- different situations.

]
G efficient learning. For this novel algorithm (General- In our approach to learning from demonstration we
ized Binary Search for Inverse Reinforcement Learn-
L adopt the formalism of inverse reinforcement learn-
s. ing, or GBS-IRL), we provide a theoretical bound on ing (IRL), where the task is represented as a reward
c sample complexity and illustrate its applicability on
function (Ng and Russel, 2000). From this representa-
[
several different tasks. To our knowledge, GBS-IRL
tion, the agent can then construct its own policy and

1 is the first active IRL algorithm with provable sam-
solve the target task. However, and unlike many sys-
v
ple complexity bounds. We also discuss our method in
8 tems that learn from demonstration, in this paper we
8 light of other existing methods in the literature and its propose to combine ideas from active learning (Set-
4 general applicability in multi-class classification prob-
tles, 2009) with IRL, in order to reduce the data re-
5
lems. Finally, motivated by recent work on learning
. quirements during learning. In fact, many agents able
1 from demonstration in robots, we also discuss how dif-
0 to learn from demonstration are designed to process
ferent forms of human feedback can be integrated in a
3 batches of data, typically acquired before any actual
1 transparent manner in our learning framework.
learningtakesplace. Suchdataacquisitionprocessfails
:
v to take advantage of any information the learner may
i
X acquire in early stages of learning to guide the acquisi-
1 Introduction
r tion of new data. Several recent works have proposed
a
that a more interactive learning may actually lead to
Social learning, where an agent uses information pro-
improved learning performance.
vided by other individuals to polish or acquire anew
WeadoptaBayesianapproachtoIRL,followingRa-
skills,islikelytobecomeoneprimaryformofprogram-
machandran and Amir (2007), and allow the learning
ming such complex intelligent systems (Schaal, 1999).
agent to actively select and query the expert for the
Paralleling the social learning ability of human infants,
desired behavior at the most informative situations.
an artificial system can retrieve a large amount of task
We contribute a theoretical analysis of our algorithm
related information by observing and/or interacting
that provides a bound on the sample complexity of our
with other agents engaged in relevant activities. For
learning approach and illustrate our method in several
example, the behavior of an expert can bias an agent’s
problems from the IRL literature.
exploration of the environment, improve its knowledge
of the world, or even lead it to reproduce parts of the Finally, even if learning from demonstration is the
observed behavior (Melo et al, 2007). main focus of our paper and an important skill for in-
In this paper we are particularly interested in learn- telligent agents interacting with human users, the abil-
1
ity to accommodate different forms of feedback is also 1.1 Related Work
useful. In fact, there are situations where the user may
be unable to properly demonstrate the intended be- There is extensive literature reporting research on in-
havior and, instead, prefers to describe a task in terms telligent agents that learn from expert advice. Many
of a reward function, as is customary in reinforcement examplesfeatureroboticagentsthatlearnsimpletasks
learning (Sutton and Barto, 1998). As an example, from different forms of human feedback. Examples in-
suppose that the user wants the agent to learn how to cludetherobotLeonardothatisabletolearnnewtasks
navigate a complex maze. The user may experience by observing changes induced in the world (as per-
difficulties in navigating the maze herself and may, in- ceived by the robot) by a human demonstrating the
stead, allow the agent to explore the maze and reward target task Breazeal et al (2004). During learning,
it for exiting the maze. Leonardo provides additional feedback on its current
understanding of the task that the human user can
Additionally, recent studies on the behavior of naïve
then use to provide additional information. We refer
users when instructing agents (namely, robots) showed
the survey works of Argall et al (2009); Lopes et al
thatthefeedbackprovidedbyhumansisoftenambigu-
(2010) for a comprehensive discussion on learning from
ous and does not map in any obvious manner to either
demonstration.
a reward function or a policy (Thomaz and Breazeal,
2008; Cakmak and Thomaz, 2010). For instance, it In this paper, as already mentioned, we adopt the
was observed that human users tend to provide learn- inverse reinforcement learning (IRL) formalism intro-
ing agents with anticipatory or guidance rewards, a duced in the seminal paper by Ng and Russel (2000).
situation seldom considered in reinforcement learning One appealing aspect of the IRL approach to learn-
(Thomaz and Breazeal, 2008). This study concludes ing from demonstration is that the learner is not just
that robust agents able to successfully learn from hu- “mimicking” the observed actions. Instead, the learner
man users should be flexible to accommodate different infers the purpose behind the observed behavior and
forms of feedback from the user. sets such purpose as its goal. IRL also enables the
learner to accommodate for differences between itself
In order to address the issues above, we discuss how
and the demonstrator (Lopes et al, 2009a).
otherformsofexpertfeedback(beyondpolicyinforma-
tion) may be integrated in a seamless manner in our Theappealingfeaturesdiscussedabovehaveledsev-
IRL framework, so that the learner is able to recover eral researchers to address learning from demonstra-
efficiently the target task. In particular, we show how tion from an IRL perspective. Abbeel and Ng (2004)
to combine both policy and reward information in our explored inverse reinforcement learning in a context of
learning algorithm. Our approach thus provides a use- apprenticeshiplearning,wherethepurposeofthelearn-
ful bridge between reinforcement learning (or learning ingagentis toreplicatethe behavior ofthe demonstra-
by trial and error) and imitation learning (or learning tor, but is only able to observe a sequence of states ex-
from demonstration), a line of work seldom explored perienced during task execution. The IRL formalism
in the literature (see, however, the works of Knox and allows the learner to reason about which tasks could
Stone, 2010, 2011, and discussion in Section 1.1). lead the demonstrator to visit the observed states and
The paper is organized as follows. In the remain- infer how to replicate the inferred behavior. Syed et al
der of this section, we provide an overview of related (Syedetal,2008;SyedandSchapire,2008)havefurther
work on social learning, particularly on learning from explored this line of reasoning from a game-theoretic
demonstration. We also discuss relevant research in perspective, and proposed algorithms to learn from
IRL and active learning, and discuss our contribu- demonstration with provable guarantees on the per-
tions in light of existing work. Section 2 revisits core formance of the learner.
background concepts, introducing the notation used Ramachandran and Amir (2007) introduced
throughout the paper. Section 3 introduces our active Bayesian inverse reinforcement learning (BIRL),
IRL algorithm, GBS-IRL, and provides a theoretical where the IRL problem is cast as a Bayesian inference
analysis of its sample complexity. Section 4 illustrates problem. Given a prior distribution over possible
the application of GBS-IRL in several problems of dif- target tasks, the algorithm uses the demonstration
ferent complexity, providing an empirical comparison by the expert as evidence to compute the poste-
withothermethodsintheliterature. Finally, Section5 rior distribution over tasks and identify the target
concludesthepaper,discussingdirectionsforfuturere- task. Unfortunately, the Monte-Carlo Markov chain
search. (MCMC) algorithm used to approximate the poste-
2
rior distribution is computationally expensive, as it ior. The system also allows the human user to provide
requires extensive sampling of the space of possible corrective feedback as the robot executes the learned
rewards. To avoid such complexity, several posterior task.1 The querying strategy in CBA can be classified
works have departed from the BIRL formulation both as stream-based and as mellow (see discussions
and instead determine the task that maximizes the in the survey works of Settles, 2009; Dasgupta, 2011).
likelihood of the observed demonstration (Lopes et al, Stream-based, since the learner is presented with a
2009b; Babes et al, 2011). stream of samples (in the case of CBA, samples cor-
Theaforementionedmaximumlikelihoodapproaches respond to possible situations) and only asks for the
of Lopes et al (2009b) and Babes et al (2011) take labels (i.e., correct actions) of those samples it feels
advantage of the underlying IRL problem structure uncertain about. Mellow, since it does not seek highly
and derive simple gradient-based algorithms to de- informative samples, but queries any sample that is at
termine the maximum likelihood task representation. all informative.
Two closely related works are the maximum entropy In the IRL literature, active learning was first ex-
approach of Ziebart et al (2008) and the gradient IRL plored in a preliminary version of this paper (Lopes
approach of Neu and Szepesvari (2007). While the for- et al, 2009b). In this early version, the learner actively
mer selects the task representation that maximizes the queries the expert for the correct action in those states
likelihood of the observed expert behavior, under the where it is most uncertain about the correct behavior.
maximum entropy distribution, the latter explores a Unlike CBA, this active sampling approach is aggres-
gradient-basedapproachtoIRL,butthewherethetask sive and uses membership query synthesis. Aggressive,
representation is selected so as to induce a behavior as since it actively selects highly informative samples.
similar as possible to the expert behavior. And, unlike CBA, it can select (“synthesize”) queries
from the whole input space. Judah et al (2011) pro-
Finally, Ross and Bagnell (2010) propose a learning
pose a very similar approach, the imitation query-by-
algorithm that reduces imitation learning to a classifi-
committee (IQBC) algorithm, which differs only from
cation problem. The classifier prescribes the best ac-
the previous active sampling approach in the fact that
tion to take in each possible situation that the learner
the learner is able to accommodate the notion of “bad
can encounter, and is successively improved by enrich-
states”,i.e., statestobeavoidedduringtaskexecution.
ing the data-set used to train the classifier.
Cohnetal(2011)proposeanothercloselyrelatedap-
All above works are designed to learn from whatever
proach that, however, uses a different criterion to se-
data is available to them at learning time, data that
lect which situations to query. EMG-AQS (Expected
is typically acquired before any actual learning takes
MyopicGainActionQueryingStrategy)queriestheex-
place. Such data acquisition process fails to take ad-
pertfor the correctactioninthosestateswheretheex-
vantage of the information that the learner acquires
pectedgain of information ispotentiallylarger. Unfor-
in early stages of learning to guide the acquisition of
tunately, as discussed by Cohn et al (2011), the deter-
new, more informative data. Active learning aims to
mination of the expected gain of information requires
reducethedatarequirementsoflearningalgorithmsby
extensive computation, rendering EMG-AQS compu-
actively selecting potentially informative samples, in
tationally costly. On a different line of work, Ross et al
contrast with random sampling from a predefined dis-
(2011); Judah et al (2012) address imitation learning
tribution (Settles, 2009). In the case of learning from
using a no-regret framework, and propose algorithms
demonstration, active learning can be used to reduce
for direct imitation learning with provable bounds on
the number of situations that the expert/human user
the regret. Finally, Melo and Lopes (2010) use active
is required to demonstrate. Instead, the learner should
learning in a metric approach to learning from demon-
proactively ask the expert to demonstrate the desired
stration.
behavior at the most informative situations.
Our approach in this paper is a modified version of
Confidence-based autonomy (CBA), proposed by
our original active sampling algorithm (Lopes et al,
Chernova and Veloso (2009), also enables a robot to
2009b). We depart from the generalized binary search
learn a task from a human user by building a mapping
(GBS) algorithm of Nowak (2011) and adapt it to the
between situations that the robot has encountered and
IRL setting. To this purpose, we cast IRL as a (multi-
the adequate actions. This work already incorporates
class) classification problem and extend the GBS al-
a mechanism that enables the learner to ask the ex-
pert for the right action when it encounters a situation 1Related ideas are further explored in the dogged learning
in which it is less confident about the correct behav- architecture of Grollman and Jenkins (2007).
3
gorithm of Nowak (2011) to this multi-class setting. of these different sources of information in face of
WeanalyzethesamplecomplexityofourGBS-IRLap- our empirical results.
proach, thus providing the first active IRL algorithm
From a broader perspective, our analysis is a non-
with provable bounds on sample complexity. Also, to
trivial extension of the results of Nowak (2011) to a
the extent of our knowledge, GBS-IRL is the first ag-
multiclass setting, having applications not only onIRL
gressive active learning algorithm for non-separable,
but on any multiclass classification problem.
multi-class data (Dasgupta, 2011).
Weconcludethisdiscussionofrelatedworkbypoint-
2 Background and Notation
ingoutthatallaboveworksdescribesystemsthatlearn
from human feedback. However, other forms of expert
This section introduces background material on
advicehavealsobeenexploredintheagentlearninglit-
Markov decision processes and the Bayesian inverse re-
erature. PriceandBoutilier(1999,2003)haveexplored
inforcement learning formalism, upon which our con-
how a learning agent can improve its performance by
tributions are developed.
observing other similar agents, in what could be seen
as “implicit” imitation learning. In these works, the
demonstrator is, for all purposes, oblivious to the fact 2.1 Markov Decision Processes
that its actions are being observed and learned from.
AMarkov decision problem (MDP)describesasequen-
Instead, the learned observes the behavior of the other
tial decision problem in which an agent must choose
agents and extracts information that may be useful for
the sequence of actions that maximizes some reward-
its own learning (for example, it may extract useful
based optimization criterion. Formally, an MDP M is
information about the world dynamics).
a tuple M = (X,A,P,r,γ), where X represents the
In a more general setting, Barto and Rosenstein
state-space, A the finite action space, P represents the
(2004) discuss how different forms of supervisory in-
transition probabilities, r is the reward function and
formation can be integrated in a reinforcement learn-
γ is a positive discount factor. P(y | x,a) denotes the
ing architecture to improve learning. Finally, Knox
probabilityoftransitioningfromstatextostateywhen
andStone(2009,2010)introducethetamerparadigm,
action a is taken, i.e.,
that enables a reinforcement learning agent to use hu-
man feedback (in addition to its reinforcement signal) P(y | x,a) = P[X = y | X = x,A = a],
t+1 t t
to guide its learning process.
where each X ,t = 1,..., is a random variable (r.v.)
t
demoting the state of the process at time-step t and A
t
1.2 Contributions
is a r.v. denoting the action of the agent at time-step
t.
Our contributions can be summarized as follows:
A policy is a mapping π : X × A → [0,1], where
π(x,a) is the probability of choosing action a ∈ A in
• A novel active IRL algorithm, GBS-IRL, that ex-
state x ∈ X. Formally,
tends generalized binary search to a multi-class
setting in the context of IRL. π(x,a) = P[A = a | X = x].
t t
• The sample-complexity analysis of GBS-IRL. We It is possible to associate with any such policy π a
establish, under suitable conditions, the exponen- value-function,
tial convergence of our active learning method, as
(cid:34) ∞ (cid:35)
a function of the number of samples. As pointed Vπ(x) = E (cid:88)γtr(X ,A ) | X = x ,
π t t 0
out earlier, to our knowledge ours is the first
t=0
work providing sample complexity bounds on ac-
where the expectation is now taken over possible tra-
tive IRL. Several experimental results confirm the
jectories of {X } induced by policy π. The purpose of
good sample performance of our approach. t
the agent is then to select a policy π∗ such that
• A general discussion on how different forms of ex- Vπ∗(x) ≥ Vπ(x),
pert information (namely action and reward in-
formation) can be integrated in our IRL setting. for all x ∈ X. Any such policy is an optimal policy
We illustrate the applicability of our ideas in sev- for that MDP and the corresponding value function is
eral simple scenarios and discuss the applicability denoted by V∗.
4
Given any policy π, the following recursion holds very large) sample of R obtained according to ρ. We
denote this sample by R , and associate with each el-
(cid:88) ρ
Vπ(x) = rπ(x)+γ Pπ(x,y)Vπ(y) ement r ∈ R a prior probability p (r ) given by
k ρ 0 k
y∈X
ρ(r )
k
(cid:80) p (r ) = .
(cid:80)where Pπ(x,y) = a∈Aπ(x,a)Pa(x,y) and rπ(x) = 0 k (cid:80)iρ(ri)
π(x,a)r(x,a). For the particular case of the op-
a∈A
timal policy π∗, the above recursion becomes Associated with each reward rk ∈ Rρ and each x ∈ X,
we define the set of greedy actions at x with respect to
 
r as
(cid:88) k
V∗(x) = maxr(x,a)+γ Pa(x,y)V∗(y).
a∈A A (x) = {a ∈ A | a ∈ argmaxQ (x,a)}
y∈X k k
where Q is the Q-function associated with the opti-
We also define the Q-function associated with a pol- k
mal policy for r , as defined in (1). From the sets
icy π as k
A (x),x ∈ X, we define the greedy policy with respect
k
(cid:88)
Qπ(x,a) = r(x,a)+γ Pa(x,y)Vπ(y) to rk as the mapping πk : X ×A → [0,1] given by
y∈X I (a)
A (x)
π (x,a) = k ,
which, in the case of the optimal policy, becomes k |A (x)|
k
Q∗(x,a) = r(x,a)+γ (cid:88)P (x,y)V∗(y) where we write IU to denote the indicator function for
a
a set U. In other words, for each x ∈ X, the greedy
y∈X
= r(x,a)+γ (cid:88)P (x,y)maxQ∗(y,b). (1) policy with respect to rk is defined as a probability
a distribution that is uniform in A (x) and zero in its
b∈A k
y∈X
complement. We assume, without loss of generality,
that for any r ,r ∈ R , A (x) (cid:54)= A (x) for at least one
i j ρ i j
2.2 Bayesian Inverse Reinforcement Learn-
x ∈ X.2
ing
For any r ∈ R , consider a perturbed version of π
k ρ k
As seen above, an MDP describes a sequential decision where, for each x ∈ X, action a ∈ A is selected with a
making problem in which an agent must choose its ac- probability
tions so as to maximize the total discounted reward. (cid:40)
β (x) if a ∈/ A (x)
In this sense, the reward function in an MDP encodes πˆ (x,a) = k k (2)
k
γ (x) if a ∈ A (x),
the task of the agent. k k
Inverse reinforcement learning (IRL) deals with the
where, typically, β (x) < γ (x).3 We note that both
k k
problem of recovering the task representation (i.e., the
π and the uniform policy can be obtained as limits of
k
rewardfunction)givenademonstrationofthebehavior
πˆ ,bysettingβ (x) = 0orβ (x) = γ (x),respectively.
k k k k
to be learned (i.e., the desired policy). In this paper
FollowingtheBayesianIRLparadigm,thelikelihoodof
we adopt the formulation in Ramachandran and Amir
observing an action a by the demonstrator at state x,
(2007), whereIRLiscastasaBayesian inference prob-
given that the target task is r , is now given by
k
lem, in which the agent is provided with samples of the
desired policy, π∗, and it must identify the target re- (cid:96) (x,a) = P[A = a | X = x,r∗ = r ] = πˆ (x,a).
k t t k k
ward function, r∗, from a general set of possible func- (3)
tions R. Prior to the observation of any policy sample
2Thisassumptionmerelyensuresthattherearenoredundant
and given any measurable set R ⊂ R, the initial belief
rewards on R . If two such rewards r ,r existed in R , we
ρ i j ρ
that r∗ ∈ R is encoded in the form of a probability could safely discard one of the two, say r , setting p (r ) ←
j 0 i
density function ρ defined on R, i.e., p (r )+p (r ).
0 i 0 j
3Policy πˆ assigns the same probability, γ (x) to all actions
k k
(cid:90)
that, for the particular reward r , are optimal in state x. Simi-
P[r∗ ∈ R] = ρ(r)dr. k
larly,itassignsthesameprobability,β (x),toallcorresponding
k
R
sub-optimal actions. This perturbed version of π is convenient
k
As discussed by Ramachandran and Amir (2007); both for its simplicity and because it facilitates our analysis.
However, other versions of perturbed policies have been consid-
Lopes et al (2009b), it is generally impractical to ex-
ered in the IRL literature—see, for example, the works of Ra-
plicitlymaintainandupdateρ. Instead,asintheafore-
machandranandAmir(2007);NeuandSzepesvari(2007);Lopes
mentionedworks,weworkwithafinite(butpotentially et al (2009b).
5
Given a history of t (independent) observations, For an indexed set of samples, {(x ,a ),λ ∈ Λ}, we
λ λ
F = {(x ,a ),τ = 0,...,t}, the likelihood in (3) can write h to denote h(x ,a ), when the index set is
t τ τ λ λ λ
now be used in a standard Bayesian update to com- clear from the context.
pute, for every r ∈ R , the posterior probability The prior distribution p over R induces an equiv-
k ρ 0 ρ
alent distribution over H, which we abusively also de-
P[r∗ = r ]P[F | r∗ = r ]
p (r ) = k t k note as p , and is such that p (h ) = p (r ). We let
t k 0 0 k 0 k
Z
the history of observations up to time-step t be
p (r )(cid:81)t (cid:96) (x ,a )
= 0 k τ=0 k τ τ ,
Z F = {(x ,a ),τ = 0,...,t},
t τ τ
where Z is a normalization constant.
andβ andγ betheestimatesofβ∗ andγ∗ associated
Fortheparticularcaseofr∗ wewritethecorrespond- h h
with the hypothesis h. Then, the distribution over H
ing perturbed policy as
after observing F can be updated using Bayes rule as
t
(cid:40)
β∗(x) if a ∈/ A∗(x)
πˆ∗(x,a) = p (h) (cid:44) P[h∗ = h | F ]
γ∗(x) if a ∈ A∗(x), t t
∝ P[a | x ,h∗ = h,F ]P[h∗ = h | F ]
t t t−1 t−1
and denote the maximum noise level as the positive = P[a | x ,h∗ = h]P[h = h∗ | F ]
t t t−1
constant α defined as
≈ γ (x )(1+ht)/2β (x )(1−ht)/2p (h), (4)
h t h t t−1
α = supβ∗(x).
x∈X where we assume, for all x ∈ X,
3 Multiclass Active Learning for In- |A (x)|γ (x) ≤ |A∗(x)|γ∗(x), (5)
h h
verse Reinforcement Learning
(cid:80)
and p (h) is normalized so that p (h) = 1. Note
t h∈H t
that,in(4),weaccommodateforthepossibilityofhav-
In this section we introduce our active learning ap-
ingaccess(foreachhypothesis)toinaccurateestimates
proach to IRL.
β and γ of β∗ and γ∗, respectively.
h h
We consider a partition of the state-space X into
3.1 Preliminaries
a disjoint family of N sets, Ξ = {X ,...,X } such
1 N
Todevelopanactivelearningalgorithmforthissetting, that all hypotheses h ∈ H are constant in each set
weconverttheproblemofdeterminingr∗intoanequiv- Xi,i = 1...,N. In other words, any two states
alent classification problem. This mostly amounts to x,y lying in the same Xi are indistinguishable, since
rewriting of the Bayesian IRL problem from Section 2 h(x,a) = h(y,a) for all a ∈ A and all h ∈ H. This
using a different notation. means that our hypothesis space H induces an equiva-
We define the hypothesis space H as follows. For lence relation in X in which two elements x,y ∈ X are
every rk ∈ Rρ, the kth hypothesis hk : X → {−1,1}|A| equivalent if {x,y} ⊂ Xi. We write [x]i to denote the
is defined as the function (any) representative of the set Xi.4
The following definitions extend those of Nowak
h (x,a) = 2I (a)−1,
k A (x) (2011).
k
where we write h (x,a) to denote the ath component
k Definition 1 (k-neighborhood). Two sets X ,X ∈ Ξ
i j
of h (x). Intuitively, h (x) identifies (with a value of
k k are said to be k-neighbors if the set
1) the greedy actions in x with respect to r , assigning
k
(cid:8) (cid:9)
avalueof−1toallotheractions. WetakeH astheset h ∈ H | A ([x] ) (cid:54)= A ([x] )
h i h j
of all such functions h . Note that, since every reward
k
prescribesatleastoneoptimalactionperstate,itholds has, at most, k elements, i.e., if there are k or fewer
that for every h ∈ H and every x ∈ X there is at least hypotheses in H that output different optimal actions
onea ∈ Asuchthath(x,a) = 1. Wewriteh∗ todenote in Xi and Xj.
the target hypothesis, corresponding to r∗.
4While this partition is, perhaps, of little relevance in prob-
As before, given a hypothesis h ∈ H, we define the
lems with a small state-space X, it is central in problems with
set of greedy actions at x according to h as large (or infinite) state-space, since the state to be queried has
tobeselectedfromasetofN alternatives,insteadofthe(much
Ah(x) = {a ∈ A | h(x,a) = 1}. larger) set of |X| alternatives.
6
Definition 2. The pair (X,H) is k-neighborly if, Algorithm 1 GBS-IRL (version 1)
for any two sets Xi,Xj ∈ Ξ, there is a sequence Require: MDP parameters M\r
{X(cid:96)0,...,X(cid:96)n} ⊂ Ξ such that Require: Reward space Rρ
Require: Prior distribution p over R
0
• X = X and X = X ; 1: Compute H from R
(cid:96)0 i (cid:96)n j ρ
2: Determine partition Ξ=X ,...X of X
1 N
• For any m, X and X are k-neighbors. 3: Set F =∅
(cid:96)m (cid:96)m+1 0
4: for all t=0,... do
The notion of k-neighborhood structures the state- 5: Set c =min W(p ,[x] )
t i=1,...,N t i
space X in terms of the hypotheses space H, and this 6: if there are 1-neighbor sets Xi,Xj such that
structurecanbeexploitedforactivelearningpurposes.
W(p ,[x] )>c , W(p ,[x] )>c
t i t t j t
A∗(p ,[x] )(cid:54)=A∗(p ,[x] ),
t i t j
3.2 Active IRL using GBS
then
In defining our active IRL algorithm, we first consider 7: Sample xt+1 from Xi or Xj with probability 1/2
8: else
a simplified setting in which the following assumption
9: Samplex fromthesetX thatminimizesW(p ,[x] ).
t+1 i t i
holds. WepostponetoSection3.3thediscussionofthe
10: end if
more general case. 11: Obtain noisy response a
t+1
12: Set F ←F ∪{(x ,a )}
t+1 t t+1 t+1
Assumption 1. For every h ∈ H and every x ∈ X, 13: Update p from p using (6)
t+1 t
|A (x)| = 1. 14: end for
h
15: return hˆ =argmax p (h).
t h∈H t
In other words, we focus on the case where all hy-
pothesis considered prescribe a unique optimal action
per state. A single optimal action per state implies critically on Assumption 1. In Section 3.3, we discuss
that the noise model can be simplified. In particular, how Algorithm 1 can be modified to accommodate sit-
thenoisemodelcannowbeconstantacrosshypothesis, uations in which Assumption 1 does not hold.
since all h ∈ H prescribes the same number of optimal Our analysis of GBS-IRL relies on the following fun-
actionsineachstate(namely, one). Wedenotebyγˆ(x) damental lemma that generalizes Lemma 3 of Nowak
and βˆ(x) the estimates of γ∗ and β∗, respectively, and (2011) to multi-class settings.
consider a Bayesian update of the form:
Lemma 1. Let H denote a hypothesis space defined
1 over a set X, where (X,H) is assumed k-neighborly.
p (h) ∝ γˆ(x )(1+ht)/2βˆ(x )(1−ht)/2p (h), (6)
t Z t t t−1 Define the coherence parameter for (X,H) as
with 1−γˆ(x) = (|A|−1)βˆ(x) and Z an adequate nor- N
(cid:88)
malizationconstant. Forthissimplercase,(5)becomes c∗(X,H) (cid:44) maxminmax h([x] ,a)µ(X ),
i i
a∈A µ h∈H
i=1
βˆ(x) ≥ β∗(x) and γˆ(x) ≤ γ∗(x), (7)
where µ is a probability measure over X. Then, for
where, as before, we overestimate the noise rate β∗(x). any probability distribution p over H, one of the two
For a given probability distribution p, define the statements below holds:
weighted prediction in x as
1. There is a set X ∈ Ξ such that
i
(cid:88)
W(p,x) = max p(h)h(x,a),
a∈A W(p,[x] ) ≤ c∗.
h∈H i
and the predicted action at x as
2. There are two k-neighbor sets X and X such that
i j
(cid:88)
A∗(p,x) = argmax p(h)h(x,a).
W(p,[x] ) > c∗ W(p,[x] ) > c∗
a∈A i j
h∈H
A∗(p,[x] ) (cid:54)= A∗(p,[x] ).
i j
We are now in position to introduce a first version
of our active learning algorithm for inverse reinforce-
Proof. See Appendix A.1.
ment learning, that we dub Generalized Binary Search
for IRL (GBS-IRL). GBS-IRL is summarized in Al- This lemma states that, given any distribution over
gorithm 1. This first version of the algorithm relies the set of hypothesis, either there is a state [x] for
i
7
whichthereisgreatuncertaintyconcerningtheoptimal Theorem 2 (Convergence Rate of GBS-IRL). Let H
action or, alternatively, there are two k-neighboring denote our hypothesis space, defined over X, and as-
states [x] and [x] in which all except a few hypothe- sume that (X,H) is 1-neighborly. If, in the update (6),
i j
sis predict the same action, yet the predicted optimal βˆ(x) > α for all x ∈ X, then
action is strikingly different in both states. In either
case, it is possible to select a query that is highly in- P(cid:2)hˆ (cid:54)= h∗(cid:3) ≤ |H|(1−λ)t, t = 0,... (8)
t
formative.
The coherence parameter c∗ is the multi-class equiv- where λ = ε·min(cid:8)1−c∗, 1(cid:9) < 1 and
2 4
alent of the coherence parameter introduced by Nowak
(2011), and quantifies the informativeness of queries.
γˆ(x)−βˆ(x) βˆ(x)−γˆ(x)
Thatc∗ alwaysexistscanbeestablishedbynotingthat ε = minγ∗(x) +β∗(x) . (9)
x γˆ(x) βˆ(x)
thepartitionofX isfinite(sinceHisfinite)and,there-
fore, the minimization can be conducted exactly. On
Proof. See Appendix A.4.
theotherhand,ifHdoesnotincludetrivialhypotheses
that are constant all over X, it holds that c∗ < 1.
We are now in position to establish the convergence Theorem 2 extends Theorem 4 of Nowak (2011) to
properties of Algorithm 1. Let P[·] and E[·] denote the multi-class case. However, due to the existence of
the probability measure and corresponding expecta- multipleactions(classes),theconstantsobtainedinthe
tion governing the underlying probability over noise above bounds differ from those obtained in the afore-
and possible algorithm randomizations in query selec- mentioned work (Nowak, 2011). Interestingly, for c∗
tion. close to zero, the convergence rate obtained is near-
optimal, exhibiting a logarithmic dependence on the
Theorem 1 (Consistency of GBS-IRL). Let F =
t
dimension of the hypothesis space. In fact, we have
{(x ,a ),τ = 1,...,t} denote a possible history of ob-
τ τ
the following straightforward corollary of Theorem 2.
servations obtained with GBS-IRL. If, in the update
(6), βˆ(x) and γˆ(x) verify (7), then
Corollary 1 (Sample Complexity of GBS-IRL). Un-
lim P(cid:2)hˆt (cid:54)= h∗(cid:3) = 0. der the conditions of Theorem 2, for any given δ > 0,
t→∞ P(cid:2)hˆ = h∗(cid:3) > 1−δ as long as
t
Proof. See Appendix A.2.
1 |H|
Theorem 1 establishes the consistency of active
t ≥ log .
learning for multi-class classification. The proof re- λ δ
lies on a fundamental lemma that, roughly speaking,
To conclude this section, we note that our reduction
ensures that the sequence p (h∗) is increasing in ex-
t
of IRL to a standard (multi-class) classification prob-
pectation. This fundamental lemma (Lemma 2 in
lem implies that Algorithm 1 is not specialized in any
Appendix A.2) generalizes a related result of Nowak
particular way to IRL problems—in particular, it can
(2011)that,duetotheconsiderationofmultipleclasses
be used in general classification problems. Addition-
in GBS-IRL, does not apply. Our generalization re-
ally, the guarantees in Theorems 1 and 2 are also gen-
quires,inparticular,strongerassumptionsonthenoise,
βˆ(x),andimpliesadifferentrateofconvergence,aswill erally applicable in any multi-class classification prob-
lems verifying the corresponding assumptions.
soon become apparent. It is also worth mentioning
that the statement in Theorem 1 could alternatively
be proved using an adaptive sub-modularity argument
3.3 Discussion and Extensions
(again relying on Lemma 2 in Appendix A.2), using
the results of Golovin and Krause (2011). We now discuss the general applicability of our re-
Theorem 1 ensures that, as the number of sam- sults from Section 3.2. In particular, we discuss two
ples increases, the probability mass concentrates on assumptions considered in Theorem 2, namely the 1-
the correct hypothesis h∗. However, it does not pro- neighborlyconditionon(X,H)andAssumption1. We
vide any information concerning the rate at which also discuss how additional forms of expert feedback
P(cid:2)hˆ (cid:54)= h∗(cid:3) → 0. The convergence rate for our ac- may be integrated in a seamless manner in our GBS-
t
tive sampling approach is established in the following IRL approach, so that the learner is able to recover
result. efficiently the target task.
8
1-Neighborly Assumption: Algorithm 2 GBS-IRL (version 2)
Require: MDP parameters M\r
This assumption is formulated in Theorem 2. The
Require: Reward space R
ρ
1-neighborly assumption states that (X,H) is 1- Require: Prior distribution p over R
0
neighborly, meaning that it is possible to “structure” 1: Compute H from Rρ
2: Determine partition Ξ=X ,...X of X
the state-space X in a manner that is coherent with 1 N
3: Set F =∅
0
the hypothesis space H. To assess the validity of this
4: for all t=0,... do
assumption in general, we start by recalling that two 5: Sample x from the set X that minimizes W(p ,[x] ).
t+1 i t i
sets Xi,Xj ∈ Ξ are 1-neighbors if there is a single hy- 6: Obtain noisy response at+1
7: Set F ←F ∪{(x ,a )}
pothesis h ∈ H that prescribes different optimal ac- t+1 t t+1 t+1
0 8: Update p from p using (6)
t+1 t
tions in X and X . Then, (X,H) is 1-neighborly if
i j 9: end for
every two sets Xi,Xj can be “connected” by a sequence 10: return hˆt =argmaxh∈Hpt(h).
of 1-neighbor sets.
In general, given a multi-class classification problem
with hypothesis space H, the 1-neighborly assumption In situations where the 1-neighborly assumption
can be investigated by verifying the connectivity of the may not be verified, Lemma 1 cannot be used to en-
1-neighborhood graph induced by H on X. We refer to sure the selection of highly informative queries once
the work of Nowak (2011) for a detailed discussion of W(p,[x]i) > c∗ for all Xi. However, it should still be
this case, as similar arguments carry to our multi-class possible to use the main approach in GBS-IRL, as de-
extension. tailed in Algorithm 2. For this situation, we can spe-
cialize our sample complexity results in the following
In the particular case of inverse reinforcement learn-
immediate corollary.
ing, it is important to assess whether the 1-neighborly
assumption is reasonable. Given a finite state-space,
Corollary 2 (Convergence Rate of GBS-IRL, ver-
X, and a finite action-space, A, it is possible to build
sion2). LetHdenoteourhypothesisspace, definedover
a total of |A||X| different hypothesis.5 As shown in
X, and let βˆ(x) > α in the update (6). Then, for all t
the work of Melo et al (2010), for any such hypothesis
such that W(p ,[x] ) ≤ c∗ for some X ,
it is always possible to build a non-degenerate reward t i i
functionthatyieldssuchhypothesisastheoptimalpol- P(cid:2)hˆ (cid:54)= h∗(cid:3) ≤ |H|(1−λ)t, t = 0,...
icy. Therefore, a sufficiently rich reward space ensures t
that the corresponding hypothesis space H includes all
where λ = ε1−c∗ and ε is defined in (9).
|A||X| possible policies already alluded to. This triv- 2
ially implies that (X,H) is not 1-neighborly.
Unfortunately, as also shown in the aforementioned Multiple Optimal Actions:
work (Melo et al, 2010), the consideration of H as
In our presentation so far, we assumed that R is such
ρ
the set of all possible policies also implies that all
that, for any r ∈ R and any x ∈ X, |A (x)| = 1 (As-
ρ r
states must be sufficiently sampled, since no gener-
sumption 1). Informally, this corresponds to assuming
alization across states is possible. This observation
that, for every reward function considered, there is a
supports the option in most IRL research to focus on
single optimal action, π∗(x), at each x ∈ X. This
problems in which rewards/policies are selected from
assumption has been considered, either explicitly or
some restricted set (for example, Abbeel and Ng, 2004;
implicitly, in several previous works on learning by
Ramachandran and Amir, 2007; Neu and Szepesvari,
demonstration (see, for example, the work of Chernova
2007; Syed and Schapire, 2008). For the particular
and Veloso, 2009). Closer to our own work on active
case of active learning approaches, the consideration
IRL, several works recast IRL as a classification prob-
of a full set of rewards/policies also implies that there
lem, focusingondeterministicpoliciesπ : X → A(Ng
k
is little hope that any active sampling will provide any
and Russel, 2000; Cohn et al, 2011; Judah et al, 2011;
but a negligible improvement in sample complexity. A
RossandBagnell,2010;Rossetal,2011)andtherefore,
related observation can be found in the work of Das-
although not explicitly, also consider a single optimal
gupta(2005)inthecontextofactivelearningforbinary
action in each state.
classification.
However, MDPs with multiple optimal actions per
5This number is even larger if multiple optimal actions are state are not uncommon (the scenarios considered in
allowed. Section 4, for example, have multiple optimal actions
9
per state). In this situation, the properties of the re- Algorithm 3 GBS-IRL (version 3)
sultingalgorithmdonotfollowfromourpreviousanal- Require: MDP parameters M\r
ysis, since the existence of multiple optimal actions Require: Reward space Rρ
Require: Prior distribution p over R
necessarily requires a more general noise model. The 0
1: Compute H from R
immediate extension of our noise model to a scenario ρ
2: Determine partition Ξ=X ,...X of X
1 N
where multiple optimal actions are allowed poses sev- 3: Set F =∅
0
eral difficulties, as optimal actions across policies may 4: for all t=0,... do
be sampled with different probabilities. 5: Set ct =mini=1,...,NW(pt,[x]i)
6: if c <cˆthen
In order to overcome such difficulty, we consider t
7: Samplex fromthesetX thatminimizesW(p ,[x] ).
t+1 i t i
a more conservative Bayesian update, that enables a 8: else
seamless generalization of our results to scenarios that 9: Return hˆ =argmax p (h).
t h∈H t
admit multiple optimal actions in each state. Our up- 10: end if
11: Obtain noisy response a
date now arises from considering that the likelihood of t+1
12: Set F ←F ∪{(x ,a )}
t+1 t t+1 t+1
observinganactionfromasetAh(x)atstatexisgiven 13: Update pt+1 from pt using (10)
by γ (x). Equivalently, the likelihood of observing an 14: end for
h
action from A−A (x) is given by β (x) = 1−γ (x).
h h h
As before, γ∗ and β∗ correspond to the values of γ
h
To analyze the performance of this version of GBS-
and β for the target hypothesis, and we again let
h
IRL, let the set of predicted optimal actions at x be
defined as
α = supβ∗(x).
x∈X (cid:40) (cid:41)
(cid:88)
A (p,x) = a ∈ A | p(h)h(x,a) > cˆ .
Such aggregated noise model again enables the consid- cˆ
h
eration of an approximate noise model that is constant
across hypothesis, and is defined in terms of estimates We have the following results.
γˆ(x) and βˆ(x) of γ∗(x) and β∗(x). Given the noise
model just described, we get the Bayesian update Theorem 3 (Consistency of GBS-IRL, ver-
sion 3). Consider any history of observations
p (h) (cid:44) P[h∗ = h | F ] F = {(x ,a ),τ = 1,...,t} from GBS-IRL. If,
t t t τ τ
∝ P[a ∈ A | x ,F ]P[h∗ = h | F ] in the update (6), βˆ and γˆ verify (7) for all h ∈ H,
t h t t−1 t−1
then for any a ∈ A (p,[x] ),
= P[a ∈ A | x ]P[h = h∗ | F ] cˆ i
t h t t−1
≈ γˆ(x)(1+ht)/2βˆ(x)(1−ht)/2p (h), (10) lim P(cid:2)h∗([x] ,a) (cid:54)= 1(cid:3) = 0.
t−1 i
t→∞
withγˆ(x)andβˆ(x)verifying(7). Thisrevisedformula- Proof. See Appendix A.5.
tion implies that the updates to p are more conserva-
t
tive,inthesensethattheyareslowerto“eliminate” hy- Note that the above result is no longer formulated
pothesis from H. However, all results for Algorithm 1 in terms of the identification of the correct hypothesis,
remain valid with the new values for γˆ and βˆ. but in terms of the identification of the set of opti-
Unfortunately, by allowing multiple optimal actions mal actions. We also have the following result on the
perstate,itisalsomucheasiertofind(non-degenerate) sample complexity of version 3 of GBS-IRL.
situations where c∗ = 1, in which case our bounds are
void. However,ifwefocusonidentifying, ineachstate, Corollary 3 (Convergence Rate of GBS-IRL, ver-
at least one optimal action, we are able to retrieve sion 3). Let H denote our hypothesis space, defined
someguaranteesonthesamplecomplexityofouractive over X, and let βˆ(x) > α in the update (10). Then,
learning approach. We thus consider yet another ver- for all t such that W(pt,[x]i) ≤ c∗ for some Xi, and all
sion of GBS-IRL, described in Algorithm 3, that uses a ∈ Acˆ(p,[x]i),
uses a threshold cˆ< 1 such that, if W(p ,[x] ) > cˆ, we
t i P(cid:2)h∗([x] ,a) (cid:54)= 1(cid:3) ≤ |H|(1−λ)t, t = 0,...
consider that (at least) one optimal action at [x] has i
i
been identified. Once this is done, it outputs the most
where λ = ε1−c∗ and ε is defined in (9) with the new
likely hypothesis. Once at least one optimal action has 2
been identified in all states, the algorithm stops. values for γˆ and βˆ.
10