Neural Personalized Response Generation as Domain Adaptation

0.73 MB·

by  Weinan Zhang

Neural Personalized Response Generation as
Domain Adaptation
Wei-NanZhang,TingLiu,YifaWang,QingfuZhu
ResearchCenterforSocialComputingandInformationRetrieval,SchoolofComputerScienceand
Technology,HarbinInstituteofTechnology,Harbin,China,150001
7
1
0
2

n Abstract
a
J
Inthispaper,wefocusonthepersonalizedresponsegenerationforconversationalsys-

9
tems. Based on the sequence to sequence learning, especially the encoder-decoder

]
L framework, we propose a two-phase approach, namely initialization then adaptation,
C
tomodeltherespondingstyleofhumanandthengeneratepersonalizedresponses. For
.
s
c evaluation, we propose a novel human aided method to evaluate the performance of
[
thepersonalizedresponsegenerationmodelsbyonlinereal-timeconversationandof-

1
v fline human judgement. Moreover, the lexical divergence of the responses generated
3
bythe5personalizedmodelsindicatesthattheproposedtwo-phaseapproachachieves
7
0
goodresultsonmodelingtherespondingstyleofhumanandgeneratingpersonalized
2
0 responsesfortheconversationalsystems.
.
1
Keywords: Personalized Response Generation, Conversational
0
7
Systems, Sequence to Sequence Learning, Domain Adaptation
1
:
v
i
X 1. Introduction
r
a
Conversational system, which is also called conversational robot, virtual agent or
chatbot,etc,isaninterestingandchallengingresearchofartificialintelligence. Itcan
beappliedtoalargenumberofscenariosofhuman-computerinteraction,suchasques-
tion answering [1], negotiation [2, 3], e-commence [4], tutoring [5], etc. Recently,
conversational system usually plays the role of virtual companion or assistant of hu-
Emailaddress:[email protected](Wei-NanZhang)
PreprintsubmittedtoarXiv January10,2017
Table1:Anexampleoftheresponsesofdifferentpersonalitytoagivenpost.
Post Isitaproperdressforthefirstdate?
Response#1 Yep.
Response#2 Honey,itisverysuitable!
Response#3 Itisbettertowearingasilkscarf.
man. For example, the virtual assistant on mobile phone is one of the most popular
applicationofdsystem,suchas,Siri,Cortana,FacebookM,Viv,etc.
Despitethefunctionalsuccessoftheexistingconversationalsystems,literaturethat
tries to model the personalized responding of a conversational system is still sparse.
Table1showsanexampleoftheresponsesofdifferentpersonalitytoagivenpost.
From Table 1, we can see that the Response #1 is a briefly definite response to
the post. The Response #2 is an emotional response. And the Response #3 gives
anothersuggestionondressing. Itisobviousthatalthoughthe3responsesarerelevant
topost,theyrepresentthedifferentpersonalitiesandlanguagestylesrespectivelythat
mayfurtherimpactthechattingprocessandtheuserexperience.
Toaddresstheproblemofgeneratingpersonalizedresponseforconversationalsys-
tems,inthispaper,weproposedanoptimizedsequencetosequence(seq2seq)learning
framework,whichisbasedontherecurrentneuralnetwork,tomodelingtheinputpost
andgeneratingthepersonalizedresponse. Unlikethepreviousseq2seqapproachesfor
responsegenerationthatlearntoencodeanddecodeinthesameway,theproposedap-
proachfullyadoptstheadvantagesoftheseq2seqapproachesandthenlearnstoadapt
togeneratepersonalizedresponse.
Thecontributionsofthispaperaretwo-fold:
• We proposed a two-phase approach, namely initialization then adaptation, to
learntogeneratethepersonalizedresponseforconversationalsystems.
• We proposed a novel human aided method to evaluate the personality of the
generatedresponse.
Therestofthepaperisorganizedasfollows:Wewilldetailtheproposedtwo-phase
2
IInniittiiaalliizzaattiioonn AAddaappttaattiioonn
... ...
... ...
General Personalized
Training Data Training Data
... ...
Figure1: Theframeworkoftheproposedapproach.
approach to generating personalized response in the next section. In the following
section,theexperimentalresultsandanalysiswillbepresented. Thelasttwosections
aretherelatedworkandconclusionrespectively.
2. OurApproach
Recently,anumberofresearchworkshavebeenproposedtogeneratesentenceor
responsebasedontherecurrentneuralnetwork(RNN)[6,7,8,9,10,11,12,13,14,
15, 16, 17, 18, 19, 20, 21]. The target of these work is that given an input sentence
x,theRNNbasedmodeltriestogenerateanoutputsentencey thatcanmaximizethe
conditionalprobabilityofp(y|x). Fortheresponsegenerationofaconversationalsys-
tem,thistypeofapproachusuallyincludestwoparts,namelytheencoderanddecoder.
The encoder is to convert the input sentence into a vector which represents the com-
pletesemanticsoftheinputsentence. Thedecoderthengeneratestheoutputsentence
characterbycharacter(orwordbyword)accordingtotheinformationontheencoding
phase.
2.1. PersonalizedResponseGeneration
InspiredbytheRNNencoder-decoderframework,whichisproposedby[17]and[6],
inthispaper,weproposedapersonalizedresponsegenerationapproachforconversa-
tionalsystems.
Figure1showstheframeworkoftheproposedapproach.
3
FromFigure1,wecanseethattheproposedapproachconsistsoftwocomponents,
namelyinitializationthenadaptation,thefirstofwhichpre-trainstherespondingmodel
oftheconversationalsystemusingthelargescalegeneraltrainingdataandthesecond
stepfine-tunesthemodelonthesmallsizeofpersonalizedtrainingdata.
Next, we will detail the proposed personalized response generation model. Typi-
cally,theencoderanddecoderareimplementedbytheGRU[22,23]orLSTM[24,25,
26] based RNN. The encoder reads the input sentence word by word and outputs the
hiddenstateofeachword. ThesestatesaredenotedasH whichisalsocalledannota-
tions.Here,h representsthehiddenstateattimeianditiscomputedbyitslasthidden
i
stateh andtheinputwordattimei,X . Therefore,thehiddenstateattimetcanbe
i−1 i
denotedas:
h =f(h ,X ); H ={h ,h ,...,h } (1)
t t−1 t 1 2 T
Here, T equals to the length (the number of words) of the input sentence and f is a
non-linearfunctionwhichcanbeimplementedasLSTM[6]orGRU[23].
Theencoderthenconvertsthesehiddenstatestoacontextvectorcasasummary
ofthesemanticsoftheinputsentence.
c=q({h ,h ,...,h }) (2)
1 2 T
Where,ccanbeimplementedinmanyways,suchas[6]setc=h .
T
Forthedecodingprocess,s denotesthehiddenstateattimei. Itisalsocomputed
i
byanon-linearfunctionf, ofwhichthevariablesaretheoutputy andthehidden
i−1
states atlasttime. Thehiddenstateofthedecoderattimetcanbecomputedas:
i−1
s =f(s ,y ) (3)
t t−1 t−1
Note that the context vector c, which is generated from the encoder, is also used to
initializethefirsthiddenstate[6]orallofthehiddenstates[17]ofthedecodertomake
surethatthedecodercanbeconditionedbytheencoder. Therefore,thehiddenstateof
thedecoderattimetisupdatedas:
s =f(s ,y ,c) (4)
t t−1 t−1
4
Theoutputofthedecoderatthestates istomaptoadistributionoverthevocabulary
t
byusingthemaxoutactivationfunction[27]
In the proposed approach, the response generation model is implemented by an
RNN encoder-decoder framework, which is inspired by [18]. For the personalized
responsegeneration, weproposedatwo-phasetrainingapproachtofirstinitializethe
modelusingthelargescalegeneraltrainingdataandthenusethepersonalizedtraining
data to fine-tune the model to generate personalized response. All the parameters of
therespondingmodelaresharedbetweentheinitializationandadaptationprocess.
Tobalancethegenerationperformanceandtrainingtimeofthemodel,inthispaper,
wechoosetheGRUasthenon-linearfunctionf forbothencoderanddecoder.
Inthispaper, weutilizeaweightedsumscheme[18]todynamicallycomputethe
c foreachstateintheencodingprocessas:
i
T
(cid:88)
c = α h (5)
i ij j
j=1
Theweightα ofeachhiddenstateh iscomputedas:
ij j
exp(e )
α = ij (6)
ij (cid:80)T exp(e )
k=1 ik
Where,e = a(s ,h )isafeedforwardneuralnetwork,whichcanbecalledasthe
ij i−1 i
alignmentmodelorattentionmodel[18].
2.2. GenerationQualityOptimization
ComparedwiththeresponsethatisgeneratedbythegeneralRNNencoder-decoder
model, the two-phase training approach could generate distinct and personalized re-
sponse. However,wefoundanotherproblemthatwhenthefirstwordisdecodedtoa
highfrequencywordinthevocabulary,suchas“We”,“I”,“Yes”,etc,thepersonalized
responsemodelistendtogenerateageneralresponse.
To address the above problem, we utilized a learning scheme to generate the first
wordindecodingprocess,namelyLearningtoStart(LTS)model[28].Unliketheclas-
sicRNNencoder-decoderframeworkthatusesaspecialcharacter“</s>”togenerate
thefirstwordindecodingprocess,theLTSmodelindependentlylearnstopredictthe
5
firstwordusingthecontextvector,whichisgeneratedfromtheencodingprocess. The
LTSmodelcanbeformalizedas:
y =σ((σ(W c)+b )E+b ) (7)
0 i i e
Here,cisthecontextvectorwhichiscomputedbyEquation(5).Erepresentstheword
embedding matrix of the decoder, b and b are bias items. W is a learnable matrix
i e i
thatistrainedtomodeltheconditionaldependenceofthecontextvectorcandthefirst
wordindecodingprocess.
Byignoringthebiasitems,theEquation(7)canbetransformedasfollows:
y =g(c,E) (8)
0
WethusfoundthattheLTSistomodeltherelationbetweenthecontextvectorthatis
generatedfromtheencoderandtheembeddingmatrixofthedecoder.Accordingtothe
distribution of the generation probability over the decoding vocabulary, LTS predicts
thefirstwordforthedecoderandthenthedecodingprocessgoesonuntiltheresponse
isgeneratedcompletely. Inthispaper,weusedifferentvocabulariesforencodingand
decodingtomodelthepersonalityofthespeakerandresponderrespectively.
3. ExperimentsandAnalysis
3.1. ExperimentDataandSettings
As the proposed personalized response generation approach includes two phases,
therearetwoseparatetrainingdatasets,namelygeneraltrainingdataandpersonalized
training data (See Figure 1). The data for general training is collected from several
Chinese online forums. It contains 1 million one-to-one post and response pairs and
thevocabularycontains35thousandswords. Here,one-to-onemeansonepostisonly
correspondedtooneresponse. Inthispaper, LTP(http://www.ltp-cloud.com/)toolkit
isusedforChinesewordsegmentationforallthedata.
Forthepersonalizedtraining,weinvited5volunteers. Eachofthemshared2,000
messages of their chatting history from the use of instant messaging service without
anyprivacyinformation.Thegeneralmessages,suchas“你好(Hello)”,“早上好(Good
Morning)”,“再见(Seeyou)”,etc,areremovedfromthepersonalizedtrainingdata.
6
Note that, to train the personalized responding model, these messages from the
chattinghistorycanonlybeusedasresponses. Therefore,weneedtocollectaunique
postforeachofthemessages. Webuiltasearchengine,whichisbasedontheLucene1
toolkit, to retrieve similar responses from the general training data for the messages.
Foreachmessage,thepostofthemostsimilarresponsetothemessagewaschosenas
the post of the message. Therefore, for each volunteer, we thus obtained 2,000 post
and message pairs for personalized training. After training, we have 5 personalized
respondingmodelsthatcorrespondtothe5volunteersforthetest.
Theparametersettingsintheresponsegenerationmodelareasfollow:
The dimension of the hidden layer of the encoder and decoder is 1,024. The di-
mension of the word embedding is 500 which is obtained by using the word2vec
toolkit[29]. Theword2vecistrainedwiththeSogouCS&CAcorpus(2008version)2,
which is widely used for Chinese text analysis [30, 31]. The details of the training
corpusfortrainingwordembeddingareshowninTable2.
Table2:Statisticsofthetrainingcorpusforwordembeddinggeneration.
Datasize #oftokens Vocabularysize #ofthreads
SogouCS&CA 8.7GB 1,520,842,220 1,354,247 10
Theencoder-decoderframeworkisimplementedbyusingTheanotoolkit[32].The
batchsizeissetto128. Theiterationtimesaresetto10and8forthegeneraltraining
andpersonalizedtrainingrespectively.
3.2. TestandEvaluation
It is a non-trivial task to evaluate the performance of a responding model auto-
matically. As [7] said: “Automatic evaluation of response generation is still an open
problem.”. TheBLEUsocre[33],whichiswidelyusedinmachinetranslation,isnot
asuitableevaluationmetricforresponsegeneration. Astheresponsestothesamepost
1http://lucene.apache.org/
2http://www.sogou.com/labs/dl/cs.html
7
maysharelesscommonwords, itisimpossibletoconstructareferencesetwithade-
quatecoverage. Meanwhile,thePerplexity,whichisanevaluationmetricforlanguage
modeling,isalsonotreasonableforevaluatetherelevancebetweenpostandresponse.
To address the above issues, we design a novel human aided evaluation method
basedontheonlinereal-timechattingandofflinehumanjudgment. Thediagramofthe
evaluationmethodisshowninFigure2.
Chatbot
Random
Shelter ？ Chatting
TTeesstteerr
VVoolluunntteeeerr
Figure2: Theevaluationmethodbasedonhumanjudgementforpersonalizedresponsegeneration.
Theevaluationmethodincludesavolunteer,atesterandachatbot. Thevolunteer
and the tester are communicating through an instant messaging service. Here, the
tester is told to talk to a volunteer through the instant messaging service without any
constraints and preconditions as well as the tester do not know the existence of the
chatbot. During the chatting, each of the tester’s messages is sent to the volunteer
and the chatbot simultaneously. The Shelter in Figure 2 represents that the volunteer
couldnotseetheresponsethatisgeneratedbythechatbotbeforeitissenttothetester.
Thequestionmark“?” denotesthatthevolunteerneedstorandomlydecidewhetherto
respondbyhimself/herselforletthechatbotsendsitsresponse.Thereasonistoreduce
thepreferenceofthevolunteertotheresponseofthechatbot.
When the chatting is finished, the tester is asked to judge whether each of the re-
sponses is from the volunteer or someone else. We defined the imitation rate, r
imi
to evaluate the personality of the proposed dialogue generation approach. Here, we
8
usen todenotethenumberofresponsesthataregeneratedbythechatbot,butare
imi
judgedastheresponsesofthevolunteer. n isthetotalnumberofresponsesthatare
gr
generatedbythechatbot. Theimitationrateisthusrepresentedas:
n
r = imi (9)
imi n
gr
WecanobviouslyseefromEquation(9)thattheimitationratecanreflecttheability
of the chatbot on imitating the personalized responding style of the volunteers. The
largertheimitationrateis,thebetterthechatbotimitatesthevolunteers.
Note that we totally recruit 5 volunteers and 1 tester. All of them are not partic-
ipating in the training data collecting and coding. They also have no experience on
developing the relevant techniques. The tester and the volunteers have known each
otherverywellbeforetheevaluationaswell.
3.3. ExperimentalResults
As we described on the above section, we use the imitation rate to evaluate the
personality of the responses which are generated by the chatbot. Table 3 shows the
experimentalresultsoftheproposedapproachtogeneratingpersonalizedresponses.
Table3: Theexperimentalresultsoftheproposedapproachtopersonalizedresponsegeneration. ngr and
nvr representsthenumberofresponsesthataregeneratedbythechatbotandthevolunteerrespectively.
ntestisthetotalnumberofpostsfortesting.nimidenotesthenumberofresponsesthatisgeneratedbythe
chatbotbutarejudgedastheresponsesofthevolunteer.rimidenotestheimitationrate,whichisdefinedin
Equation(9).
Volunteer#1 Volunteer#2 Volunteer#3 Volunteer#4 Volunteer#5 Sum
n 29 26 21 33 33 142
gr
n 21 24 29 17 17 106
vr
n 50 50 50 50 50 250
test
n 11 9 8 13 9 50
imi
r 37.93% 34.62% 38.10% 39.40% 27.27% 35.21%
imi
Here,thetestersends50poststoeachvolunteerandthe5testsetssharedasmall
amountofcommonmessages. Meanwhile, themessagessenttothetesterarehybrid
9
Table4:Theevaluationresultsofthe5personalizedresponsegenerationmodelsbyhumanjudgement.ngr
representsthenumberofresponsesthataregeneratedbythechatbot.nimidenotesthenumberofresponses
thatisgeneratedbythechatbotbutarejudgedastheresponsesofthevolunteer.rimidenotestheimitation
rate,whichisdefinedinEquation(9).PRMisshortforthepersonalizedrespondingmodel.
PRM#1 PRM#2 PRM#3 PRM#4 PRM#5
n 50 50 50 50 50
gr
n 6 8 8 13 10
imi
r 12% 16% 16% 26% 20%
imi
responses. Thatistosay,inaconversation,theresponsestothetesteraregivenbythe
chatbotandavolunteerinarandominsertingway. Therefore,basedontheabovecon-
ditions,thejudgementofthetesteronthehybridresponsescanbeseenasa“relative”
evaluationontheabilityofthechatbotforimitatingthepersonalizedrespondingstyle
ofthevolunteer.
Moreover, we plan to evaluate the “absolute” ability of the chatbot on imitating
thepersonalizedrespondingstyleofthevolunteer. Wefirstrandomlysample50posts
thatare sentto thevolunteers bythe testerin previousevaluation. We thenuse the5
trainedmodelstogeneratepersonalizedresponsesofthe50postsrespectively. Finally,
weagainaskthetestertojudgewhethereachoftheresponsesisfromthevolunteeror
someoneelse. Table4showstheresultsofthehumanjudgementonthe5personalized
responsegenerationmodels.
From Table 4, we can see that the r scores of the 5 personalized responding
imi
models (PRM #1 - #5) are much lower than those of the corresponding volunteers in
Table3. Therearetworeasons. First, ontheonlinereal-timechatting, theresponses
generated by chatbot are randomly inserted to the conversation with the volunteers’
responses. It mayincreasethe confusionofthe testeronthe judgement. Second, the
online real-time chatting process is context-aware. Due to the “coherent model” in
mind,thevolunteersmaytendtoextendtheconversationtopic. Itmaymakethetester
believethatthepreviousresponsesarealsotypedbythevolunteerratherthangenerated
byachatbot.
10