Personalised Query Suggestion for Intranet Search with Temporal User Profiling

0.35 MB·

by  Thanh Vu

Personalised Query Suggestion for Intranet Search with
Temporal User Profiling
Thanh Vu1, Alistair Willis1, Udo Kruschwitz2, and Dawei Song1,3
1 TheOpenUniversity,MiltonKeynes,UnitedKingdom
2 UniversityofEssex,Essex,UnitedKingdom
3 TianjinUniversity,Tianjin,P.R.China
{thanh.vu, alistair.willis, dawei.song}@open.ac.uk, [email protected]
ABSTRACT blebywebsearchengines. Forexample,websearchengines
cannot access those Intranet documents which require au-
Recentresearchhasshowntheusefulnessofusingcollective
7 thorisedlogins. Thesearcher,therefore,mayneedtousean
userinteractiondata(e.g.,querylogs)torecommendquery
1 Intranet search engine to locate relevant documents.
modificationsuggestionsforIntranetsearch. However,most
0 Usingcollectiveuserinteractiondata(e.g.,querylogs)for
of the query suggestion approaches for Intranet search fol-
2 querysuggestionshasbeenshownusefulforIntranetsearch
lowan“onesizefitsall”strategy,wherebydifferentuserswho
n submitanidenticalquerywouldgetthesamequerysugges- [1, 2]. Existing Intranet search approaches appear to follow
a tion list. This is problematic, as even with the same query, a“onesizefitsall”strategy. Thatisdifferentuserswhosub-
J mit the same query receive the same query suggestion list.
different users may have different topics of interest, which
However,differentusersmayhavedifferenttopicsofinterest.
9 may change over time in response to the user’s interaction
with the system. Consequently,theuserswhohavesubmittedthesamequery
] Weaddresstheproblembyproposingapersonalisedquery mayhavedifferentsearchintentions. Forexample,asociol-
R ogystudentsubmittingthequery“lecturenotes”islikelyto
suggestion framework for Intranet search. For each search
I session,weconstructtwotemporaluserprofiles: aclickuser be more interested in sociology classes than maths classes.
. Moreover, users’ interests and search intentions may be dy-
s profile using the user’s clicked documents and a query user
c profile using the user’s submitted queries. We then use the namicallyevolvingdependingontheirinteractionswiththe
[ system (e.g., clicks on documents), and when the interac-
twoprofilestore-rankthenon-personalisedquerysuggestion
1 list returned by a state-of-the-art query suggestion method tions are made, during a search session [3].
To address these issues, we propose a unified framework
v for Intranet search. Experimental results on a large-scale
topersonalisequerysuggestionsforIntranetsearch. Specifi-
0 query logs collection show that our personalised framework
cally,weusetheinteractiondataofeachuserduringasearch
5 significantly improves the quality of suggested queries.
sessiontobuilduserprofiles,whichrepresenttheuser’stop-
0 Categories and Subject Descriptors: H.3.3 [Information
icsofinterestsandmaychangeovertimeinresponsetothe
2 SystemsApplications]: InformationSearchandRetrieval
user’s interaction with the system.
0 Keywords: InteractiveIR,IntranetSearch;PersonalisedQuery
Itisworthnotingthatsearchpersonalisation(e.g.,search
. Suggestion;TemporalUserProfiles;LearningtoRank;
1 result re-ranking, query suggestion, query auto-completion,
0 etc.) has been studied extensively in the context of web
7 1. INTRODUCTION search engines [3, 8, 9, 10, 11, 12, 13]. However, little at-
1 Query suggestion is an important feature in web search tention has been paid to the same task for Intranet search.
v: engines (e.g., Bing, Google) as well as in domain-specific Moreover, personalisation methods on web search engines
i searchengines(e.g.,Intranetsearch)[1]. Querysuggestions typically construct users’ profiles using their click informa-
X helpusersquicklyrefinetheinputquerytobetter meetthe tion [3, 10, 11, 12], but less account has been taken of how
r user’sinformationneedbyrecommendingpossibletermsto users modify their queries for building the profiles.
a modify the original input query. In our proposed framework, we construct two temporal
In this paper, we focus on query suggestion for Intranet topic-based user profiles for each search session. The first
search, which is different from web search [6]. Specifically, is a click user profile based on the clicked documents. The
Intranetsearch(e.g.,universityintranets)isdomain-specific secondisaqueryuserprofile basedontheuser’squerymod-
and built to satisfy the user’s information need related to ification history within the search session. We then use the
a specific domain (e.g., the university’s document corpus). two profiles within a learning-to-rank framework to re-rank
Moreover,theIntranetmaynotbefullyindexedandaccessi- suggested queries generated by a non-personalised method
forquerysuggestiononIntranetsearch[1]. Experimentalre-
sults show that our approach helps to significantly improve
Permissiontomakedigitalorhardcopiesofpartorallofthisworkforpersonalor the query suggestion performance.
classroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributed
forprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitation
onthefirstpage.Copyrightsforthird-partycomponentsofthisworkmustbehonored. 2. QUERYSUGGESTIONFRAMEWORK
Forallotheruses,contacttheowner/author(s).
ToappearinProceedingsofCHIIR2017
2.1 BuildingTemporalUserProfiles
(cid:13)c 2017Copyrightheldbytheowner/author(s).
Eachsearchsessioncontainstwotypesofevent(i.e.,queries
ACMISBN978-1-4503-4677-1/17/03...$15.00
DOI:http://dx.doi.org/10.1145/3020165.3022129 andclicks). Givenasearchsession,weproposetobuildtwo
temporalprofilesforthespecificuser. Theseareaclickuser as P (Z|u)). The probability of a topic z given u (i.e.,
Q
profile(denotedasprofile(C)),builtusingtheuser’sclicked p (z|u)) is defined as a mixture of probabilities of z given
Q
documents,andaqueryuserprofile(denotedasprofile(Q)), query qi ∈Q as follows
built using the user’s submitted queries within the session.
(cid:88)
Clickprofilehavebeenextensivelyusedinothersearchper- pQ(z|u)= λip(z|qi) (3)
qi∈Q
sonalisation methods [3]; we expect that the query profile
will enrich the representation of the user’s search interests.
p(z|q ) is defined in Equation 2. Similar to the click user
Since a user’s interests and search intentions may change i
overtime,themorerecentlyclickeddocumentsandsubmit- profile, λi = M1 αtqi−1 is the exponential decay function of
t ,whichistheorderofthequeryq submittedbytheuser
ted queries could better represent the user’s current inter- qi i
uinthesearchsession. t =1indicatesthatq isthemost
ests. In this paper, we propose to use a decay function to qi i
recent query; M is the normalisation factor. α is the decay
capture this characteristic as in [3, 12].
parameter (0≤α≤1).
2.1.1 ExtractingTopicsfromClickedDocuments
2.2 Re-rankingSuggestedQueries
We consider that click information (e.g., clicked docu-
We use the two user profiles in a learning-to-rank mech-
ments) is a good indicator of those documents’ relevance
anismtore-rankthequerysuggestionlistreturnedbyanon-
to the user’s interests [7]. To build the user profile, we
personalisedquerysuggestionmethodproposedbyAdeyanju
use the topics discussed in the documents. We first extract
et al. [1], denoted as Adeyanju’s. Specifically, Adeyanju’s
clickeddocumentsfromtheIntranetsearch’squerylogs. Af-
first constructs a domain knowledge structure in the form
terthat,weemployLatentDirichletAllocation(LDA)[4]to
ofaconceptsubsumptionhierarchyusingboththeIntranet
automaticallyextractlatenttopics(denotedasZ)fromthe
document collection and collective users’ query logs. Next,
clicked documents (denoted as D). After training an LDA
the suggestion list is generated using the top n terms most
model using the clicked documents, we apply the model to
relevant to the query in the hierarchy.
extracttopicsfortheremainingdocumentsinthecollection.
For each input query, our re-ranking method is detailed
Finally,eachdocumentisdescribedasamultinomialdistri-
as follows
bution over the topics (denoted as P(Z|D)), in which each
(1)Wegeneratethetopnrankedsuggestedqueriesusing
topic is represented as a multinomial distribution over the
Adeyanju’s method. We denote a suggested query as q .
entire vocabulary. s
(2) We then compute similarity scores between q and
s
2.1.2 BuildingaClickUserProfile profile(C), and between qs and profile(Q). Both the sug-
gested query q and a user profile (denoted as pf which is
Werepresentthetemporalclickuserprofileasamultino- s
mial distribution over the topics as in [3]. Specifically, the either profile(C) or profile(Q)) are modelled as distribu-
user set is denoted as U. Let u be an instance of U. Let tionsovertopicsZ (Section2.1). Tomeasurethesimilarity
tDhce=use{rdcu1,indct2h,e...c,udrcnre}ntbesetahrcehsesetssoifoncl,icwkeeddedfionceumtheentcslicokf bdievtewregeenncqes(aDndJSt(cid:98)h.|e|.u(cid:99))s,erwphriochfileispaf,pwopeuulaser Jmeentsheond-Sohfanmneoan-
user profile of the user u (given the clicked document set suringthedivergence(similarity)betweentwodistributions,
Dc) as a distribution over topics Z (denoted as PC(Z|U)). to measure the similarity between qs and pf
The probability p (z|u) indicates how much the user u is
ipnrtoebraesbtielidtieinstoofpzicgCziv∈enZd. p∈C(Dz|u)asisfodlelofiwnesdasamixtureof Sim(qs|pf)=−DJS(cid:98)Q||P(cid:99)=−(cid:0)12DKL(cid:98)Q||M(cid:99)+12DKL(cid:98)P||M(cid:99)(cid:1)
ci c (4)
pC(z|u)=(cid:88)dci∈Dcλip(z|dci) (1) sHpeecreti,vQelya.nDdKPLa(cid:98)r.e||.d(cid:99)isistrtihbeutKiounlslboavcekr-Ltoepibiclesrodfiqvseargnednpcef,arned-
M = 1(Q+P). We consider the scores as the personalised
λi = N1αtdci−1 is the exponential decay function of tdci, feature2s. We also extract other non-personalised features of
whichistheorderofthedocumentdci clickedbytheuseru theinputqueryqandthesuggestedqueryqs. Table1shows
inthesearchsession. tdci =1indicatesthatdci isthemost the features extracted for re-ranking the suggestion list.
recentlyclickeddocument;N isthenormalisationfactor. α
is the decay parameter (0≤α≤1). Table1: Thepersonalisedquerysuggestionfeatures
Feature Description
2.1.3 BuildingaQueryUserProfile PersonalisedFeatures
ClickPersonalisedScore Thesimilarityscorebetweenthesuggestedqueryandtheuserclickprofile
Let Q = {q ,q ,,q } be the submitted query set of u QueryPersonalisedScore Thesimilarityscorebetweenthesuggestedqueryandtheuserqueryprofile
1 2 m Non-personalisedFeatures
in the search session. Because the number of Intranet doc- QueryRank Rankofthesuggestedqueryontheoriginallist
uments is smaller and can be assumed to change less fre- QuerySim Thecosinesimilarityscorebetweenthecurrentqueryandthepreviousquery
QueryNo TotalnumberofqueriesthathavebeensubmittedtotheSearchEngine
quently than web search engines’, we make the simplify- SuggestedQueryCosine Thecosinesimilarityscorebetweenthecurrentqueryandthesuggestedquery
SuggestedQueryJaccard TheJaccarddistancescorebetweenthecurrentqueryandthesuggestedquery
ing assumption of describing each query by the set of doc- SuggestedQueryEdit Theeditdistancebetweenthecurrentqueryandthesuggestedquery
uments that contain all the query words, denoted as D = SuggestedQueryLevenshtein TheLevenshteindistancebetweenthecurrentqueryandthesuggestedquery
qi SuggestedQueryPreUsed Whetherthesuggestedquerywasusedbytheuserinthesamesearchsession?
{d ,d ,...,d }. Then,eachsearchqueryq (giventhedoc-
umi1enti2set Dik) is modelled as a distributioin over topics Z (3)Afterextractingthequeryfeatures,tore-rankthetop
(denotedasPqi(Z|q )). Theprobabilityofatopicz∈Z given n suggested queries, we employ LambdaMART [5] to train
i
q ∈Q (i.e., p(z|q )) is defined as a mixture of probabilities ranking models. Among many learning-to-rank algorithms,
i i
of z given a document d ∈D as follows LambdaMARTisregardedasoneofthebest-performingal-
ij qi
gorithmsandhasbeenchosenasthebaselearningalgorithm
p(z|qi)=(cid:88)dij∈Dqi |D1qi|p(z|dij) (2) in various recent approaches to search personalisation [3].
|D | is the size of the document set D .
qi qi
Wethenmodelthequeryuserprofileoftheuseru(given
the query set Q) as a distribution over topics Z (denoted
3. EXPERIMENTALMETHODOLOGY tailed in Section 2.1. The number of topics (i.e., 300 in our
experiments) is decided by using a held-out validation set
3.1 EvaluationMethodologyandDataset whichconsistsof10%ofallclickeddocuments. Theselected
Evaluation methodology Forevaluation,weuseAuto- number of topics is the one that gives the lowest perplexity
Eval, an automated evaluation framework, which measures value. The decay parameter α for the two user profiles is
the performance of query suggestions automatically based set to 0.95 as in [3]. The ranking function is learned using
on the actual query logs of an Intranet search [2]. For each LambdaMART [5]. We used the default setting for Lamb-
querysuggestionlist,weassignapositivelabelforasugges- daMART’s prior parameters1.
tionifitisanactualrefinement,whichisthenextsubmitted EvaluationmetricsTheevaluationisbasedonthecom-
queryinthesearchsession,andthereisatleastoneuserclick parison between our personalised approach and the base-
onretrievedresultsaftertherefinement. Inotherwords,we lines. Weusefourevaluationmetrics: MeanAveragePreci-
interprettheuserclickafterareformulationasthecriterion sion(MAP),Precision(P@k),MeanReciprocalRank(MMR),
of a relevant suggestion. The remainder of the suggestion and Normalized Discounted Cumulative Gain (nDCG@k).
listisassignednegative(irrelevant)labels. Weusetherank
positions of the positively labelled queries as an approxi- 4. RESULTS
mation of the ground truth to evaluate the performance of
query suggestions before and after re-ranking. 4.1 OverallPerformance
We also follow the experimental methodology in [1], that
Table 3 shows promising results when user profiles are
is,themodelisevaluatedcontinuouslyatperiodicintervals.
used to personalise the query suggestion list. We can see
Specifically, we use the logs in week i for training the re-
that,evenusingonlytheclickuserprofile,theClick method
ranking model and the following week i+1 for testing the
hasledtoanimprovementof7.22%onMAPoverAdeyanju’s
trained model; where, in our experiments, 1 ≤ i ≤ w, the
method. The combination of query and click profiles (i.e.,
number of weeks in the test period.
Ours)achievesthehighestimprovementof10.97%overAde-
Dataset The dataset used in our experiments contains
yanju’s intermsofMAPscore. Theimprovementsindicate
large-scale query logs collected from the search engine in-
thatpersonalisationhelpsimprovethequerysuggestionper-
stalledattheWebsiteoftheUniversityofEssexduringthe
formance. TheimprovementsoverAdeyanju’s areallsignif-
twoyearscovering1January2012-31December2013. Each
icant (paired t-test, p < 0.01).
logsample contains a session identifier, the event type (i.e.,
a query or a click), an auto-increment id, the event content
Table 3: Overall performance of the methods. %rel denotes the
(i.e., query text, click URL), and the event time-stamp. relativeimprovementoverAdeyanju’s.
We apply a simple pre-processing step to remove single
Model MAP P@1 P@5 MRR@10 nDCG@5 nDCG@10
event search sessions because, with those sessions, it is not Adeyanju’s 0.5440 0.4113 0.1823 0.5447 0.5714 0.6000
possible to determine whether the user found the required Click 0.5833 0.4271 0.1981 0.5839 0.6193 0.6583
%rel +7.22% +3.84% +8.67% +7.22% +8.38% +9.73%
information. We also remove those queries whose positive Ours 0.6037 0.4526 0.2026 0.6043 0.6413 0.678
label set is empty from the dataset. We then analyse the %rel +10.97% +10.04% +11.14% +10.94% +12.23% +13.02%
remainder of the query logs. Table 2 shows basic statistics. In the comparison between personalisation methods (i.e.,
Table2: Basicstatisticsoftheevaluationsearchlogs Ours and Click), Table 3 shows that using both the click
andqueryuserprofiles(i.e.,Ours)significantlyimprovesthe
Item 2012 2013 Total
#searchsessions 397,461 338,391 735,804 suggestionqualityovertheClick baseline(p<0.01). Inter-
#events 1,263,179 1,083,992 2,347,171 estingly, our method produces a significantly better quality
#events/session 3.22 3.25 3.23 ofthefirstqueryinthesuggestionlistwiththeimprovement
#queries 757,645 659,284 1,416,929
of5.97%onP@1overClick. TheimprovementsofOurs over
#query/session 1.91 1.95 1.93
#clickedurl 505,534 424,708 930,242 Click also indicate that the query user profile is important
#clicks/session 1.27 1.26 1.26 inthequerysuggestiontask,especiallyforthequalityofthe
3.2 ExperimentalSettings first suggested query.
OurpersonalisationmethodandbaselinesWename 4.2 PerformanceonDifferentQueryPositions
our proposed re-ranking model as Ours. We choose two
With more submitted queries and clicked documents, we
baselinestocompareourworkagainst. Thefirst baseline is
areabletobuildricheruserprofiles. Inthisexperiment,we
Adeyanju’s method[1],whichwereimplementedtogenerate
aim to study whether the position of a query in a search
the original suggestion list for re-ranking.
session has any effect on the performance of personalised
We use the session-based approach proposed by Bennett
query suggestion. For each search session, we label queries
etal. [3]asthesecondbaseline. Specifically,inthebaseline,
bytheirpositionsduringthesession. Becausetherearefew
we use only the click user profile (i.e., profile(C)) together
sessions containing more than three queries (i.e., 7.65% of
with the non-personalised features detailed in Table 1 to
sessions in the query logs), we label the first three queries
re-rank the suggestion list. We name the baseline as Click.
fromonetothreeaccordingtotheorderofsubmissioninthe
It is worth noting that Adeyanju’s is a non-personalised
search session; the remaining queries are labelled as ≥4.
approachandachievedagoodperformanceofquerysugges-
Itisworthnotingthatforthefirstquery,wecannotbuild
tiononIntranetsearch[1]. Click isapersonalisedapproach
the click user profile because there is no previously clicked
and achieved good performances in web search personalisa-
document. However,wecanstillbuildthequeryuserprofile
tion [3]. Moreover, instead of the session-based approach
forthefirstquery. Weshowtheimprovementinperformance
by Bennett et al. [3] as our personalised baseline we could
ofthepersonalisedmethodsoverAdeyanju’sintermofMAP
alternatively have used Shokouhi et al. [9] or Shokouhi [8].
LDA & LambdaMART We train an LDA model on 1Number of leaves = 10, minimum documents per leaf =
theclickeddocumentsextractedfromthequerylogs,asde- 200, number of trees = 100 and learning rate = 0.15
Figure1: RelativeMAPimprovementsoverAdeyanju’s withdif- Figure2: RelativeMAPimprovementsoverAdeyanju’s withdif-
ferentquerypositions. ferentquerylengths.
metric with different query positions in Figure 1. Here the Ours over Click are 6.7% and 18.2% on the query length 3
statistical significance is verified by t-test (p < 0.01). For and the query length ≥4, respectively.
the first query within a search session, our method, which
can use only the query user profile, significantly improves 5. CONCLUSIONS
thequerysuggestionperformanceoverAdeyanju’s. Itagain
Inthispaper,weproposedapersonalisedquerysuggestion
confirms the effectiveness of the query information on per-
frameworkandshowedhowitperformedonIntranetsearch.
sonalised query suggestion for Intranet search.
Webuilttwosession-specifictemporaluserprofiles,aquery
From the second query, we can build both the query and
userprofileusingthesubmittedqueries,andaclickuserpro-
click user profiles. One can see that the higher position
fileusingtheclickeddocuments. Wethenextractedtheper-
of a query is, the larger improvement in performance the
sonalisedfeaturesusingthetwoprofilesandcombinedthem
personalised query suggestion can be achieved. Specifically,
withnon-personalisedfeaturestolearnarankingmodelus-
from the query with high positions (i.e., ≥4), the improve-
ing LamdaMART. Finally, we used the ranking model to
ments of Click and Ours are 11.37% and 18.22%, respec-
re-rankthequerysuggestionlistreturnedbyastate-of-the-
tively. Figure 1 also shows that Ours outperforms Click
art query suggestion approach for Intranet search.
significantly with the improvements of at least 3.45% (p <
Experimental results on a large-scale query log dataset
0.01). It indicates that richer user profiles (by observing
collectedfromauniversityintranetsearchengineshowthat
more clicked documents and submitted queries during the
personalisation significantly improved the query suggestion
search session) help achieve better query suggestion perfor-
performance. Using both the click user profile and query
mances. The findings offer future research directions that
userprofileachievedthehighestperformanceindicatingthat
use user profiles which go beyond single sessions
personalisedquerysuggestionforIntranetsearchshouldtake
4.3 PerformanceonDifferentQueryLengths into account both click and query information. Moreover,
thepositiveimpactofpersonalisedquerysuggestionsismore
Thequerylengthisdefinedbythenumberofwordsinthe
pronouncedwithlongerqueriesandqueriessubmittedlater
query (e.g., the query“University webmail”has the length
within a session.
of 2). The length of a query might give an indication as
to how specific the information need of an individual user
is (i.e., a longer query can typically be assumed to reflect a 6. REFERENCES
[1] I.A.Adeyanjuetal.Adaptationoftheconcepthierarchy
morespecificinformationneed). Inthisexperiment,weaim
modelwithsearchlogsforqueryrecommendationon
to show the impact of personalisation on query suggestion
intranets.InSIGIR’12,2012.
with different query lengths. We label each query by its
[2] M.-D.Albakouretal.Autoeval: Anevaluation
length,whichisthenumberofwordsinthequery(i.e.,from methodologyforevaluatingquerysuggestionsusingquery
one to three and ≥ 4 words because there are few queries logs.InECIR’11,2011.
containing more than three words (i.e., 5.7% of queries in [3] P.N.Bennettetal.Modelingtheimpactofshort-and
the query logs)). long-termbehavioronsearchpersonalization.InSIGIR’12,
Figure 2 shows the improvement in performance of Click 2012.
[4] D.M.Bleietal.Latentdirichletallocation.J. Mach.
and Ours over Adeyanju’s in term of MAP with different
Learn. Res.,2003.
querylengths. Weseethatpersonalisationmethodsachieve
[5] C.J.C.Burgesetal.Learningtorankwithnon-smooth
significantly better performances than the non-personalised
costfunctions.InNIPS’06,2006.
method does (p < 0.01). Even for short queries (length 1
[6] D.Hawking.Enterprisesearch.InModern Information
and 2) which tend to be more generic, the Click and Ours Retrieval, 2nd Ed.2010.
methods outperform Adeyanju’s method with the improve- [7] T.Joachimsetal.Accuratelyinterpretingclickthrough
ment of more than 6.11% and 9.09%, respectively. We see dataasimplicitfeedback.InSIGIR’05,2005.
that the longer a query is, the higher improvement person- [8] M.Shokouhi.Learningtopersonalizequery
alisedmethodscanachieve. Specifically,withalongerquery auto-completion.InSIGIR’13,2013.
(i.e., with length ≥ 4), the Click and Ours methods yield [9] M.Shokouhietal.Fightingsearchengineamnesia:
Rerankingrepeatedresults.InSIGIR’13,2013.
the highest improvements, i.e., 29.82% and 53.85%, respec-
[10] T.Vuetal.Improvingsearchpersonalisationwithdynamic
tively. This indicates that a longer query would also get
groupformation.InSIGIR’14,2014.
more benefit from personalisation.
[11] T.Vuetal.Modellingtime-awaresearchtasksforsearch
Figure 2 also indicates that by combining the query user personalisation.InWWW’15 Companion,2015.
profilewiththeclickuserprofile,Ourssignificantlyimproves [12] T.Vuetal.Temporallatenttopicuserprofilesforsearch
the query suggestion performance over Click (p < 0.01). personalisation.InECIR’15,2015.
Moreover,theimprovementsarelargerwiththelongerqueries [13] T.Vuetal.Searchpersonalizationwithembeddings.In
(i.e., with length ≥ 3). In particular, the improvements of ECIR’17,2017.