On Low Overlap Among Search Results of Academic Search Engines

0.17 MB·

by  Anasua Mitra

On Low Overlap Among Search Results of Academic
Search Engines
Anasua Mitra, Amit Awekar
IndianInstituteofTechnology,Guwahati,Assam,India
anasua.mitra, [email protected]
ABSTRACT significant role in deciding which research papers are read
7 Number of published scholarly articles is growing exponen- byresearchers. ThereforeitisnecessarytostudyASEssep-
1 tially. To tackle this information overload, researchers are arately from general search engines.
0 increasingly depending on niche academic search engines.
2 Recentworkshaveshownthattwomajorgeneralwebsearch 2. RELATEDWORK
engines: Google and Bing, have high level of agreement in
n Overlap in the coverage and search results of web search
their top search results. In contrast, we show that vari-
a enginesiswellstudied. Duringmid90s,multiplesearchen-
ous academic search engines have low degree of agreement
J gines such as AltaVista, Excite, and Lycos were competing
among themselves. We performed experiments using 2500
0 with each other. These search engines covered only about
queries over four academic search engines. We observe that
1 3 to 4% of the web[5] and overlap in their search results
overlap in search result sets of any pair of academic search
was as low as 1.4%[2]. By the year 2005, major search en-
engines is significantly low and in most of the cases the
] gines were indexing more than 60% of the web, but overlap
R search result sets are mutually exclusive. We also discuss
in their results was still only about 11%. Currently, there
implications of this low overlap.
I are only two major web search engines: Google and Bing.
.
s Recent work by Agrwal et. al. has shown that both these
c CategoriesandSubjectDescriptors
web search engines have high level of agreement between
[
H.3.3 [Information Search and Retrieval]: Search pro- them[1]. This convergence can be attributed to multiple
1 cess factors such as high coverage of both search engines, matu-
v rityinrankingalgorithms,andpossibilityofcopyingresults
7 Keywords fromeachother. Bibliographicdatasetshavebeenshownto
1 have low overlap in their coverage[4]. Various works have
Web mining, Scholarly data, Search engine comparison
6 triedtopredictcoverageofASEs[6]. Howevertothebestof
2 ourknowledge,thereisnoexistingworkthatsystematically
1. INTRODUCTION
0 studies overlap in the search results of ASEs.
. Number of published research papers approximately dou-
1
bles after nine years[3]. This robust trend which is consis-
0 3. OURWORK
tentlyobservedafterthesecondworldwarcanbeattributed
7
to many factors. Sheer number of researchers is increasing Motivation for systematically studying agreement among
1
with improvements in academic and research infrastructure multiple ASEs primarily came from our own experience of
:
v in countries such as China and India. Researchers increas- usingASEsforresearchliteraturereview. Almostmutually
i inglyfacepublish-or-perishpardigminresearchuniversities exclusiveresultsobtainedfrommultipleASEs,promptedus
X
and laboratories. Web based systems for paper submission, toverifywhetheritwasjustacoincidence. Thisobservation
r reviewing and publication have significantly brought down wasinsharpcontrastwithrecentworkbyAgrawalet. al.[1]
a
the timespan and cost of publishing a paper. As a result, where they observed strong agreement among web search
large number of low quality and even predatory research engines.
conferences and journals have emerged.
We queried four ASEs: Google Scholar (GS)2, Semantic
It is impossible to manually keep track of all relevant lit- Scholar(SS)3,MicrosoftAcademic(MA)4,andScopus(SC)5.
erature for any research topic. For example, in 2016 more MainreasonforchoosingtheseASEswasthattheyarepop-
than2000paperswerepublishedwithwords“deeplearning” ular in computer science and engineering domain. We col-
mentionedinthetitle1. Actualnumberofpaperspublished lected over 2300 query terms from 2012 ACM Computing
in 2016 on deep learning will be far more than that. In re- Classification System6. This system arranges various com-
sponse to this information overload, many academic search puter science topics into a poly-hierarchy ontology. It suf-
engines(ASEs)havebeendevelopedtofindappropriatere-
2http://scholar.google.com/
searcharticles. TheseASEsdifferinmultipleaspects: broad
3http://www.semanticscholar.org/
vs. specifictopiccoverage,webcrawlvs. curateddatabases
4https://academic.microsoft.com/
as source of data, commercial vs. non profit. ASEs play
5https://www.scopus.com/home.uri
1Scource: Google Scholar 6https://www.acm.org/publications/class-2012
Figure 1: Overlap in Search Result Sets of Academic Search Engines
ficiently covers broad spectrum of topics in computer sci- scoresiswideforpairsconsistingofGS,MA,andSS.Where
ence from coarse to fine granularity. We also collected over as pairs involving SC, the blue box almost flattens to line
200keywordsfrompaperspublishedinACMSIGKDD2016 overlapping with minimum value of zero.
conference.
4. CONCLUSIONANDFUTUREWORK
We sent these 2500 queries to all four selected ASEs. For
Unlike web search engines, ASEs do not have one or two
each ASE, we considered results only from the first page as
dominantplayers. Thishasledtomultiplesystemsthatdif-
usersseldomgobeyondthefirstpageofsearchresults. Var-
ferincoverageofresearchliteratureandrankingalgorithms.
ious ASEs return different number of results on their first
Therefore overlap among search results of ASEs is signfi-
page. WelookedattopeightresultsaseachASEreturnsat
cantly low. As a result, users of ASEs have to look across
least eight results on the first page. We ignored the order
multiple ASEs to find relevant research literature. We are
in results and treated them as sets. If we consider order of
working on extending this study in three ways. First, we
results,thenitwillfurtherdrivedownthesimilarityscores.
areincludingmoreASEsinthecomparison. Second,weare
WecomputedsimilaritybetweenanytwosetsusingtheJac-
using more diverse queries related to other subjects apart
cardsimilarity,thatisratioofsizeofintersectionandunion.
from just computer science. Third, we want to compare
ASEs based on quality of search results.
Please refer to Figure 1. X axis represents pairs of various
ASEs. TherearetotalsixpairsasweconsideredfourASEs.
5. REFERENCES
YaxisrepresentsJaccardsimilarityforeachpairusingbox-
[1] Agrawal, R., Golshan, B., and Papalexakis, E. A
plots. Using2500queries,weobtained2500similarityscores
study of distinctiveness in web results of two search
for each ASE pair. Out of these scores, the figure depicts
engines. In International Conference on World Wide
maximum (top black line), minimum (bottom black line),
Web (2015), pp. 267–273.
median(redline),25percentile,and75percentile(bluebox)
scores for each pair. [2] Bharat, K., and Broder, A. A technique for
measuring the relative size and overlap of public web
For all pairs, minimum score is always zero. It means that
search engines. Computer Networks and ISDN Systems
we have at least one query per pair such that their result
30, 1 (1998), 379–388.
sets are mutually exclusive. For all pairs, median score is
alsozeroindicatingthatformostofthequeriessearchresult [3] Bornmann, L., and Mutz, R. Growth rates of
sets of ASEs are mutually exclusive. For all 2500 queries, modern science: A bibliometric analysis based on the
intersection set of all four ASEs considered together was number of publications and cited references. Journal of
always empty. In other words, for each query, no research the Association for Information Science and Technology
article appears in the top results list of all four ASEs. This 66, 11 (2015), 2215–2222.
showsstrongdiagreementamongASEs. Ourqueriescovered
[4] Hood, W. W., and Wilson, C. S. Overlap in
topics of coarse as well as fine granularity. However, we
bibliographic databases. Journal of the American
did not see any correlation between topic granularity and
Society for Information Science and Technology 54, 12
overlap in search results.
(2003), 1091–1103.
GSandMAhavecomparablecoverageofresearchliterature [5] Lawrence, S., and Giles, C. L. Searching the world
(160millionand150milliondocumentsrespectively). These
wide web. Science 280, 5360 (1998), 98–100.
twoASEscovermultipletopicsincludingcomputerscience.
While SS covers only about 10 million documents, mostly [6] Orduna-Malea, E., Ayllo´n, J. M.,
from computer science. Compared to these three ASEs, SC Mart´ın-Mart´ın, A., and Lo´pez-Co´zar, E. D.
has far small share of computer science documents. All our Methods for estimating the size of google scholar.
queries are from computer science domain.Therefore blue Scientometrics 104, 3 (2015), 931–949.
box comprising of 25 percentile and 75 percentile similarity