Table Of ContentGehannoetal.BMCMedicalInformaticsandDecisionMaking2013,13:7
http://www.biomedcentral.com/1472-6947/13/7
RESEARCH ARTICLE Open Access
Is the coverage of google scholar enough to be
used alone for systematic reviews
Jean-François Gehanno1,2*, Laetitia Rollin1,2 and Stefan Darmoni2
Abstract
Background: Insearches for clinical trials and systematic reviews, it is said that Google Scholar (GS) should never
be used in isolation, butin addition to PubMed, Cochrane, and other trusted sources of information. We therefore
performed a study to assess thecoverage of GS specifically for thestudies included insystematic reviews and
evaluate if GS was sensitive enough to be used alone for systematic reviews.
Methods: All the original studies included in29 systematic reviews publishedin theCochraneDatabase Syst Rev or
inthe JAMA in 2009 were gathered ina goldstandard database. GS was searched for all thesestudies one by one
to assess thepercentage of studies which could have been identified bysearching only GS.
Results: Allthe 738 original studies included inthegold standard database were retrieved inGS (100%).
Conclusion: The coverage of GS for thestudies included in thesystematic reviews is 100%. If theauthors of the
29 systematic reviews had used only GS,no reference would have been missed. With some improvementin the
research options,to increaseitsprecision, GS could become the leading bibliographic database inmedicineand
could be used alone for systematic reviews.
Keywords: Bibliometrics, Google scholar,Informationretrievalmethods, Systematic reviews
Background Doctors are encouraged to consult GS for browsing
The release of the beta version of Google Scholar (GS) and serendipitous discovery, not for literature reviews
(http://scholar.google.com) in November 2004 generated [1]. In searches for clinical trials and systematic reviews,
much media coverage and academic commentary. It has it is said that GS should never be used in isolation, but
been met with both enthusiasm and criticism but Google in addition to PubMed, Cochrane, and other trusted
and GS now lead more visitors to many biomedical sources of information [1]. Many studies have demons-
journal websites than does Medline via its PubMed inter- trated that a single search engine does not capture all of
face[1-3]. the available articles, and using two or more databases
GS searches retrieve results that include scholarly li- providesgreater coverageofallpossible citations[6-17].
terature citations as well as peer-reviewed publications, Nevertheless, the coverage of GS is increasing and,
theses, books, abstracts, and other articles from acade- despite the fact that it is said to be not exhaustive, is it
mic publishers, professional organizations, and preprint exhaustive enough for the studies that are considered of
repositories, universities, and other scholarly organiza- enough quality orrelevancefor systematic reviews[18].
tions. Therefore, GS is able to retrieve more types of Therefore, the objective of this study was to assess the
literature compared with medical literature database re- coverage of GS, and its potential recall, specifically for
trieval search engines, like PubMed [4]. GS is also able suchstudies,andthereforetoassessifthisdatabasecould
to identify some of the references of PubMed, but not beusedaloneforsystematicreviews.
all[5].
Methods
*Correspondence:[email protected] The first step aimed at identifying a subset of studies
1InstituteofOccupationalHealth,RouenUniversityHospitalandUniversityof
selected by experts to be included in systematic re-
Rouen,1ruedeGermont,76000,Rouen,France
2CISMeF-TIBS-LITISEA4108,RouenUniversityHospital,Rouen,France views. We searched Medline in December 2009 for the
©2013Gehannoetal.;licenseeBioMedCentralLtd.ThisisanOpenAccessarticledistributedunderthetermsoftheCreative
CommonsAttributionLicense(http://creativecommons.org/licenses/by/2.0),whichpermitsunrestricteduse,distribution,and
reproductioninanymedium,providedtheoriginalworkisproperlycited.
Gehannoetal.BMCMedicalInformaticsandDecisionMaking2013,13:7 Page2of5
http://www.biomedcentral.com/1472-6947/13/7
Table1RecallofGooglescholarforthe29systematicreviews
Sourceofthe Titleofthesystematicreview Numberof Numberof Numberof
systematicreview databasessearched studiesincluded studiesfound
bytheauthors inthereview inGoogleScholar
CochraneLibrary Antidepressantsversusplacebofordepression 8 14 14
inprimarycare
CochraneLibrary Artemisinin-basedcombinationtherapyfortreating 6 49 49
uncomplicatedmalaria
CochraneLibrary Briefinterventionsforheavyalcoholusersadmitted 5 11 11
togeneralhospitalwards
CochraneLibrary CombinedDTP-HBV-HIBvaccineversusseparately 3 18 18
administeredDTP-HBVandHIBvaccinesforprimary
preventionofdiphtheria,tetanus,pertussis,hepatitis
BandHaemophilusinfluenzaeB(HIB)
CochraneLibrary ErythropoietinorDarbepoetinforpatientswith 3 39 39
cancer–meta-analysisbasedonindividualpatientdata
CochraneLibrary Greentea(Camelliasinensis)forthepreventionofcancer 7 51 51
CochraneLibrary Incentivespirometryforpreventionofpostoperative 5 11 11
pulmonarycomplicationsinupperabdominalsurgery
CochraneLibrary Interventionstopreventoccupationalnoiseinduced 10 20 20
hearingloss
CochraneLibrary Non-pharmacologicalinterventionsforassistingthe 7 17 17
inductionofanaesthesiainchildren
CochraneLibrary Oralironsupplementationforpreventingortreating 5 68 68
anaemiaamongchildreninmalaria-endemicareas
CochraneLibrary Pharmacotherapyforanxietydisordersinchildren 4 25 25
andadolescents
CochraneLibrary Singledoseoralflurbiprofenforacutepostoperative 4 11 11
paininadults
CochraneLibrary Theeffectsofantimicrobialtherapyonbacterialvaginosis 5 24 24
innon-pregnantwomen
CochraneLibrary Therapeuticinterventionsforsymptomatictreatmentin 4 20 20
Huntington’sdisease
JAMA Acute-onsetfloatersandflashes:isthispatientatriskfor 2 17 17
retinaldetachment?
JAMA Adiponectinlevelsandriskoftype2diabetes:asystematic 3 14 14
reviewandmeta-analysis
JAMA Allogeneicstemcelltransplantationforacutemyeloid 3 17 17
leukemiainfirstcompleteremission:systematicreview
andmeta-analysisofprospectiveclinicaltrials
JAMA Aspirinforthepreventionofcardiovasculareventsin 4 15 15
patientswithperipheralarterydisease:ameta-analysis
ofrandomizedtrials.
JAMA Bedbugs(Cimexlectularius)andclinicalconsequencesof 2 49 49
theirbites.
JAMA Cancersurvivorsandunemployment:ameta-analysisand 5 24 24
meta-regression.
JAMA Cardiorespiratoryfitnessasaquantitativepredictorofall-cause 2 32 32
mortalityandcardiovasculareventsinhealthymenand
women:ameta-analysis.
JAMA CombinedcorticosteroidandantiviraltreatmentforBellpalsy: 6 17 17
asystematicreviewandmeta-analysis.
JAMA Corticosteroidsinthetreatmentofseveresepsisandseptic 4 19 19
shockinadults:asystematicreview
JAMA Diagnosticperformanceofcomputedtomography 3 20 20
angiographyinperipheralarterialdisease:asystematic
reviewandmeta-analysis
Gehannoetal.BMCMedicalInformaticsandDecisionMaking2013,13:7 Page3of5
http://www.biomedcentral.com/1472-6947/13/7
Table1RecallofGooglescholarforthe29systematicreviews(Continued)
JAMA Interactionbetweentheserotonintransportergene 3 14 14
(5-HTTLPR),stressfullifeevents,andriskofdepression:
ameta-analysis.
JAMA Lipoprotein(a)concentrationandtheriskofcoronary 2 36 36
heartdisease,stroke,andnonvascularmortality.
JAMA PredictivevalueoffactorVLeidenandProthrombin 5 46 46
G20210Ainadultswithvenousthromboembolismandin
familymembersofthosewithamutation.Asystematic
review
JAMA Sexualabuseandlifetimediagnosisofsomaticdisorders: 9 22 22
asystematicreviewandmeta-analysis
JAMA Treatmentoffibromyalgiasyndromewithantidepressants: 6 18 18
ameta-analysis.
Total 738 738(100%)
systematic reviews published in the JAMA or the Co- studies. All the 738 studies were identified in GS, leading
chrane Library. For the JAMA, we used the most specific to100%coverage.
search string proposed by Montori et al., with limits for ThedetailedresultsarepresentedinTable 1.
the years 2008 and 2009 [19]. For the Cochrane Library, As a side result, we discovered that a striking number
we examined all the systematic reviews published in the of bibliographic references included major errors, i.e. er-
CochraneDatabaseSystRev.2009Jul8;(3). rors that involve the data elements by which references
We excluded the systematic reviews using less than 2 are searched by users in Medline [20]. Overall, 10 refe-
bibliographic databases in their search and those which rences contained at least one major error, some of them
restrictedthesearchtoEnglish language studies. containingupto3majorerrors.
The gold standard database was then built by gathe- Someofthereviewsconcentrated these citation errors.
ring all the studies included in the systematic reviews we For example, among the 24 references included in the
selected, excluding abstracts and personal communica- Cochrane review " The effects of antimicrobial therapy
tions. We considered Gray literature (i.e. written mater- on bacterial vaginosis in non-pregnant women", 5 con-
ial that is not published commercially or is not generally tained atleastone majorerror.
accessible) as a specific subset, but we included these re-
ferencesinthe goldstandard database. Discussion
GS was searched for each reference, one by one, by Performing systematic reviews is a complex and time
searching with the title ofeach of the studies included in consuming task, because of the body of literature to be
the gold standard database. Recall (i.e. the proportion of searched and the high number of databases that must be
studies retrieved from the database) of GS were com- used, considering that no one of them is considered
puted for each review published in the Cochrane Library exhaustive. The use of GS is increasing, as well as its
ortheJAMA. coverage, and we wanted to assess if this coverage is
high enough to be used alone in systematic reviews.
Results GS allowed to retrieve 100% of the studies included in
Overall, 14 reviews from the Cochrane library and 15 the systematic reviews we studied, and which covered
reviews from the JAMA were included. To identify all many differentfieldsofmedicine.
the possible relevant studies, each systematic review Although GS does not cover all the medical literature,
from the Cochrane Library and from the JAMA had we therefore observed that its coverage of the studies of
searched between 3 and 10 (mean: 5.4) and between 2 sufficient quality or relevance to be included in a sys-
and 9 (mean : 4) different databases, respectively. All of tematic review was complete. In other words, if the
them searched Medline and 17 mentioned to have also authors of these 29 systematic reviews had used only
scannedthe reference listofthe studiestheyincluded. GS, they would have obtained the very same results.
The 29 systematic reviews had included 755 original The validity of our gold standard database could ne-
studies. Among them, 733 were published in peer-re- vertheless be questioned. To identify the studies that
viewed journals and 5 were detailed only in document worth to be included in a systematic review, we relied
belonging to the gray literature. The 18 remaining stu- on the works of the experts used as reviewer in the sys-
dies were referenced only as an abstract or as personal tematicreviewsweincluded,sinceallofthemusedatleast
communication and were therefore not included in the 2 independent reviewers. Furthermore, we excluded from
goldstandarddatabase,whichincludedfinally738original our gold standard database personal communications,
Gehannoetal.BMCMedicalInformaticsandDecisionMaking2013,13:7 Page4of5
http://www.biomedcentral.com/1472-6947/13/7
because they cannot be retrieved by any database, and resultsinachronologicalorder,GSplacesmorerelevance
abstracts because it has been clearly demonstrated that on articles that are cited most often. Therefore, the cita-
such abstracts often display non-valid results [21,22]. tions located are reportedly biased toward older literature
Considering the methods used by the authors of the sys- [26,27].Thislastpointcanalsobeviewedasanadvantage,
tematic reviews we selected, the use of at least two in- since it allows to identify quickly landmark articles, i.e.
dependent reviewers to select relevant articles in these articles of importance in a field. Yet, when comparing
reviews, the high number of databases searched and the searches with PubMed and Google Scholar by evalu-
absence of restriction to English studies in each of them, ating the first 20 articles recovered for four clinical
we can also assume that, for each topic covered, all the questions for relevance and quality, Nourbakhsh and
relevantstudieswereidentified.Therefore,wecanassume coll.demonstratedthatGSprovidedmorerelevantresults
that our gold-standard database really included all the that PubMed, although the difference was not significant
studies of sufficient quality and relevant to the topics co- (p=0.116)[28].
veredbythesystematicreviews,andonlythem. GS has been reported to be less precise than PubMed,
We chose tostudythe systematic reviews published by since it retrieves hundreds or thousands of documents,
the JAMA and Cochrane because they usually don’t res- most of them being irrelevant [29,30]. Nevertheless, we
trict their search to English literature and they use more should not overestimate the precision of PubMed in real
than one database to perform the search, which is not life since Precision and recall of a search in a database is
the case of most of the systematic reviews published by highly dependent on the skills of the user [10]. Many of
theAnnalsofInternalMedicine,for example. them overestimate the quality of their searching per-
AlthoughtherecallofGSwas100%,theamount ofin- formance, and experienced reference librarians typically
formationdelivered byGS washeterogeneous.Yet,some retrieve about twice as many citations as do less experi-
of the studies were only identified as "citations", which enced users [31,32].
means that GS only displayed the authors, the title of Althoughthiswasnotthepurposeofourstudy,wetried
the article and the name, year and pages of the journals. toassesstheprecisionofGSforsomeoftheclinicalques-
Thiscanbeconsideredasinsufficient,buttraditionalbio- tionsthatwerestudiedbythesystematicreviews.
medical databases (such as Medline or Embase) do the For example, searching for "(Erythropoietin or Darbe-
same for old articles or for articles published in another poetin) cancer" in GS gave a recall of 100% and a preci-
language that English. Furthermore, this is exactly the sion of 0.1% (36,630 articles found, for 36 included in
same situation when authors of systematic reviews per- the systematic review). In GS, the search string "(depres-
form hand searching in the reference list of selected ar- sion treatment placebo antidepressant) ("general practice"
ticles.Therefore,weconsideredvalidtoincludethesehits OR "Primary care")" identified 16100 articles, leading to a
aspositiveresults. recallof100%andaprecisionof0.09(14articlesincluded
This 100% coverage of GS can be seen as amazing, inthecorrespondingsystematicreview).
since no single database is supposed to be exhaustive,
even for good quality studies. For example, the recall Conclusion
ratios of Medline for randomized control trials (RCTs) In conclusion, the coverage of GS is much higher than
only stand between 35% and 56% [23,24]. Since GS ac- previously thought for high quality studies. GS is highly
cesses only 1 million of the some 15 million records at sensitive, easy to search and could be the first choice for
PubMed, how can our results be explained? In fact, systematic reviews or meta-analysis. It could even be
through agreements with publishers, GS accesses the used alone. It just requires some improvement in the
“invisible” or “deep” Web, that is, commercial Web sites advanced search features to improve its precision and to
the automated “spiders” used by search engines such as becometheleading bibliographicdatabaseinmedicine.
Google cannot access. Furthermore, we observed in our
study that most of the articles indentified by GS were Competinginterests
Theauthorsdeclaretheyhavenocompetinginterest.
found directly on the publishing journal web-sites, and
not onthePubMedweb-site.
Authors’contribution
Nevertheless, while its advantages are substantial, GS JFGconceivedofthestudy.JFGandLRcollectedthedata.JFG,LRandSJD
is not without flaws. The shortcomings of the system analyzedthedataanddraftedthemanuscript.Allauthorsreadand
approvedthefinalmanuscript.
and its search interface have been well documented in
the literature and include lack of reliable advanced search Received:20August2012Accepted:4January2013
functions (e.g. no MeSH term subheading search func- Published:9January2013
tion), lack of controlled vocabulary, lack of a “similar
pages”feature,andissuesregardingscopeofcoverageand R1.eferGeinuscteinsiD:HowGoogleischangingmedicine.BMJ2005,331:1487–1488.
currency [4,5,25]. Furthermore, whereas PubMed displays 2. LindbergDA:Searchingthemedicalliterature.NEJM2006,354:2393.
Gehannoetal.BMCMedicalInformaticsandDecisionMaking2013,13:7 Page5of5
http://www.biomedcentral.com/1472-6947/13/7
3. WangY,HowardP:GoogleScholarUsage:AnAcademicLibrary's 29. AndersME,EvansDP:ComparisonofPubMedandGoogleScholar
Experience.JWebLibrarianship2012,6(2):94–108. literaturesearches.RespirCare2012,55:578–583.
4. FreemanMK,LauderdaleSA,KendrachMG,WoolleyTW:GoogleScholar 30. MastrangeloG,FaddaE,RossiCR,ZamprognoE,BujaA,CegolonL:
versusPubMedinlocatingprimaryliteraturetoanswerdrug-related Literaturesearchonriskfactorsforsarcoma:PubMedandGoogle
questions.AnnPharmacother2009,43:478–484. Scholarmaybecomplementarysources.BMCResNotes2010,3:131.
5. ShultzM:ComparingtestsearchesinPubMedandGoogleScholar. 31. HershWR,HickamDH:Howwelldophysiciansuseelectronicinformation
JMedLibrAssoc2007,95:442–445. retrievalsystems?Aframeworkforinvestigationandsystematicreview.
6. WilkinsT,GilliesRA,DaviesK:EMBASEversusMEDLINEforfamily JAMA1998,280:1347–1352.
medicinesearches:canMEDLINEsearchesfindtheforestoratree? 32. HaynesRB,McKibbonKA,WalkerCJ,RyanN,FitzgeraldD,RamsdenMF:
CanFamPhysician2005,51:849. OnlineaccesstoMEDLINEinclinicalsettings.Astudyofuseand
7. VerbeekJ,SalmiJ,PasternackI,JauhiainenM,LaamanenI,SchaafsmaF, usefulness.AnnInternMed1990,112:78–84.
HulshofC,vanDijkF:Asearchstrategyforoccupationalhealth
interventionstudies.OccupEnvironMed2005,62:682–687. doi:10.1186/1472-6947-13-7
8. MinozziS,PistottiV,ForniM:Searchingforrehabilitationarticleson Citethisarticleas:Gehannoetal.:Isthecoverageofgooglescholar
MEDLINEandEMBASE.Anexamplewithcross-overdesign.ArchPhys enoughtobeusedaloneforsystematicreviews.BMCMedicalInformatics
MedRehabil2000,81:720–722. andDecisionMaking201313:7.
9. McDonaldS,TaylorL,AdamsC:Searchingtherightdatabase.A
comparisonoffourdatabasesforpsychiatryjournals.HealthLibrRev
1999,16:151–156.
10. WatsonRJ,RichardsonPH:Identifyingrandomizedcontrolledtrialsof
cognitivetherapyfordepression:comparingtheefficiencyofEmbase,
MedlineandPsycINFObibliographicdatabases.BrJMedPsychol1999,
72:535–542.
11. FarriolM,Jordà-OlivesM,PadróJB:Bibliographicinformationretrievalin
thefieldofartificialnutrition.ClinNutr1998,17:217–222.
12. GehannoJF,ParisC,ThirionB,CaillardJF:Assessmentofbibliographic
databasesperformanceininformationretrievalforoccupationaland
environmentaltoxicology.OccupEnvironMed1998,55:562–566.
13. WoodsD,TrewheellarK:MedlineandEmbasecomplementeachotherin
literaturesearches.BMJ1998,316:1166.
14. BarillotMJ,SarrutB,DoreauCG:Evaluationofdruginteractiondocument
citationinnineon-linebibliographicdatabases.AnnPharmacother1997,
31:45–49.
15. BrazierH,BegleyCM:Selectingadatabaseforliteraturesearchesin
nursing:MEDLINEorCINAHL?JAdvNurs1996,24:868–875.
16. BurnhamJ,ShearerB:ComparisonofCINAHL,EMBASE,andMEDLINE
databasesforthenurseresearcher.MedRefServQ1993,12:45–57.
17. GallagherKE,HulbertLA,SullivanCP:Full-textandbibliographicdatabase
searchinginthehealthsciences:anexploratorystudycomparingCCML
andMEDLINE.MedRefServQ1990,9:17–25.
18. BeckmannM,vonWehrdenH:Whereyousearchiswhatyouget:
literaturemining–GoogleScholarversusWebofScienceusingadata
setfromaliteraturesearchinvegetationscience.JVegSci2012,
23(6):1197–1199.
19. MontoriVM,WilczynskiNL,MorganD,HaynesRB:Optimalsearch
strategiesforretrievingsystematicreviewsfromMedline:analytical
survey.BMJ2005,330:68.
20. GehannoJF,DarmoniSJ,CaillardJF:Majorinaccuraciesinarticlesciting
occupationalorenvironmentalmedicinepapersandtheirimplications.
JMedLibrAssoc2005,93:118–121.
21. SchererRW,LangenbergP,VonElmE:Fullpublicationofresultsinitially
presentedinabstracts.CochraneDatabaseSystRev2007,2:MR000005.
22. RollinL,DarmoniS,CaillardJ,GehannoJ:Fateofabstractspresentedat
anInternationalCommissiononOccupationalHealth(ICOH)congress-
followedbypublicationinpeer-reviewedjournals?ScandJWorkEnviron
Health2009,35:461–465.
23. TürpJC,SchulteJ,AntesG:Nearlyhalfofdentalrandomizedcontrolled
trialspublishedinGermanarenotincludedinMedline.EurJOralSci Submit your next manuscript to BioMed Central
2002,110:405–411. and take full advantage of:
24. HopewellS,ClarkeM,LusherA,LefebvreC,WestbyM:Acomparisonof
handsearchingversusMEDLINEsearchingtoidentifyreportsof
• Convenient online submission
randomizedcontrolledtrials.StatMed2002,21:1625–1634.
25. AguilloIF:IsGoogleScholarusefulforbibliometrics?Awebometric • Thorough peer review
analysis.Scientometrics2012,91:343–351. • No space constraints or color figure charges
26. HendersonJ:GoogleScholar:asourceforclinicians?CMAJ2005,
172:1549–1550. • Immediate publication on acceptance
27. VineR:GoogleScholar.JMedLibrAssoc2006,94:97–99. • Inclusion in PubMed, CAS, Scopus and Google Scholar
28. NourbakhshE,NugentR,WangH,CevikC,NugentK:Medicalliterature • Research which is freely available for redistribution
searches:acomparisonofPubMedandGoogleScholar.HealthInfoLibrJ
2012,29(3):214–222.
Submit your manuscript at
www.biomedcentral.com/submit
