Analyzing and Influencing Search Engine Results: Business and Technology Impacts on Web Information Retrieval

232 Pages·2010·3.39 MB·English

by  Sönke Lieberam-Schmidt (auth.)

Sönke Lieberam-Schmidt
Analyzing and Infl uencing Search Engine Results
GABLER RESEARCH
Sönke Lieberam-Schmidt

Analyzing and Infl uencing
Search Engine Results

Business and Technology Impacts
on Web Information Retrieval
With a foreword by Prof. Dr. Erwin Pesch
RESEARCH
Bibliographic information published by the Deutsche Nationalbibliothek
The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografi e;
detailed bibliographic data are available in the Internet at http://dnb.d-nb.de.
Doctoral thesis, University of Siegen, 2009

1st Edition 2010
All rights reserved
© Gabler Verlag | Springer Fachmedien Wiesbaden GmbH 2010
Editorial Offi ce: Ute Wrasmann | Sabine Schöller
Gabler Verlag is a brand of Springer Fachmedien.
Springer Fachmedien is part of Springer Science+Business Media.
www.gabler.de
No part of this publication may be reproduced, stored in a retrieval system
or transmitted, in any form or by any means, electronic, mechanical, photo-
copying, recording, or otherwise, without the prior written permission of the
copyright holder.
Registered and/or industrial names, trade names, trade descriptions etc. cited in this publica-
tion are part of the law for trade-mark protection and may not be used free in any form or by
any means even if this is not specifi cally marked.
Cover design: KünkelLopka Medienentwicklung, Heidelberg
Printed on acid-free paper
Printed in Germany
ISBN 978-3-8349-2453-7
Foreword
Search engines are indexing several billions of webpages and thus enable an ex-
tremely quick access to all sorts of contents of the Web. Most of these requests
consistofoneword.Asresult,severalthousandsofdocumentsaredeliveredwhere
the first documents are not necessarily the needed ones. Where the specification
of the query depends on the searcher’s knowledge, the order of listed results de-
pends on the search engine used. Knowing the search engines leads to a better
understandingofresultsandhelpsdevelopingspecificWeb-basedsalesstrategies.
Thepresentwork“AnalyzingandInfluencingSearchEngineResults”picksup
anup-to-datetopicduetothefactthatsearchengineshavebecomeanimportant
influencefactorineconomyandsociety.
The author presents a comprehensive overview of recent Web-search technol-
ogy. Hedescribestherelatedproblemsfromasearchengine’saswellasfromthe
user’sperspective. Drivenbytheusers’needs, newapproachesforsearchingthe
Webaredeveloped,implementedandtested.
Readersofthisworkwillbecomefamiliarwiththeprecautionsthathavetobe
taken for reaching upper result page positions. Different strategies are explained
including spamming techniques that are used to influence the results of ranking
methods.
Following the principle of business informatics, the author explores the topic
from a business and a technology view. He complements the scientific view by
approachesforpracticalproblemsandgivesconcreteinstructionsforpractitioners.
TheworkdeliversadeepinsightintothemethodsandproblemsofWebsearchand
iswellsuitedforWebsiteownersthatwanttoimprovetheirvisibilityintheWeb.
Seeing the high relevance of search engines in many different areas, I expect
thisworktoattracthighattentionofthescientificcommunityandthepractice.
ProfessorDr.ErwinPesch
Acknowledgements
Duringthedevelopmentofthiswork,severalpeopleandorganizationssupported
meexploringtheworldofWebsearch. Istronglyappreciatetheircontributionand
Iwanttoexpressmythankstoallofthem. Nevertheless, Iliketonamesomeof
theminthefollowing.
First,IwanttothankProf.Dr.ErwinPeschandhisteamofthechairofmanage-
ment information sciences for supplying the research environment, their input in
stimulatingdiscussionsandtheirconstructivefeedback. Prof.Dr.HerbertKopfer
andProf.Dr.JoachimEiglerreceivemythanksforconductingasmoothandfast
review and examination process. I thank my colleagues and the staff of Siegen
Universityforbuildingaproductiveandconvenientworkingatmosphere.
Furthermore, I want to thank the contraco Consulting & Software Ltd., Pots-
dam,fortheircontributiontothefundingofthisworkandthevariousinspirations
I have received from their management and staff. Different co-operations with
VivawasserAG,Kiel,havealsoresultedintoproductiveinputthatIgratefullyac-
knowledge.
The companies T-Online International AG, Darmstadt, and FAST Search &
Transfer ASA, Oslo, gave me useful insights into the details and practical prob-
lemsofsearchengineoperations. IthanktheInteressengemeinschaftFrankfurter
Kreditinstitute GmbH who expressed their appreciation for this work by funding
itspublication.
Lastbutnotleast,Iwishtoexpressmydeepestgratitudetomyfamilyandmy
friends – and especially my fiancée – for the strong support they have given me
duringthetimeIhavebeenworkingonthisproject.
Dr.SönkeLieberam-Schmidt
Contents
ListofFigures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XIII
ListofTables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XVI
ListofAbbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . XIX
Introduction 1
1 WebSearch 5
1.1 InformationDemand . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.1 User’sInformationNeed . . . . . . . . . . . . . . . . . . 7
1.1.2 UserBehavior . . . . . . . . . . . . . . . . . . . . . . . 10
1.2 SearchTools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.3 HowSearchEnginesWork . . . . . . . . . . . . . . . . . . . . . 19
1.3.1 Crawler . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.3.2 Indexer . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
1.3.3 Analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.3.4 Searcher. . . . . . . . . . . . . . . . . . . . . . . . . . . 36
1.3.5 Presenter . . . . . . . . . . . . . . . . . . . . . . . . . . 37
1.4 ANewApproachtoRelevancy . . . . . . . . . . . . . . . . . . . 38
1.5 EvolutionofWebSearchEngines . . . . . . . . . . . . . . . . . 46
2 WebStructure 49
2.1 EvolutionofInformationSupply . . . . . . . . . . . . . . . . . . 49
2.2 WebContent . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.3 TheWebGraph . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.3.1 ReasonsforRegardingtheWebGraph . . . . . . . . . . . 54
2.3.2 PropertiesoftheWebGraph . . . . . . . . . . . . . . . . 55
2.3.3 RepresentationandStorageoftheWebGraph . . . . . . . 61
2.4 LinkAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
2.4.1 Ranking . . . . . . . . . . . . . . . . . . . . . . . . . . 67
2.4.2 AuthoritiesandHubs . . . . . . . . . . . . . . . . . . . . 71
2.4.3 OtherLink-basedMethods . . . . . . . . . . . . . . . . . 77
2.5 DataStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
2.5.1 FeasibleDataStructures . . . . . . . . . . . . . . . . . . 84
X Contents
2.5.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 89
2.5.3 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . 93
2.6 HITSinPractice . . . . . . . . . . . . . . . . . . . . . . . . . . 95
2.6.1 ImplementationoftheAlgorithm . . . . . . . . . . . . . 95
2.6.2 ExperimentalResultsandEvaluation . . . . . . . . . . . 100
3 ResultClustering 105
3.1 ClusteringMethods . . . . . . . . . . . . . . . . . . . . . . . . . 106
3.2 WebClustering . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
3.3 TextBasedApproaches . . . . . . . . . . . . . . . . . . . . . . . 109
3.4 LinkBasedApproaches . . . . . . . . . . . . . . . . . . . . . . . 112
3.4.1 RobustClusteringAlgorithm . . . . . . . . . . . . . . . . 113
3.4.2 InclusionofLinkDirection. . . . . . . . . . . . . . . . . 119
3.4.3 Architecture. . . . . . . . . . . . . . . . . . . . . . . . . 121
3.4.4 ExperimentalResults . . . . . . . . . . . . . . . . . . . . 123
3.5 CombinedApproaches . . . . . . . . . . . . . . . . . . . . . . . 125
3.5.1 Architecture. . . . . . . . . . . . . . . . . . . . . . . . . 125
3.5.2 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . 126
3.5.3 DocumentRepresentation . . . . . . . . . . . . . . . . . 130
3.5.4 ClusteringMethodCombination . . . . . . . . . . . . . . 133
3.5.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . 151
3.5.6 ExperimentalResults . . . . . . . . . . . . . . . . . . . . 153
4 SearchEngineOptimization 163
4.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
4.1.1 WebsiteVisitors . . . . . . . . . . . . . . . . . . . . . . 166
4.1.2 ResultPagePosition . . . . . . . . . . . . . . . . . . . . 168
4.2 PossibilitiesandMeans . . . . . . . . . . . . . . . . . . . . . . . 171
4.2.1 Visibility . . . . . . . . . . . . . . . . . . . . . . . . . . 173
4.2.2 StaticRelevance . . . . . . . . . . . . . . . . . . . . . . 174
4.2.3 DynamicRelevance . . . . . . . . . . . . . . . . . . . . 175
4.3 KeywordSelection . . . . . . . . . . . . . . . . . . . . . . . . . 176
4.3.1 KeywordSelectionPositionModel(KSPM) . . . . . . . . 178
4.3.2 EnhancedKeywordSelectionPositionModel(EKSPM) . 182
4.3.3 KeywordSelectionTopPositionModel(KSTPM) . . . . 184
4.4 Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
4.4.1 SearchEngineSpam . . . . . . . . . . . . . . . . . . . . 186
4.4.2 SpammingMethods . . . . . . . . . . . . . . . . . . . . 187
4.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Contents XI
4.5.1 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
4.5.2 WebsiteCreationProcess . . . . . . . . . . . . . . . . . . 191
4.6 PracticalResults. . . . . . . . . . . . . . . . . . . . . . . . . . . 195
4.6.1 ProjectandTestWebsites . . . . . . . . . . . . . . . . . 195
4.6.2 ReportingandResults . . . . . . . . . . . . . . . . . . . 197
5 Conclusion 205
Bibliography 207
Index 219
List of Figures
1.1 Emailandsearchengineusage . . . . . . . . . . . . . . . . . . . 6
1.2 Classicalsearchmodel . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Characteristicsofsearchtools . . . . . . . . . . . . . . . . . . . 15
1.4 Searchenginerelationshipchart2002–2004 . . . . . . . . . . . . 17
1.5 Searchenginerelationshipchart2006–2009 . . . . . . . . . . . . 18
1.6 Searchengineresultflow . . . . . . . . . . . . . . . . . . . . . . 20
1.7 MapReducetaskprocessing . . . . . . . . . . . . . . . . . . . . 23
1.8 MapReducesampleinput . . . . . . . . . . . . . . . . . . . . . . 24
1.9 Mapandreducestep . . . . . . . . . . . . . . . . . . . . . . . . 25
1.10 MapReducesampleoutput . . . . . . . . . . . . . . . . . . . . . 26
1.11 MapReduceexecutionoverview . . . . . . . . . . . . . . . . . . 27
1.12 TrivialscheduleofWebcrawling . . . . . . . . . . . . . . . . . . 30
1.13 SamplescheduleofWebcrawling . . . . . . . . . . . . . . . . . 31
1.14 Generalsearchenginearchitecture . . . . . . . . . . . . . . . . . 34
1.15 HighlevelGOOGLEarchitecture . . . . . . . . . . . . . . . . . . 35
1.16 F-shapeofsearch . . . . . . . . . . . . . . . . . . . . . . . . . . 39
1.17 Informationsupplyengine . . . . . . . . . . . . . . . . . . . . . 48
2.1 ClassificationofWebcontent . . . . . . . . . . . . . . . . . . . . 51
2.2 Random-graphexamplesI . . . . . . . . . . . . . . . . . . . . . 57
2.3 Random-graphexamplesII . . . . . . . . . . . . . . . . . . . . . 58
2.4 Gridnetworkmodel . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.5 ConnectivityoftheWeb–the“bowtie”structure . . . . . . . . . 60
2.6 ConnectivityServer . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.7 S-Noderepresentation . . . . . . . . . . . . . . . . . . . . . . . 63
2.8 PartitioningtheWebgraph . . . . . . . . . . . . . . . . . . . . . 64
2.9 S-NoderepresentationofaWebgraphexample . . . . . . . . . . 66
2.10 PageRankcalculationexamples . . . . . . . . . . . . . . . . . . 68
2.11 TheHITSalgorithm–base-setcreation . . . . . . . . . . . . . . 72
2.12 HITSalgorithm–Expandingtherootset . . . . . . . . . . . . . . 74
2.13 Authorityandhubvalues . . . . . . . . . . . . . . . . . . . . . . 75
2.14 Vicinitygraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78