Data science and complex networks : real cases studies with Python

139 Pages·2016·3.198 MB·English

by  Caldarelli, Guido, Chessa, Alessandro

DATA SCIENCE AND COMPLEX NETWORKS

Data Science and Complex Networks
Real Cases Studies with Python
Guido Caldarelli
IMT,PiazzaS.Francesco19,Lucca,Italy;
ISC,IstitutodeiSistemiComplessi,CNR,Italy;
LIMS,LondonInstituteofMathematicalSciences,London,UK;
Linkalab,ComplexSystemsComputationalLaboratory,Cagliari,Italy;
Catchy,LaboratoryofBigDataAnalysis,Roma,Italy
Alessandro Chessa
IMT,PiazzaS.Francesco19,Lucca,Italy;
ISC,IstitutodeiSistemiComplessi,CNR,Italy;
Linkalab,ComplexSystemsComputationalLaboratory,Cagliari,Italy;
Catchy,LaboratoryofBigDataAnalysis,Roma,Italy
3
3
GreatClarendonStreet,Oxford,OX26DP,
UnitedKingdom
OxfordUniversityPressisadepartmentoftheUniversityofOxford.
ItfurtherstheUniversity’sobjectiveofexcellenceinresearch,scholarship,
andeducationbypublishingworldwide.Oxfordisaregisteredtrademarkof
OxfordUniversityPressintheUKandincertainothercountries
©GuidoCaldarelliandAlessandroChessa2016
Themoralrightsoftheauthorshavebeenasserted
FirstEditionpublishedin2016
Impression:1
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedin
aretrievalsystem,ortransmitted,inanyformorbyanymeans,withoutthe
priorpermissioninwritingofOxfordUniversityPress,orasexpresslypermitted
bylaw,bylicenceorundertermsagreedwiththeappropriatereprographics
rightsorganization.Enquiriesconcerningreproductionoutsidethescopeofthe
aboveshouldbesenttotheRightsDepartment,OxfordUniversityPress,atthe
addressabove
Youmustnotcirculatethisworkinanyotherform
andyoumustimposethissameconditiononanyacquirer
PublishedintheUnitedStatesofAmericabyOxfordUniversityPress
198MadisonAvenue,NewYork,NY10016,UnitedStatesofAmerica
BritishLibraryCataloguinginPublicationData
Dataavailable
LibraryofCongressControlNumber:2016949583
ISBN978–0–19–963960–1
Printedandboundby
CPIGroup(UK)Ltd,Croydon,CR04YY
LinkstothirdpartywebsitesareprovidedbyOxfordingoodfaithand
forinformationonly.Oxforddisclaimsanyresponsibilityforthematerials
containedinanythirdpartywebsitereferencedinthiswork.
Preface
The audience for this book comprises researchers, not necessarily trained in mathe-
maticsandphysics,whohavetheproblemofanalysingandsortingouthugequantities
ofdata.Thetheoryofcomplexnetworkshasalreadyprovedtobeagoodchoicewhen
looking for solutions to such problems. The reasons for that success rely on the fact
thatinalargevarietyofsituationsitisveryeasytoidentifythebasicunitsofthesys-
tem under consideration; network theory models them by vertices and describes their
interactions by means of links, thereby forming a clear mathematical set up for many
scientificquestions.Aparamountexampleisprovidedbytechnologicalnetworks.Sites
andhyperlinksareimmediatelydescribedbyverticesandedges.Thesameassociation
for server and cables is almost as immediate. On the one hand, such a simple model
reveals the inner structure of what we are observing (if we want to measure the reli-
ability of the Internet, we need to know its skeleton). On the other hand, even if the
structureweobservehasnoobviousnetworkstructure,wecanstillusenetworktheory
to explore it. An example is given by the big-data world in which we now live. Such
datamayspanbetweenpersonalinformationasregisteredbymobilephones,GPSde-
vices, financial transactions, or results from high-throughput biological experiments.
In all these cases networks can provide useful metrics to filter information. Networks
allow us to filter information by selecting more relevant and important correlations
and they allow for a quantitative description of any approximation made.
Exactly in this spirit, we present in this book the latest results in the field of
complexnetworks.TheycansolveimmediateproblemswhenstudyingtheInternetand
the WWW, and they can help to sort information on a variety of other systems. This
book is structured in such a way as to start with a specific problem and then present
the theoretical tools needed to model or to sort out the most relevant information. At
the end of every chapter we present the structure of the computer program needed to
apply these ideas to the data.
Working copies of the software are available from the web site of the book,
http://book.complexnetworks.net.
Practitioners(andmoregenerallyallthereaders)arenotrequiredtohaveexpertise
in statistical physics or in computer programming; this text can provide all the nec-
essary information. We are indebted to many friends and colleagues for their critical
reading and suggestions, amongst them we want to cite Bruno Gon¸calves and Nicola
Perra; a special thank goes to Sebastiano Vigna for his precious comments.
Guido Caldarelli acknowledges for support the EU projects Multiplex nr. 317532,
Simpol nr. 610704, Dolfins nr. 640772 SoBigData 654024 CoeGSS 676547.
Alessandro Chessa acknowledges support from PNR national project CRISIS-Lab
(IMT School for Advanced Studies).

Contents
Introduction 1
1 Food Webs 4
1.1 Introduction 4
1.2 Data from EcoWeb and foodweb.org 6
1.3 Store and measure a graph: size, measure, and degree 8
1.4 Degree sequence 14
1.5 Clustering coefficient and motifs 15
2 International Trade Networks and World Trade Web 26
2.1 Introduction 26
2.2 Data from COMTRADE 27
2.3 Projecting and symmetrising a bipartite network 30
2.4 Neighbour quantities: reciprocity and assortativity 33
2.5 Multigraphs 37
2.6 The bipartite network of products and countries 40
3 The Internet Network 45
3.1 Introduction 45
3.2 Data from CAIDA 45
3.3 Importance or centrality 50
3.4 Robustness and resilience, giant component 56
4 World Wide Web, Wikipedia, and Social Networks 63
4.1 Introduction 63
4.2 Data from various sources 64
4.3 Bringing order to the WWW 70
4.4 Communities and Girvan–Newman algorithm 82
4.5 Modularity 85
5 Financial Networks 91
5.1 Introduction 91
5.2 Data from Yahoo! Finance 91
5.3 Prices time series 96
5.4 Correlation of prices 98
5.5 Minimal spanning trees 99
6 Modelling 103
6.1 Introduction 103
6.2 Exponential growth, chains, and random graph 103
6.3 Random graphs 107
viii Contents
6.4 Configuration models 110
6.5 Gravity model 112
6.6 Fitness model 114
6.7 Barab´asi–Albert model 115
6.8 Reconstruction of networks 117
References 122
Index 129
Introduction
Complex networks, thanks to their intuitive and interdisciplinary basis, are becoming
moreandmorepopularamongscientistsandpractitioners(AlbertandBarab´asi,2002;
CatanzaroandCaldarelli,2012).Notonlyisbasicmodellingrelativelysimpletounder-
stand, but also the “local” network properties (i.e. relative to single vertices/nodes),
such as the degree or the clustering, can immediately be visualised and computed. In
fact, while the first approach may produce immediate results, less trivial quantities
and measurements require a greater technical knowledge. In particular, going from
local graph quantities to more global ones (such as the resilience or the optimisation)
and passing through two- and three-vertices properties (such as reciprocity and assor-
tativity) may prove to be less intuitive. In parallel with the success of network theory,
wearealsowitnessinganotherphenomenon,thatis,thegrowthinnumbersandsizeof
data repositories. Every day an increasing flow of news, media, comments in the form
ofblogs,likes,andsharingonsocialnetworksisproducedandstored.Algorithmsand
techniques, some simple and some more elaborate, are necessary to extract valuable
information from this amount of data.
The aim of this book is to show how network theory is a crucial tool both to
model and to filter information in these data-producing systems. In this spirit we
want to guide readers through this fascinating field by presenting both datasets and
network concepts. The idea is to start from basic (and indeed simple and immediate)
concepts, moving towards more elaborate quantities and to give the instruments (es-
pecially the software) to the reader to continue their research even further. Towards
this purpose, following the recent success of complex networks, there is now a series
of books and manuals on the topic. Spanning between introductory and more ad-
vanced courses (Dorogovtsev and Mendes, 2003; Caldarelli, 2007; Barrat et al., 2008;
Dorogovtsev, 2010; Newman, 2010) and considering the various reviews and publica-
tions available (Newman, 2003; Barab´asi and Bonabeau, 2003; Boccaletti et al., 2006;
Estrada et al., 2010), there is now an important corpus of work available to scientists
and practitioners. With respect to all of these books, we want to present a different
approach where the activity starts by working directly on raw data. Indeed, the con-
ceptswrittenintheoreticalbooks,needtobeusedinrealsituations,i.e.inmostofthe
cases they need to be translated into software. Since in practical situations the data
we are interested in are often in a raw and unformatted form (very different to the
exempla we see in textbooks and publications), the absence of powerful pieces of code
resultsinanobstacletotheworkofresearchers.Thissituationisrathergeneral,since
theneedforfastandeasyimplementationofthebasicconceptsofcomplexnetworksis
growing in many different fields, namely biology, ecology, finance and economics, not
tomentiontheinterestforpolicymakers,stakeholders,andpractitioners.Inthisbook
we have selected a series of topics and we have shown the data available for them.
DataScienceandComplexNetworks.FirstEdition.GuidoCaldarelliandAlessandroChessa.
(cid:13)c GuidoCaldarelliand Alessandro Chessa 2016. Published in 2016 by Oxford University
Press.