Table Of ContentSSSeeerririeieesss I IISSSSSSNNN:: : 1 11999333555---333222333555 BBB
AAA MMM
RRR
SSSYYYNNNTTTHHHEEESSSIIISSS LLLEEECCCTTTUUURRREEESSS OOONNN RORORO &&&CCC MMMooorrrgggaaannn& & & CCClllaaayyypppoooooolll PPPuuubbbllliiissshhheeerrrsss
SSS
OOO
CCCOOOMMMPPPUUUTTTEEERRR AAARRRCCCHHHIIITTTEEECCCTTTUUURRREEE • • •
C C C
LLL
III
DDD
AAA
SSSeeerrriiieeesss E EEdddiiitttooorrr::: MMMaaarrrkkk DDD.. . HHHiiillllll,, , UUUnnniiivvveeerrrsssiiitttyyy o oofff W WWiiissscccooonnnsssiiinnn RRR
AAA
SSS TTThhheee DDDaaatttaaaccceeennnttteeerrr
• • •
TTThhheee DDDaaatttaaaccceeennnttteeerrr aaasss aaa CCCooommmpppuuuttteeerrr H H H
ÖÖÖ
LLL
AAAnnn IIInnntttrrroooddduuuccctttiiiooonnn tttooo ttthhheee DDDeeesssiiigggnnn ooofff WWWaaarrreeehhhooouuussseee---SSScccaaallleee MMMaaaccchhhiiinnneeesss,,, SSSeeecccooonnnddd EEEdddiiitttiiiooonnn ZZZ
LLL
EEE
LLLuuuiiizzz A AAnnndddrrrééé B BBaaarrrrrrooosssooo,, , J JJiiimmmmmmyyy C CCllliiidddaaarrraaasss,, , a aannnddd U UUrrrsss H HHööölllzzzllleee,, , G GGoooooogggllleee,, , I IInnnccc... aaasss aaa CCCooommmpppuuuttteeerrr
AAAsss c ccooommmpppuuuttataattitioioonnn c ccooonnnttitininnuuueeesss t totoo m mmooovvveee i ininnttotoo t ththheee c cclloloouuuddd, ,, t ththheee c ccooommmpppuuutttiininnggg p ppllalaattftfofoorrrmmm o ooff f i ininntteteerrreeessstt t n nnooo l loloonnngggeeerrr r rreeessseeemmmbbblleleesss a aa TTT
HHH
pppiizizzzzzaaa b bboooxxx o oorrr a aa r rreeefffrrriigiggeeerrraaattotoorr,r ,, b bbuuutt t a aa w wwaaarrreeehhhooouuussseee f fufuulllll l o ooff f c ccooommmpppuuutteteerrrsss. .. T TThhheeessseee n nneeewww l lalaarrrgggeee d ddaaattataaccceeennntteteerrrsss a aarrreee q qquuuiititetee d ddiififfffefeerrreeennnttt EEE
D D D
fffrrrooommm t ttrrraaadddiitittiioioonnnaaall l h hhooossstttiininnggg f fafaaccciilililitittiieieesss o ooff f e eeaaarrrllilieieerrr t ttiimimmeeesss a aannnddd c ccaaannnnnnooottt b bbeee v vviieieewwweeeddd s ssiimimmpppllylyy a aasss a aa c ccooollllleleeccctttiioioonnn o ooff f c ccooo---lloloocccaaattteeeddd AAAnnn IIInnntttrrroooddduuuccctttiiiooonnn tttooo ttthhheee DDDeeesssiiigggnnn
AAA
TTT
ssseeerrrvvveeerrrsss. .. L LLaaarrrgggeee p ppooorrrtttiioioonnnsss o ooff f t tthhheee h hhaaarrrdddwwwaaarrreee a aannnddd s ssoooffftttwwwaaarrreee r rreeesssooouuurrrccceeesss i ininn t tthhheeessseee f fafaaccciilililitittiieieesss m mmuuusssttt w wwooorrrkkk i ininn c ccooonnnccceeerrrttt t ttooo AAA
CCC
eeefffffificicciieieennnttltlylyy d ddeeellilivivveeerrr g ggooooooddd l leleevvveeellslss o ooff f I IInnntteteerrrnnneeett t s sseeerrrvvviicicceee p ppeeerrrffofoorrrmmmaaannnccceee, ,, s ssooommmeeetththhiininnggg t ththhaaatt t c ccaaannn o oonnnllylyy b bbeee a aaccchhhiieieevvveeeddd b bbyyy a aa h hhooollilisissttiticicc EEE ooofff WWWaaarrreeehhhooouuussseee---SSScccaaallleee MMMaaaccchhhiiinnneeesss
NNN
aaapppppprrroooaaaccchhh t ttooo t tthhheeeiirirr d ddeeesssiigiggnnn a aannnddd d ddeeepppllolooyyymmmeeennnttt. .. I IInnn o oottthhheeerrr w wwooorrrdddsss, ,, w wweee m mmuuusssttt t ttrrreeeaaattt t tthhheee d ddaaatttaaaccceeennnttteeerrr i itittssseeellflf f a aasss o oonnneee m mmaaassssssiivivveee TTT
EEE
wwwaaarrreeehhhooouuussseee---ssscccaaallelee c ccooommmpppuuutteteerrr ( ((WWWSSSCCC))). .. W WWeee d ddeeessscccrrriibibbeee t ththheee a aarrrccchhhiititeteecccttutuurrreee o ooff f W WWSSSCCCsss, ,, t ththheee m mmaaaiininn f fafaacccttotoorrrsss i ininnfflfluluueeennnccciininnggg t ththheeeiirirr RRR
dddeeesssiigiggnnn, ,, o oopppeeerrraaatttiioioonnn, ,, a aannnddd c ccooosssttt s sstttrrruuuccctttuuurrreee, ,, a aannnddd t tthhheee c cchhhaaarrraaacccttteeerrriisisstttiiciccsss o ooff f t tthhheeeiirirr s ssoooffftttwwwaaarrreee b bbaaassseee. .. W WWeee h hhooopppeee i ititt w wwiilillll l b bbeee u uussseeeffufuulll A A A
SSS SSSeeecccooonnnddd EEEdddiiitttiiiooonnn
tttooo a aarrrccchhhiititteeeccctttsss a aannnddd p pprrrooogggrrraaammmmmmeeerrrsss o ooff f t ttooodddaaayyy’s’’ss W WWSSSCCCsss, ,, a aasss w wweeelllll l a aasss t tthhhooossseee o ooff f f fufuutttuuurrreee m mmaaannnyyy---cccooorrreee p ppllalaatttffofoorrrmmmsss w wwhhhiicicchhh m mmaaayyy A A A
C C C
ooonnneee d ddaaayyy i imimmppplleleemmmeeennntt t t ththheee e eeqqquuuiivivvaaalleleennntt t o ooff f t totoodddaaayyy’s’’ss W WWSSSCCCsss o oonnn a aa s ssiininngggllelee b bboooaaarrrddd... OOO
MMM
PPP
NNNOOOTTTEEESSS F FFOOORRR T TTHHHEEE S SSEEECCCOOONNNDDD E EEDDDIIITTTIIIOOONNN UUU
TTT
AAAfffttteeerrr n nneeeaaarrrllylyy f fofoouuurrr y yyeeeaaarrrsss o ooff f s ssuuubbbssstttaaannntttiiaiaall l a aacccaaadddeeemmmiicicc a aannnddd i ininnddduuussstttrrriiaiaall l d ddeeevvveeelloloopppmmmeeennntttsss i ininn w wwaaarrreeehhhooouuussseee---ssscccaaallelee c ccooommmpppuuutttiininnggg,,,
EEE
wwweee a aarrreee d ddeeelliligigghhhttteeeddd t ttooo p pprrreeessseeennnttt o oouuurrr f fifirirrsssttt m mmaaajjojoorrr u uupppdddaaattteee t ttooo t tthhhiisiss l leleeccctttuuurrreee. .. T TThhheee i ininncccrrreeeaaassseeeddd p ppooopppuuullalaarrriitittyyy o ooff f p ppuuubbbllilicicc c cclloloouuudddsss RRR LLLuuuiiizzz AAAnnndddrrrééé BBBaaarrrrrrooosssooo
, S, S, S
hhhaaasss m mmaaadddeee W WWSSSCCC s ssooofffttwtwwaaarrreee t teteeccchhhnnniiqiqquuueeesss r rreeelleleevvvaaannntt t t totoo a aa l lalaarrrgggeeerrr p ppooooooll l o ooff f p pprrrooogggrrraaammmmmmeeerrrsss s ssiininnccceee o oouuurrr f fifirirrssstt t e eedddiitititioioonnn. ..T TThhheeerrreeeffofoorrreee,,, EEE
CCC
wwweee e eexxxpppaaannndddeeeddd C CChhhaaapppttteeerrr 2 22 t ttooo r rreeeffflleleecccttt o oouuurrr b bbeeetttttteeerrr u uunnndddeeerrrssstttaaannndddiininnggg o oofff W WWSSSCCC s ssoooffftttwwwaaarrreee s ssyyysssttteeemmmsss a aannnddd t tthhheee t ttoooooollblbboooxxx o oofff OOO JJJiiimmmmmmyyy CCCllliiidddaaarrraaasss
NNN
sssooofffttwtwwaaarrreee t teteeccchhhnnniiqiqquuueeesss f fofoorrr W WWSSSCCC p pprrrooogggrrraaammmmmmiininnggg. .. I IInnn C CChhhaaappptteteerrr 3 33, ,, w wweee a aaddddddeeeddd t totoo o oouuurrr c ccooovvveeerrraaagggeee o ooff f t ththheee e eevvvooollvlvviininnggg l lalaannndddssscccaaapppeee DDD
oooff f w wwiimimmpppyyy v vvsss. .. b bbrrraaawwwnnnyyy s sseeerrrvvveeerrr t ttrrraaadddeee---oooffffffsss, ,, a aannnddd w wweee n nnooowww p pprrreeessseeennnttt a aannn o oovvveeerrrvvviieieewww o ooff f W WWSSSCCC i ininnttteeerrrcccooonnnnnneeeccctttsss a aannnddd s sstttooorrraaagggeee ED ED ED UUUrrrsss HHHööölllzzzllleee
sssyyyssstteteemmmsss t ththhaaatt t w wwaaasss p pprrrooommmiisisseeeddd b bbuuutt t l lalaaccckkkiininnggg i ininn t ththheee o oorrriigiggiininnaaall l e eedddiitititioioonnn. ..T TThhhaaannnkkksss l lalaarrrgggeeellylyy t totoo t ththheee h hheeellplpp o ooff f o oouuurrr n nneeewww c ccooo---aaauuutththhooorr,r,, ITITIT
III
GGGoooooogggllelee D DDiisisstttiininnggguuuiisisshhheeeddd E EEnnngggiininneeeeeerrr J JJiimimmmmmyyy C CCllilididdaaarrraaasss, ,, t tthhheee m mmaaattteeerrriiaiaall l o oonnn f fafaaccciilililitittyyy m mmeeeccchhhaaannniiciccaaall l a aannnddd p ppooowwweeerrr d ddiisisstttrrriibibbuuutttiioioonnn OOO
NNN
dddeeesssiigiggnnn hhhaaasss bbbeeeeeennn uuupppdddaaattteeeddd aaannnddd gggrrreeeaaatttllylyy eeexxxttteeennndddeeeddd (((ssseeeeee CCChhhaaapppttteeerrrsss 444 aaannnddd 555))). .. CCChhhaaapppttteeerrrsss 666 aaannnddd 777 hhhaaavvveee aaallslssooo bbbeeeeeennn
rrreeevvvaaammmpppeeeddd s ssiigiggnnniififificiccaaannnttltlylyy. .. W WWeee h hhooopppeee t ththhiisiss r rreeevvviisisseeeddd e eedddiitititioioonnn c ccooonnnttitininnuuueeesss t totoo m mmeeeeeett t t ththheee n nneeeeeedddsss o ooff f e eeddduuucccaaattotoorrrsss a aannnddd p pprrroooffefeessssssiioioonnnaaallslss
iininn t ththhiisiss a aarrreeeaaa...
AAABBBOOOUUUTTT SSSYYYNNNTTTHHHEEESSSIIISSS MMM
OOO
TTThhhiisiss v vvooolluluummmeee i isiss a aa p pprrriininnttteeeddd v vveeerrrsssiioioonnn o ooff f a aa w wwooorrrkkk t tthhhaaattt a aappppppeeeaaarrrsss i ininn t tthhheee S SSyyynnnttthhheeesssiisiss RRR
DDDiigiggiititataall l L LLiibibbrrraaarrryyy o oofff E EEnnngggiininneeeeeerrriininnggg a aannnddd C CCooommmpppuuutteteerrr S SSccciieieennnccceee.. . S SSyyynnnttthhheeesssiisiss L LLeeeccctttuuurrreeesss GGG
AAA
ppprroroovvviididdeee c ccooonnnccciisisseee, ,, o oorrriigiggiininnaaall l p pprrereessseeennnttataattitioioonnnsss o ooff f i imimmpppooorrrttataannntt t r rereessseeeaaarrcrcchhh a aannnddd d ddeeevvveeelloloopppmmmeeennnttt NNN
ttotoopppiiciccsss, ,, p ppuuubbbllilisisshhheeeddd q qquuuiicicckkkllylyy, ,, i ininn d ddiigiggiititataall l a aannnddd p pprrriininntt t f fofoorrrmmmaaattstss. .. F FFooorrr m mmooorrreee i ininnffofoorrrmmmaaattitioioonnn &&&
CCC
vvviisissiitit t w wwwwwwww..m.mmooorrrgggaaannncccllalaayyypppooooooll.l.c.ccooommm
LLL SSSYYYNNNTTTHHHEEESSSIIISSS LLLEEECCCTTTUUURRREEESSS OOONNN
IIISSSBBBNNN::: 999777888---111---666222777000555---000000999---888 AAA
MMMooorrrgggaaannn & & & CCClllaaayyypppoooooolll PPPuuubbbllliiissshhheeerrrsss 999000000000000 YYY
PPP CCCOOOMMMPPPUUUTTTEEERRR AAARRRCCCHHHIIITTTEEECCCTTTUUURRREEE
OOO
wwwwwwwww...mmmooorrrgggaaannnccclllaaayyypppoooooolll...cccooommm
999777888111666222777000555000000999888 OOO
LLL MMMaaarrrkkk DDD.. . HHHiiillllll,, , SSSeeerrriiieeesss EEEdddiiitttooorrr
The Datacenter as a Computer
An Introduction to the Design of
Warehouse-Scale Machines
Second Edition
iii
Synthesis Lectures on Computer
Architecture
Editor
Mark D. Hill, University of Wisconsin, Madison
Synthesis Lectures on Computer Architecture publishes 50- to 100-page publications on topics
pertaining to the science and art of designing, analyzing, selecting, and interconnecting hardware
components to create computers that meet functional, performance, and cost goals. The scope will
largely follow the purview of premier computer architecture conferences, such as ISCA, HPCA,
MICRO, and ASPLOS.
The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines,
Second Edition
Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle
2013
Shared-Memory Synchronization
Michael L. Scott
2013
Resilient Architecture Design for Voltage Variation
Vijay Janapa Reddi , Meeta Sharma Gupta
2013
Multithreading Architecture
Mario Nemirovsky, Dean M. Tullsen
2013
Performance Analysis and Tuning for General Purpose Graphics Processing Units (GPGPU)
Hyesoon Kim, Richard Vuduc, Sara Baghsorkhi, Jee Choi, Wen-mei Hwu
2012
Automatic Parallelization: An Overview of Fundamental Compiler Techniques
Samuel P. Midkiff
2012
iv
Phase Change Memory: From Devices to Systems
Moinuddin K. Qureshi, Sudhanva Gurumurthi, Bipin Rajendran
2011
Multi-Core Cache Hierarchies
Rajeev Balasubramonian, Norman P. Jouppi, Naveen Muralimanohar
2011
A Primer on Memory Consistency and Cache Coherence
Daniel J. Sorin, Mark D. Hill, David A. Wood
2011
Dynamic Binary Modification: Tools, Techniques, and Applications
Kim Hazelwood
2011
Quantum Computing for Computer Architects, Second Edition
Tzvetan S. Metodi, Arvin I. Faruque, Frederic T. Chong
2011
High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities
Dennis Abts, John Kim
2011
Processor Microarchitecture: An Implementation Perspective
Antonio González, Fernando Latorre, Grigorios Magklis
2010
Transactional Memory, 2nd edition
Tim Harris , James Larus , Ravi Rajwar
2010
Computer Architecture Performance Evaluation Methods
Lieven Eeckhout
2010
Introduction to Reconfigurable Supercomputing
Marco Lanzagorta, Stephen Bique, Robert Rosenberg
2009
On-Chip Networks
Natalie Enright Jerger, Li-Shiuan Peh
2009
v
The Memory System: You Can’t Avoid It, You Can’t Ignore It, You Can’t Fake It
Bruce Jacob
2009
Fault Tolerant Computer Architecture
Daniel J. Sorin
2009
The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
Luiz André Barroso, Urs Hölzle
2009
Computer Architecture Techniques for Power-Efficiency
Stefanos Kaxiras, Margaret Martonosi
2008
Chip Multiprocessor Architecture: Techniques to Improve Throughput and Latency
Kunle Olukotun, Lance Hammond, James Laudon
2007
Transactional Memory
James R. Larus, Ravi Rajwar
2006
Quantum Computing for Computer Architects
Tzvetan S. Metodi, Frederic T. Chong
2006
Copyright © 2013 by Morgan & Claypool
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in
any form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quotations
in printed reviews, without the prior permission of the publisher.
The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines,
Second Edition
Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle
www.morganclaypool.com
ISBN: 9781627050098 print
ISBN: 9781627050104 ebook
DOI 10.2200/S00516ED2V01Y201306CAC024
A Publication in the Morgan & Claypool Publishers series
SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE
Lecture #24
Series Editor: Mark D. Hill, University of Wisconsin, Madison
Series ISSN 1935-3235 Print 1935-3243 Electronic
The Datacenter as a Computer
An Introduction to the Design of
Warehouse-Scale Machines
Second Edition
Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle
Google, Inc.
SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE #24
M
&C MORGAN & CLAYPOOL PUBLISHERS
viii
ABSTRACT
As computation continues to move into the cloud, the computing platform of interest no longer re-
sembles a pizza box or a refrigerator, but a warehouse full of computers. These new large datacenters
are quite different from traditional hosting facilities of earlier times and cannot be viewed simply as
a collection of co-located servers. Large portions of the hardware and software resources in these
facilities must work in concert to efficiently deliver good levels of Internet service performance,
something that can only be achieved by a holistic approach to their design and deployment. In
other words, we must treat the datacenter itself as one massive warehouse-scale computer (WSC).
We describe the architecture of WSCs, the main factors influencing their design, operation, and
cost structure, and the characteristics of their software base. We hope it will be useful to architects
and programmers of today’s WSCs, as well as those of future many-core platforms which may one
day implement the equivalent of today’s WSCs on a single board.
NOTES FOR THE SECOND EDITION
After nearly four years of substantial academic and industrial developments in warehouse-scale
computing, we are delighted to present our first major update to this lecture. The increased popu-
larity of public clouds has made WSC software techniques relevant to a larger pool of programmers
since our first edition. Therefore, we expanded Chapter 2 to reflect our better understanding of
WSC software systems and the toolbox of software techniques for WSC programming. In Chap-
ter 3, we added to our coverage of the evolving landscape of wimpy vs. brawny server trade-offs,
and we now present an overview of WSC interconnects and storage systems that was promised
but lacking in the original edition. Thanks largely to the help of our new co-author, Google Dis-
tinguished Engineer Jimmy Clidaras, the material on facility mechanical and power distribution
design has been updated and greatly extended (see Chapters 4 and 5). Chapters 6 and 7 have also
been revamped significantly. We hope this revised edition continues to meet the needs of educators
and professionals in this area.
KEYWORDS
computer organization and design, Internet services, energy efficiency, fault-tolerant computing,
cluster computing, data centers, distributed systems, cloud computing
ix
Contents
Acknowledgments .................................................. xiii
Note to the Reader .................................................. xv
1 Introduction ....................................................... . 1
1.1 Warehouse-Scale Computers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Cost Efficiency at Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Not Just a Collection of Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 One Datacenter vs. Several Datacenters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.5 Why WSCS Might Matter to You . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.6 Architectural Overview of WSCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.6.1 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.6.2 Networking Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6.3 Storage Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6.4 Quantifying Latency, Bandwidth, and Capacity . . . . . . . . . . . . . . . . . 9
1.6.5 Power Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6.6 Handling Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2 Workloads and Software Infrastructure ................................. 15
2.1 Datacenter vs. Desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Performance and Availability Toolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 Platform-Level Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4 Cluster-Level Infrastructure Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.1 Resource Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.2 Hardware Abstraction and Other Basic Services . . . . . . . . . . . . . . . 22
2.4.3 Deployment and Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.4 Programming Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Application-Level Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.5.1 Workload Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.5.2 Online: Web Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5.3 Offline: Scholar Article Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.6 A Monitoring Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.6.1 Service-Level Dashboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
