Systems Performance: Enterprise and the Cloud

1345 Pages·2013·33.06 MB·English

by  Brendan Gregg

About This eBook
ePUB is an open, industry-standard format for eBooks. However, support of
ePUB and its many features varies across reading devices and applications. Use
your device or app settings to customize the presentation to your liking. Settings
that you can customize often include font, font size, single or double column,
landscape or portrait mode, and figures that you can click or tap to enlarge. For
additional information about the settings and features on your reading device or
app, visit the device manufacturer’s Web site.
Many titles include programming code or configuration examples. To
optimize the presentation of these elements, view the eBook in single-column,
landscape mode and adjust the font size to the smallest setting. In addition to
presenting code and configurations in the reflowable text format, we have
included images of the code that mimic the presentation found in the print book;
therefore, where the reflowable format may compromise the presentation of the
code listing, you will see a “Click here to view code image” link. Click the link
to view the print-fidelity code image. To return to the previous page viewed,
click the Back button on your device or app.
Systems Performance
Enterprise and the Cloud
Brendan Gregg
Upper Saddle River, NJ • Boston • Indianapolis • San Francisco
New York • Toronto • Montreal • London • Munich • Paris • Madrid
Capetown • Sydney • Tokyo • Singapore • Mexico City
Many of the designations used by manufacturers and sellers to distinguish their
products are claimed as trademarks. Where those designations appear in this
book, and the publisher was aware of a trademark claim, the designations have
been printed with initial capital letters or in all capitals.
The author and publisher have taken care in the preparation of this book, but
make no expressed or implied warranty of any kind and assume no responsibility
for errors or omissions. No liability is assumed for incidental or consequential
damages in connection with or arising out of the use of the information or
programs contained herein.
The publisher offers excellent discounts on this book when ordered in quantity
for bulk purchases or special sales, which may include electronic versions and/or
custom covers and content particular to your business, training goals, marketing
focus, and branding interests. For more information, please contact:
U.S. Corporate and Government Sales
[phone]
[e-mail]
For sales outside the United States, please contact:
International Sales
[e-mail]
Visit us on the Web: informit.com/ph
Library of Congress Cataloging-in-Publication Data
Gregg, Brendan.
Systems performance : enterprise and the cloud / Brendan Gregg.
pages cm
Includes bibliographical references and index.
ISBN-13: 978-0-13-339009-4 (alkaline paper)
ISBN-10: 0-13-339009-8 (alkaline paper)
1. Operating systems (Computers)—Evaluation. 2. Application software—
Evaluation. 3. Business
Enterprises—Data processing. 4. Cloud computing. I. Title.
QA76.77.G74 2014
004.67'82—dc23
2013031887
Copyright © 2014 Pearson Education, Inc.
All rights reserved. Printed in the United States of America. This publication is
protected by copyright, and permission must be obtained from the publisher
prior to any prohibited reproduction, storage in a retrieval system, or
transmission in any form or by any means, electronic, mechanical,
photocopying, recording, or likewise. To obtain permission to use material from
this work, please submit a written request to Pearson Education, Inc.,
Permissions Department, One Lake Street, Upper Saddle River, New Jersey
07458, or you may fax your request to [phone].
ISBN-13: 978-0-13-339009-4
ISBN-10: 0-13-339009-8
Text printed in the United States on recycled paper at Edwards Brothers Malloy
in Ann Arbor, Michigan.
First printing, October, 2013
Contents
Preface
Acknowledgments
About the Author
Chapter 1 Introduction
1.1 Systems Performance
1.2 Roles
1.3 Activities
1.4 Perspectives
1.5 Performance Is Challenging
1.5.1 Performance Is Subjective
1.5.2 Systems Are Complex
1.5.3 There Can Be Multiple Performance Issues
1.6 Latency
1.7 Dynamic Tracing
1.8 Cloud Computing
1.9 Case Studies
1.9.1 Slow Disks
1.9.2 Software Change
1.9.3 More Reading
Chapter 2 Methodology
2.1 Terminology
2.2 Models
2.2.1 System under Test
2.2.2 Queueing System
2.3 Concepts
2.3.1 Latency
2.3.2 Time Scales
2.3.3 Trade-offs
2.3.4 Tuning Efforts
2.3.5 Level of Appropriateness
2.3.6 Point-in-Time Recommendations
2.3.7 Load versus Architecture
2.3.8 Scalability
2.3.9 Known-Unknowns
2.3.10 Metrics
2.3.11 Utilization
2.3.12 Saturation
2.3.13 Profiling
2.3.14 Caching
2.4 Perspectives
2.4.1 Resource Analysis
2.4.2 Workload Analysis
2.5 Methodology
2.5.1 Streetlight Anti-Method
2.5.2 Random Change Anti-Method
2.5.3 Blame-Someone-Else Anti-Method
2.5.4 Ad Hoc Checklist Method
2.5.5 Problem Statement
2.5.6 Scientific Method
2.5.7 Diagnosis Cycle
2.5.8 Tools Method
2.5.9 The USE Method
2.5.10 Workload Characterization
2.5.11 Drill-Down Analysis
2.5.12 Latency Analysis
2.5.13 Method R
2.5.14 Event Tracing
2.5.15 Baseline Statistics
2.5.16 Static Performance Tuning
2.5.17 Cache Tuning
2.5.18 Micro-Benchmarking
2.6 Modeling
2.6.1 Enterprise versus Cloud
2.6.2 Visual Identification
2.6.3 Amdahl’s Law of Scalability
2.6.4 Universal Scalability Law
2.6.5 Queueing Theory
2.7 Capacity Planning
2.7.1 Resource Limits
2.7.2 Factor Analysis
2.7.3 Scaling Solutions
2.8 Statistics
2.8.1 Quantifying Performance
2.8.2 Averages
2.8.3 Standard Deviations, Percentiles, Median
2.8.4 Coefficient of Variation
2.8.5 Multimodal Distributions
2.8.6 Outliers
2.9 Monitoring
2.9.1 Time-Based Patterns
2.9.2 Monitoring Products
2.9.3 Summary-since-Boot
2.10 Visualizations
2.10.1 Line Chart
2.10.2 Scatter Plots
2.10.3 Heat Maps
2.10.4 Surface Plot
2.10.5 Visualization Tools
2.11 Exercises
2.12 References
Chapter 3 Operating Systems
3.1 Terminology
3.2 Background
3.2.1 Kernel
3.2.2 Stacks
3.2.3 Interrupts and Interrupt Threads
3.2.4 Interrupt Priority Level
3.2.5 Processes
3.2.6 System Calls
3.2.7 Virtual Memory
3.2.8 Memory Management
3.2.9 Schedulers
3.2.10 File Systems
3.2.11 Caching
3.2.12 Networking
3.2.13 Device Drivers
3.2.14 Multiprocessor
3.2.15 Preemption
3.2.16 Resource Management
3.2.17 Observability
3.3 Kernels
3.3.1 Unix
3.3.2 Solaris-Based
3.3.3 Linux-Based
3.3.4 Differences
3.4 Exercises
3.5 References
Chapter 4 Observability Tools
4.1 Tool Types
4.1.1 Counters
4.1.2 Tracing
4.1.3 Profiling
4.1.4 Monitoring (sar)
4.2 Observability Sources
4.2.1 /proc
4.2.2 /sys
4.2.3 kstat
4.2.4 Delay Accounting
4.2.5 Microstate Accounting
4.2.6 Other Observability Sources
4.3 DTrace
4.3.1 Static and Dynamic Tracing
4.3.2 Probes
4.3.3 Providers
4.3.4 Arguments
4.3.5 D Language
4.3.6 Built-in Variables
4.3.7 Actions
4.3.8 Variable Types
4.3.9 One-Liners
4.3.10 Scripting
4.3.11 Overheads
4.3.12 Documentation and Resources
4.4 SystemTap
4.4.1 Probes
4.4.2 Tapsets
4.4.3 Actions and Built-ins
4.4.4 Examples
4.4.5 Overheads
4.4.6 Documentation and Resources
4.5 perf