Table Of ContentOpenStack
Operations Guide
SET UP AND MANAGE YOUR OPENSTACK CLOUD
Tom Fifield, Diane Fleming, Anne Gentle,
Lorin Hochstein, Jonathan Proulx,
Everett Toews & Joe Topjian
Join the global community!
OVER 70
GLOBAL
USER
GROUPS
Get Involved and get more
out of OpenStack!
Take the User Survey and influence the OpenStack Roadmap
Find a local User Group near you and attend a meet up
Attend a Training Course
OpenStack Operations Guide
by Tom Fifield, Diane Fleming, Anne Gentle, Lorin
Hochstein, Jonathan Proulx, Everett Toews, and Joe
Topjian
OpenStack Operations Guide
by Tom Fifield, Diane Fleming, Anne Gentle, Lorin Hochstein, Jonathan Proulx, Everett Toews, and Joe
Topjian
Copyright © 2014 OpenStack Foundation. All rights reserved.
Printed in the United States of America.
Published by O'Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O'Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://my.safaribooksonline.com). For more information, contact our corpo‐
rate/institutional sales department: 800-998-9938 or [email protected].
Editors: Andy Oram and Brian Anderson Interior Designer: David Futato
Cover Designer: Karen Montgomery
March 2014: First Edition
See http://oreilly.com/catalog/errata.csp?isbn=9781491946954 for release details.
Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of
O'Reilly Media, Inc. OpenStack Operations Guide, the image of a Crested Agouti, and related trade dress
are trademarks of O'Reilly Media, Inc.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
trademarks. Where those designations appear in this book, and O'Reilly Media, Inc., was aware of a trade‐
mark claim, the designations have been printed in caps or initial caps.
While every precaution has been taken in the preparation of this book, the publisher and authors assume
no responsibility for errors or omissions, or for damages resulting from the use of the information con‐
tained herein.
978-1-491-94695-4
[LSI]
Table of Contents
Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
1. Provisioning and Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Automated Deployment 21
Disk Partitioning and RAID 22
Network Configuration 23
Automated Configuration 23
Remote Management 24
2. Cloud Controller Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Hardware Considerations 25
Separation of Services 27
Database 27
Message Queue 27
Application Programming Interface (API) 28
Extensions 28
Scheduler 29
Images 29
Dashboard 30
Authentication and Authorization 30
Network Considerations 30
3. Scaling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
The Starting Point 33
Adding Controller Nodes 35
Segregating Your Cloud 35
iii
Cells and Regions 36
Availability Zones and Host Aggregates 37
Scalable Hardware 38
Hardware Procurement 38
Capacity Planning 39
Burn-in Testing 39
4. Compute Nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
CPU Choice 41
Hypervisor Choice 41
Instance Storage Solutions 42
Off Compute Node Storage – Shared File System 42
On Compute Node Storage – Shared File System 43
On Compute Node Storage – Non-shared File System 44
Issues with Live Migration 44
Choice of File System 44
Overcommitting 45
Logging 45
Networking 45
5. Storage Decisions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
OpenStack Storage Concepts 47
Object Storage 48
Block Storage 48
File-level Storage 49
Choosing Storage Back-ends 49
Commodity Storage Back-end Technologies 51
Notes on OpenStack Object Storage 53
6. Network Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Management Network 55
Public Addressing Options 55
IP Address Planning 56
Network Topology 57
VLANs 58
Multi-NIC 59
Multi-host and Single-host Networking 59
Services for Networking 59
NTP 59
DNS 60
7. Example Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
iv | Table of Contents
Overview 61
Rationale 62
Why Not Use the OpenStack Network Service (quantum)? 64
Why Use Multi-host Networking? 64
Detailed Description 64
Optional Extensions 66
8. Lay of the Land. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Client Command Line Tools 69
Installing the Tools 70
Administrative Command Line Tools 70
Getting Credentials 71
Command Line Tricks and Traps 72
Servers and Services 74
Diagnose your compute nodes 76
Network 77
Users and Projects 77
Running Instances 78
9. Managing Projects and Users. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Projects or Tenants? 81
Managing Projects 81
Adding Projects 81
Quotas 82
Set Compute Service Quotas 83
Set Block Storage quotas 85
User Management 87
Creating New Users 87
Associating Users with Projects 88
Customizing Authorization 89
Users that Disrupt Other Users 91
10. User-facing Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Images 93
Adding Images 93
Deleting Images 94
Other CLI Options 94
The Image Service and the Database 94
Example Image Service Database Queries 95
Flavors 95
How do I modify an existing flavor? 96
Security groups 97
Table of Contents | v
Block Storage 99
Block Storage Creation Failures 100
Instances 101
Starting Instances 101
Instance Boot Failures 101
Instance-specific Data 102
Associating Security Groups 104
Floating IPs 104
Attaching Block Storage 105
Taking Snapshots 106
Ensuring snapshots are consistent 107
Instances in the Database 108
11. Maintenance, Failures, and Debugging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Cloud Controller and Storage Proxy Failures and Maintenance 111
Planned Maintenance 111
Rebooting a cloud controller or Storage Proxy 111
After a Cloud Controller or Storage Proxy Reboots 112
Total Cloud Controller Failure 112
Compute Node Failures and Maintenance 112
Planned Maintenance 113
After a Compute Node Reboots 113
Instances 114
Inspecting and Recovering Data from Failed Instances 114
Volumes 117
Total Compute Node Failure 117
/var/lib/nova/instances 118
Storage Node Failures and Maintenance 118
Rebooting a Storage Node 119
Shutting Down a Storage Node 119
Replacing a Swift Disk 119
Handling a Complete Failure 120
Configuration Management 121
Working with Hardware 121
Adding a Compute Node 121
Adding an Object Storage Node 122
Replacing Components 122
Databases 122
Database Connectivity 123
Performance and Optimizing 123
HDWMY 123
Hourly 123
vi | Table of Contents
Daily 123
Weekly 124
Monthly 124
Quarterly 124
Semi-Annually 124
Determining which Component Is Broken 124
Tailing Logs 125
Running Daemons on the CLI 125
Example of Complexity 125
Upgrades 126
Uninstalling 127
12. Network Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Using “ip a” to Check Interface States 129
Network Traffic in the Cloud 130
Finding a Failure in the Path 131
tcpdump 131
iptables 133
Network Configuration in the Database 133
Manually De-Associating a Floating IP 133
Debugging DHCP Issues 134
Debugging DNS Issues 137
13. Logging and Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Where Are the Logs? 139
Cloud Controller 139
Compute Nodes 139
Block Storage Nodes 140
How to Read the Logs 140
Tracing Instance Requests 141
Adding Custom Logging Statements 142
RabbitMQ Web Management Interface or rabbitmqctl 143
Centrally Managing Logs 143
rsyslog Client Configuration 143
rsyslog Server Configuration 144
StackTach 145
Monitoring 145
Process Monitoring 146
Resource Alerting 146
OpenStack-specific Resources 147
Intelligent Alerting 149
Table of Contents | vii
Trending 150
14. Backup and Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
What to Backup 153
Database Backups 154
File System Backups 154
Compute 154
Image Catalog and Delivery 155
Identity 155
Block Storage 155
Object Storage 155
Recovering Backups 155
15. Customize. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
DevStack 157
Middleware Example 160
Nova Scheduler Example 165
Dashboard 170
16. Upstream OpenStack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Getting Help 171
Reporting Bugs 172
Confirming & Prioritizing 173
Bug Fixing 174
After the Change is Accepted 174
Join the OpenStack Community 175
Features and the Development Roadmap 175
How to Contribute to the Documentation 177
Security Information 177
Finding Additional Information 178
17. Advanced Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Differences between various drivers 179
Periodic tasks 180
Specific configuration topics 181
OpenStack Compute (Nova) 181
A. Use Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
B. Tales From the Cryp^H^H^H^H Cloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
viii | Table of Contents
