Appendix A

109 Pages·2016·1.49 MB·English

by  John M. Fulton

Hyper-Converged Red Hat OpenStack
Platform 10 and Red Hat Ceph Storage 2
John M. Fulton
Version 3.14, 2017-1-14
Table of Contents
1. Comments and Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1. Staying In Touch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2. Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3. Technical Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3.1. Using the RHsyseng HCI GitHub Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4. Hardware Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4.1. Required Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4.2. Recommended Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5. Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

5.1. Deploy the Undercloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

5.2. Register and Introspect Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

6. Define the Overcloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.1. Custom Enviornment Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.2. Network Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.3. Hyper Converged Role Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6.4. Ceph Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

6.5. Overcloud Layout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

7. Resource Isolation and Tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

7.1. Nova Reserved Memory and CPU Allocation Ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

7.2. Ceph NUMA Pinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.3. Reduce Ceph Backfill and Recovery Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

7.4. Regarding tuned. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

8. Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

8.1. Verify Ironic Nodes are Available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

8.2. Run the Deploy Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

8.3. Verify the Deployment Succeeded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

8.4. Configure Controller Pacemaker Fencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

9. Operational Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

9.1. Configuration Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

9.2. Adding Compute/Ceph Storage Nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

9.3. Removing Compute/Ceph Storage Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

10. Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Appendix A: Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Appendix B: References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Appendix C: Environment Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

C.1. OSPd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
C.2. Overcloud Controller / Ceph Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

C.3. Overcloud Compute / Ceph OSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

C.4. Network Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Appendix D: Custom Heat Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Appendix E: Nova Memory and CPU Calculator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Appendix F: Example Fencing Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Appendix G: GitHub Repository of Example Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
100 East Davie Street
Raleigh NC 27601 USA
Phone: [phone]
Phone: [phone]
Fax: [phone]
PO Box 13588
Research Triangle Park NC 27709 USA
Linux is a registered trademark of Linus Torvalds. Red Hat, Red Hat Enterprise
Linux and the Red Hat "Shadowman" logo are registered trademarks of Red Hat,
Inc. in the United States and other countries.
Ceph is a registered trademark of Red Hat, Inc.
UNIX is a registered trademark of The Open Group.
Intel, the Intel logo and Xeon are registered trademarks of Intel Corporation or its
subsidiaries in the United States and other countries. All other trademarks
referenced herein are the property of their respective owners.
© 2017 by Red Hat, Inc. This material may be distributed only subject to the terms
and conditions set forth in the Open Publication License, V1.0 or later (the latest
version is presently available at http://www.opencontent.org/openpub/).
The information contained herein is subject to change without notice. Red Hat, Inc.
shall not be liable for technical or editorial errors or omissions contained herein.
Distribution of modified versions of this document is prohibited without the
explicit permission of Red Hat Inc.
Distribution of this work or derivative of this work in any standard (paper) book
form for commercial purposes is prohibited unless prior permission is obtained
from Red Hat Inc.
The GPG fingerprint of the [email protected] key is: CA 20 86 86 2B D6 9D FC 65
F6 EC C4 21 91 80 CD DB 42 A6 0E
Send feedback to [email protected]
www.redhat.com 1 [email protected]
1. Comments and Feedback
In the spirit of open source, we invite anyone to provide feedback and comments on any reference
architectures. Although we review our papers internally, sometimes issues or typographical errors are
encountered. Feedback allows us to not only improve the quality of the papers we produce, but allows
the reader to provide their thoughts on potential improvements and topic expansion to the papers.
Feedback on the papers can be provided by emailing [email protected]. Please refer to the
title within the email.
1.1. Staying In Touch
Join us on some of the popular social media sites where we keep our audience informed on new
reference architectures as well as offer related information on things we find interesting.
1.1.1. Like us on Facebook:
https://www.facebook.com/rhrefarch
1.1.2. Follow us on Twitter:
https://twitter.com/RedHatRefArch
1.1.3. Plus us on Google+
https://plus.google.com/u/0/b/114152126783830728030/
[email protected] 2 www.redhat.com
2. Executive Summary
This reference architecture describes how to deploy Red Hat OpenStack Platform and Red Hat Ceph
Storage in a way that both the OpenStack Nova Compute services and the Ceph Object Storage Daemon
(OSD) services reside on the same node. A server which runs both compute and storage processes is
known as a hyper-converged node. There is increasing interest in the field for hyper-convergence for
cloud (NFVi and Enterprise) deployments. The reasons include smaller initial deployment foot prints, a
lower cost of entry, and maximized capacity utilization.
The first section of this reference architecture provides a technical summary for an implementer to
quickly deploy a hyper-converged overcloud by using templates and scripts from this reference
architecture on GitHub.
The second section provides general hardware and network guidance.
The third section covers software prerequisites for the undercloud and managing hardware with
Ironic.
The fourth section covers how to define a hyper-converged overcloud in Heat which is stored in a
deployment plan.
The fith section covers how to isolate resources in a hyper-converged overcloud to address contention
between OpenStack and Ceph which could result in degradation of either service.
The sixth section covers how to deploy a hyper-converged overcloud using the deployment plan
defined in the previous sections.
The seventh section covers some of the operational concerns of running a hyper-converged OpenStack
and Ceph deployment. It covers configuration updates, adding new Compute/Ceph Storage nodes, and
removing running Compute/Ceph Storage nodes.
This reference architecture has been completed with Red Hat Enterprise Linux 7.3, Red Hat OpenStack
Platform 10, Red Hat OpenStack Platform director (OSPd) version 10, and Red Hat Ceph Storage 2.0. All
of the steps listed were performed by the Red Hat Systems Engineering team. The complete use case
was deployed in the Systems Engineering lab on bare metal servers, except where otherwise noted.
Hyper-converged deployments in Red Hat OpenStack Platform version 10 are Tech

Preview. In order for the steps in this reference architecture to be supported, a
support exception must be filed.
www.redhat.com 3 [email protected]
3. Technical Summary
This reference architecture describes the step-by-step procedures to deploy Red Hat OpenStack
Platform and Red Hat Ceph Storage in a way that both the OpenStack Nova Compute services and the
Ceph Object Storage Daemon (OSD) services reside on the same node.
In order to facilitate the implementation process of this reference architecture, all of the scripts and
Heat templates may be accessed directly on GitHub. The following section shows an example of how an
implementer may use the GitHub repository to make changes to implement a similar environment.
3.1. Using the RHsyseng HCI GitHub Repository
After installing the undercloud as described in Deploy the Undercloud, ssh into the OSPd server as the
stack user and clone the RHsyseng HCI GitHub Repository:
git clone https://github.com/RHsyseng/hci
Copy the custom-templates directory, provided in the repository, and customize the templates as
described in Define the Overcloud.
cp -r hci/custom-templates ~/
Copy the nova_mem_cpu_calc.py script as provided in the repository:
cp hci/scripts/nova_mem_cpu_calc.py ~/
Run nova_mem_cpu_calc.py to determine the appropriate resource isolation required for the HCI
deployment as described in Resource Isolation and Tuning and then update the Heat environment
templates as described in Nova Memory and CPU Calculator. Ensure the overcloud will have access to
NUMA related packages as descrbed in Ceph NUMA Pinning.
Copy the deploy.sh script and use the script to deploy the overcloud as described in Deployment.
cp hci/scripts/deploy.sh ~/
~/deploy.sh
While the above steps provide a quick way to modify and potentially create a similar

environment as this reference architecture, it is not meant to replace the
comprehensiveness of this full document.
[email protected] 4 www.redhat.com
4. Hardware Recommendations
This reference architecture focuses on:
• Providing configuration instruction details
• Validating the interoperability of Red Hat OpenStack Platform Nova Computes and Red Hat Ceph
Storage on the same physical servers.
• Providing automated methods to apply resource isolation to avoid contention between Nova
Compute and Ceph OSD services.
Red Hat’s experience with early hyper-converged adopters reflect a wide variety of hardware
configurations. Baseline hardware performance and sizing recommendations for non-hyper-
converged Ceph clusters can be found in the Hardware Selection Guide for Ceph.
Additional considerations for hyper-converged Red Hat OpenStack Platform with Red Hat Ceph Storage
server nodes include:
• Network: the recommendation is to configure 2x 10GbE NICs for Ceph. Additional NICs are
recommended to meet Nova VM workload networking requirements that include bonding of NICs
and trunking of VLANs.
• RAM: the recommendation is to configure 2x RAM needed by the resident Nova VM workloads.
• OSD Media: the recommendation is to configure 7,200 RPM enterprise HDDs for general-purpose
workloads or NVMe SSDs for IOPS-intensive workloads. For workloads requiring large amounts of
storage capacity, it may be better to configure separate storage and compute server pools (non
hyper-converged).
• Journal Media: the recommendation is to configure SAS/SATA SSDs for general-purpose workloads
or NVMe SSDs for IOPS-intensive workloads.
• CPU: the recommendation is to configure a minimum dual-socket 16-core CPUs for servers with
NVMe storage media, or dual-socket 10-core CPUs for servers with SAS/SATA SSDs.
Details of the hardware configuration for this reference architecture can be found in Appendix:
Environment Details.
4.1. Required Servers
The minimum infrastructure requires at least six bare metal servers and either a seventh bare metal
server or virtual machine hosted separately, not hosted on the six bare metal servers. These servers
should be deployed in the following roles:
• 1 OSPd server (can be virtualized for small deployments)
• 3 Cloud Controllers/Ceph Monitors (Controller/Mon nodes)
• 3 Compute Hypervisors/Ceph Storage servers (Compute/OSD nodes)
www.redhat.com 5 [email protected]
As part of this reference architecture, a fourth Compute/Ceph Storage node is added to demonstrate
scaling of an infrastructure.
Additional Compute/Ceph storage nodes may be initially deployed or added later.
 However, for deployments spanning more than one datacenter rack (42 nodes), Red
Hat recommends the use of standalone storage and compute, and not a hyper-
converged approach.
4.2. Recommended Networks
This reference architecture uses six networks to serve the roles described in this section. How these
networks may be trunked as VLANs to connect to the servers is illustrated in the Network Separation
Diagram. Further details of the Networks in this reference architecture are located in Appendix:
Environment Details.
4.2.1. Ceph Cluster Network
The Ceph OSDs use this network to balance data according to the replication policy. This private
network only needs to be accessed by the OSDs.
In a hyper-converged deployment, the compute role needs access to this network. Thus, Define the
Overcloud, describes how to modify nic-configs/compute-nics.yaml to ensure that OSPd deploys
compute nodes with a connection to this network.
The Heat Provider that OSPd uses to define this network can be referenced with the following:
 OS::TripleO::Network::StorageMgmt: /usr/share/openstack-tripleo-heat-
templates/network/storage_mgmt.yaml
4.2.2. Ceph Storage
The Ceph monitor nodes are accessed via this network.
The Heat Provider that OSPd uses to define this network can be referenced with the following:
 OS::TripleO::Network::Storage: /usr/share/openstack-tripleo-heat-
templates/network/storage.yaml
4.2.3. External
OSPd uses the external network to download software updates for the overcloud, and the cloud
operator uses this network to access Director to manage the overcloud.
[email protected] 6 www.redhat.com
The Controllers use the external network to route traffic to the Internet for tenant services that are
externally connected via reserved floating IPs. Overcloud users use the external network to access the
overcloud.
The Compute nodes do not need to be directly connected to the external network, as their instances
communicate via the Tenant network to the Controllers who then route external traffic on their behalf
to the external network.
The Heat Provider that OSPd uses to define this network can be referenced with the following:
 OS::TripleO::Network::External: /usr/share/openstack-tripleo-heat-
templates/network/external.yaml
4.2.4. OpenStack Internal API
OpenStack provides both public facing and private API endpoints. This is an isolated network for the
private endpoints.
The Heat Provider that OSPd uses to define this network can be referenced with the following:
 OS::TripleO::Network::InternalApi: /usr/share/openstack-tripleo-heat-
templates/network/internal_api.yaml
4.2.5. OpenStack Tenant Network
OpenStack tenants create private networks implemented by VLAN or VXLAN on this network.
The Heat Provider that OSPd uses to define this network can be referenced with the following:
 OS::TripleO::Network::Tenant: /usr/share/openstack-tripleo-heat-
templates/network/tenant.yaml
4.2.6. OSPd Provisioning
OSPd serves DHCP and PXE services from this network to install the operating system and other
software on the overcloud nodes, Controller/Mon and Compute/OSD, from baremetal. OSPd uses this
network to manage the overcloud nodes, and the cloud operator uses it to access the overcloud nodes
directly by ssh if necessary. The overcloud nodes must be configured to PXE boot from this network
provisioning.
www.redhat.com 7 [email protected]