SlideShare a Scribd company logo
© 2014 VMware Inc. All rights reserved.
OpenStack: Toward a More
Resilient Cloud
Or: “How I Learned to Stop Worrying &
Embrace My Inner Cloudiness”
Mark T. Voelker
OpenStack Architect
Percona University Smart Data Event
February 12, 2015
• Who is this guy?
• A little background on OpenStack
• Building more resilient clouds
– Withstanding failures
– Quickly recovering from failures
• Questions?
@marktvoelker
• OpenStack Architect @ VMware, OpenStack ATC, Ex-StackForge Puppet
core dev, Triangle OpenStack Meetup founder, OS Foundation Member #54
• Fact: can be bribed with doughnuts
• Currently works in VMware’s Software Defined Datacenter R&D group
• In copious (hah!) spare time: OpenStack solutions, Big Data, Massively
Scalable Data Centers, Devops, making sawdust with extreme prejudice
• Tech lead, manager, software developer, architect
• Started in OpenStack in 2011 at the Diablo Design Summit

Recommended for you

What's New in Grizzly & Deploying OpenStack with Puppet
What's New in Grizzly & Deploying OpenStack with PuppetWhat's New in Grizzly & Deploying OpenStack with Puppet
What's New in Grizzly & Deploying OpenStack with Puppet

The document outlines the agenda for the May Triangle OpenStack Meetup, including: - Welcome and introductions starting at 4:30pm - Two technical talks from 4:45-5:30pm on new features in the Grizzly release of OpenStack and automating OpenStack with Puppet - An open question and answer forum from 5:30-5:45pm - Pizza will be served around 5:45pm Bios of the three meetup organizers are also provided.

puppetgrizzlymeetup
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...

This Edureka OpenStack Cloud tutorial explains how to setup your own OpenStack Cloud infrastructure and also tells you the different OpenStack Cloud deployment models. It also focuses on how to setup different types of cloud infrastructure according to your needs like storage focused, network focused etc. Below are the topics covered in this tutorial: 1. Cloud Computing Virtualization Concepts 2. Cloud Services 3. Types of Cloud 4. What is Openstack? 5. Openstack Architecture 6. Openstack Community 7. Openstack Cloud Deployment Models 8. Openstack Cloud Design Guide

openstack edurekaedurekaopenstack services
Build public private cloud using openstack
Build public private cloud using openstackBuild public private cloud using openstack
Build public private cloud using openstack

The document discusses building a public or private cloud using OpenStack. It begins with an introduction to cloud computing and OpenStack. OpenStack is an open source cloud platform that is scalable and used by many large companies. The document then covers use cases for OpenStack like building test environments, provides examples of OpenStack implementations at CERN and Rackspace, and discusses how the group has setup OpenStack and built a sample Rails application using OpenStack Swift for photo storage. It acknowledges the challenges of implementing OpenStack and hopes to continue using it to support building large scale web applications.

….I’ve built a few clouds.
Today’s talk won’t be overly formal….
…because I tend to get excited by this stuff.
• Who is this guy?
• A little background on OpenStack
• Building more resilient clouds
– Withstanding failures
– Quickly recovering from failures
• Questions?

Recommended for you

2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)
2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)
2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)

Mirantis, the Global Engineering Services leader for OpenStack™ presents 2-day Bootcamp for OpenStack www.mirantis.com/training This two-day intensive course provides hands-on technical training for OpenStack aimed at system administrators and IT professionals looking to get started on an OpenStack Cloud deployment. Each of the two days will consist of lecture, demos and group exercises. Topics include: • OpenStack Overview & Architecture: Project goals and use cases, basic operating and deployment principles • Cloud Usage Patterns: OpenStack codebase overview; creating networks, tenants, roles, troubleshooting; Nexenta Volume Driver • In Production: Deploying OpenStack for real-world use, and practice of OpenStack operation on multiple nodes • Swift Object Storage: use cases, architecture, capabilities, configuration, security and deployment • Advanced Topics: Software Defined Networking, deployment and issues workshop, VMWare/OpenStack comparison PRE-REQUISITES: Comfortable with Linux CLI, understanding of virtualization & hypervisors, Some experience with Linux networking All course materials will be provided by Mirantis, including access to shared compute resources for labs. A light breakfast and lunch will be available to all course participants. Mirantis instructors are active code committers to the OpenStack project, with proven experience building OpenStack clouds in the real world. In parallel to delivering expert training, they also consult for some of the notable global companies using OpenStack – including Cisco, NASA, Dell and Internap.

quantum networknexentalinux
OpenStack Explained: Learn OpenStack architecture and the secret of a success...
OpenStack Explained: Learn OpenStack architecture and the secret of a success...OpenStack Explained: Learn OpenStack architecture and the secret of a success...
OpenStack Explained: Learn OpenStack architecture and the secret of a success...

OpenStack can help your business in cutting costs and have a faster time to market. A lot of people are looking at OpenStack as an alternative to VMware and most of the vendors are trying to let you think that visualization is cloud. While Cloud implies a virtualized environment, virtualization is not a cloud. This ebook will go through the concept of Cloud and help you understand the architecture of OpenStack and its benefits. It also explores DevOps and reveal the "secret ingredient" to have a successful cloud project. This ebook was created to raise funds for the Nepalese population after the Earthquake in 2015.

openstackcloudcloud computing
OpenStack + VMware: Everything You Need to Know (Kilo-edition)
OpenStack + VMware: Everything You Need to Know (Kilo-edition)OpenStack + VMware: Everything You Need to Know (Kilo-edition)
OpenStack + VMware: Everything You Need to Know (Kilo-edition)

Talk presented at OpenStack Summit in Vancouver. Provides an overview of OpenStack + VMware, including VMware Integrated OpenStack.

openstack vmware nova neutron cinder vio
“OpenStack is a global collaboration of developers and cloud computing
technologists producing the ubiquitous open source cloud computing
platform for public and private clouds. The project aims to deliver
solutions for all types of clouds by being simple to implement, massively
scalable, and feature rich. The technology consists of a series of
interrelated projects delivering various components for a cloud
infrastructure solution.”
-- openstack.org
Basically, it’s software to run cloud
services—including compute, network,
storage, and security—and the
community behind that software.
OpenStack: Toward a More Resilient Cloud
OpenStack: Toward a More Resilient Cloud
Source: IDG Connect Survey

Recommended for you

OpenStack Scale-out Networking Architecture
OpenStack Scale-out Networking ArchitectureOpenStack Scale-out Networking Architecture
OpenStack Scale-out Networking Architecture

(1) The document describes OpenStack networking architecture that scales to over 1,000 servers without using Neutron. It uses a layer 3 spine and leaf topology with no VLANs. (2) Key components include Nova network which is distributed, NAT services on the edge, and a VIF driver that sets up routing and DHCP for each VM. (3) Challenges included moving OpenStack targets and an immature Neutron. The architecture provides a solid underlay for SDN and future integration with Neutron.

networkingopenstacksdn
Introduction to Apache CloudStack by David Nalley
Introduction to Apache CloudStack by David NalleyIntroduction to Apache CloudStack by David Nalley
Introduction to Apache CloudStack by David Nalley

Apache CloudStack is a mature, easy to deploy IaaS platform. That doesn't mean that it can be done without thought or preparation. Learn how CloudStack can be most efficiently deployed, and the problems to avoid in the process. About David Nalley David is a recovering sysadmin with a decade of experience. He’s a committer on the Apache CloudStack (incubating) project, a contributor to the Fedora Project and the Vice President of Infrastructure at the Apache Software Foundation.

apache cloudstackapache software foundationdavid nalley
An Introduction to OpenStack
An Introduction to OpenStackAn Introduction to OpenStack
An Introduction to OpenStack

I gave this presentation on 5/17 to the New Mexico VMUG in Santa Fe. The presentation provides an overview of OpenStack, what it is (and isn't), and some things you might learn to get started with OpenStack.

openstackcloud
• IRC Channels and Mailing Lists
• User/Meetup Groups
• Social Networking
– Twitter
– LinkedIn
– Facebook
– Ohloh
• Code in cgit, mirrored on GitHub, Bugs/Milestones in Launchpad
• For now…may move to StoryBoard in future
• Over 20 million lines of code by over 1,419 contributors
• Two Annual Design Summit/Conferences (coinciding roughly
w/releases)
• Want to contribute? Start here.
• Don’t be intimidated.
• HolycrapthingsmovereallyreallyfastinOpenStack
• Jump in feet first: be agile and flexible.
• This is going to feel a little different for some of you.
www.meetup.com/Tri
angle-OpenStack-
Meetup
Next meetups: Feb. 24 & Mar. 11!
Now 550!
Horizon
Nova
Neutron
Swift (Object Storage)
Cinder (Block storage)
Glance
(VM Image Service)
Keystone
(Identity Service)
AWS Management Console
EC2
VPC
S3
EBS
Ceilometer
(Telemetry Service)
Trove
(Database Service) Heat
(Orchestration Service)Sahara
(Data Processing Service)

Recommended for you

OpenStack Framework Introduction
OpenStack Framework IntroductionOpenStack Framework Introduction
OpenStack Framework Introduction

Presentation of OpenStack survey to Internet Research Lab at National Taiwan University, Taiwan. OpenStack framework and architecture overview. (ppt slide for download.) Materials collected from various resources, not originally produced by the author. Briefly explained Nova, Swift, Glance, Keystone, and Quantum.

cloud computingcloudopenstack
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander DibboOpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo

The Science and Technology Facilities Council is a UK Research Council which funds research and provides large facilities to the UK Scientific Community. This includes running a Tier 1 site for the LHC computing project, the JASMIN Super Data Cluster and a number of other HPC and HTC facilities. The Scientific Computing Department at the Rutherford Appleton Laboratory has been developing a cloud for use across both sites of the Department and in the wider scientific community. This is an OpenNebula backed by Ceph block storage. I will give a brief background of the project, describe our set up, some use cases and the work we have done around OpenNebula (including a simplified web front-end and a number of hooks to provide us with traceability). I will also discuss how we are creating an elastic boundary between our HTC batch farm and cloud. Author Biography I am a Systems Administrator in the Scientific Computing Department of the UK’s Science and Technology Facilities Council. I work as part of the cloud team and I also work on a number of Grid services including our HTC batch farm for the LHC computing project. Prior to my position here I worked in IT at a SMB focusing on Storage and Virtualisation, in particular Hyper-V and VMWare.

opennebulaconf 2015
Successfully Deliver and Operate OpenStack in Production with VMware VIO
Successfully Deliver and Operate OpenStack in Production with VMware VIOSuccessfully Deliver and Operate OpenStack in Production with VMware VIO
Successfully Deliver and Operate OpenStack in Production with VMware VIO

This document discusses VMware's approach to integrating their virtualization technologies with OpenStack. It emphasizes that VMware contributes to OpenStack projects while also competing to make VMware the best platform for running OpenStack. Specifically, it promotes using vSphere for compute (Nova), NSX for networking (Neutron), and vSAN for block storage (Cinder), arguing these provide the best features for reliability, ease-of-use, and management. It also describes how VMware technologies can help simplify and improve OpenStack operations through tools like vCenter and vCOps.

Library Projects
Supporting Projects
Documentation
Oslo (common code libraries)
Client libraries
Incubated Projects
(may become core
components in the future)
Designate (DNS service)
Zaqar (queuing service)
Gating Projects
CI & Infrastructure
DevStack (deployment script)
Tempest (integration test)
Barbican (key management)
Manila (shared FS as a
service)
OpenStack: Toward a More Resilient Cloud
• Who is this guy?
• A little background on OpenStack
• Building more resilient clouds
– Withstanding failures
– Quickly recovering from failures
• Questions?
What’s a “resilient” cloud?
re·sil·ient
/rəˈzilyənt/
(adjective) Able to withstand or recover quickly from difficult
conditions.

Recommended for you

VMUG22 Filip Verloy VIO
VMUG22 Filip Verloy VIOVMUG22 Filip Verloy VIO
VMUG22 Filip Verloy VIO

VMware Integrated OpenStack (VIO) provides a tightly-integrated product that combines OpenStack APIs with VMware technologies for a less complex build and operation compared to a loosely-integrated framework. VIO includes common OpenStack projects like Nova, Neutron, Cinder, and Glance that are optimized to run on VMware vSphere and integrate with VMware management and automation tools for a unified experience. VIO addresses the challenges of operating OpenStack at scale through this tight integration with VMware technologies and single support contact.

Bridging The Gap: Explaining OpenStack To VMware Administrators
Bridging The Gap: Explaining OpenStack To VMware AdministratorsBridging The Gap: Explaining OpenStack To VMware Administrators
Bridging The Gap: Explaining OpenStack To VMware Administrators

Updated from Kenneth Hui and Scott Lowe's joint talk at the Fall 2013 OpenStack Summit in Hong Kong. This is from a talk given by Cody Bunch and Kenneth Hui at the New England VTUG 2014 Winter Warmer.

openstackvmwarevsphere
Introduction to openstack
Introduction to openstackIntroduction to openstack
Introduction to openstack

OpenStack is an open-source cloud computing platform that manages large pools of compute, storage, and networking resources throughout a datacenter. It includes several independent services like Nova (compute), Neutron (networking), Swift (object storage), and Glance (image service). Hands-on experience with OpenStack can be gained through all-in-one installations or multi-node configurations on physical or virtual machines using various OpenStack distributions from companies like Red Hat, Ubuntu, and Mirantis. Neutron provides virtual networking and integration with technologies like Open vSwitch, namespaces, and plugins to enable multi-tenant isolation.

• Today we’ll primarily focus on the cloud itself
• Workloads running *in* clouds are another story…but we’ve only
got one hour!
8am: “Uh-oh. Something tells me it’s going to be an interesting day in
the datacenter….”
• Hardware Failures
• OpenStack software bugs (yep, those exist)
• Underpinning software failures (database, message queue, etc)
• Operating system failures
• Network/storage/power failures
• Planned maintenance windows
• Hackers and malcontents
• Upgrades
• Automation failures
• “Whoops, did I do that?”
Some causes of outages in the past year
….did you plan for these?
CONFIDENTIAL
24

Recommended for you

Cloud Computing Open Stack Compute Node
Cloud Computing Open Stack Compute NodeCloud Computing Open Stack Compute Node
Cloud Computing Open Stack Compute Node

The latest technology - Cloud Computing that offers its users with a variety of compliments such as ubiquitous access,cheaper rates, etc .

compute nodecloud computingopenstack
Oct meetup open stack 101 clean
Oct meetup open stack 101   cleanOct meetup open stack 101   clean
Oct meetup open stack 101 clean

The document provides information about an OpenStack Meetup happening on October 7th, 2014 in San Antonio, Texas. It includes details about the event location and organizers, as well as an agenda for introductory OpenStack training modules covering an overview of OpenStack, its architecture, and installing OpenStack.

openstackcloudiaas
Cloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: OpenstackCloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: Openstack

Openstack is an open source cloud computing platform that consists of several independent components that work together to provide infrastructure as a service capabilities. It allows users to provision compute, storage, and networking resources on demand in a self-service manner similar to public cloud providers like AWS. Some key components include Nova for compute, Glance for images, Swift for object storage, Cinder for block storage, Neutron for networking, and Keystone for identity services. Openstack can be used to build public, private, or hybrid clouds and supports a variety of use cases and workloads.

openstackcloud
Sometimes things break (in *any* system).
25
Withstand what you can. Quickly recover from the rest.
Because you don’t look this cute when your cloud is down.
OpenStack: Toward a More Resilient Cloud
High
Availability?
Sounds
great--I’ll
take two!
General Premise:
Assume hardware and software fail.
(because, shockingly, that actually happens)
CONFIDENTIAL
28

Recommended for you

OpenStack 101
OpenStack 101OpenStack 101
OpenStack 101

Mark Voelker VMware - OpenStack Architect Tuesday, Oct 20th 10:30 am - Cloud Find more by Mark Voelker: http://www.slideshare.net/markvoelker/

ato2015all things openopenstack
Getting Started with Apache CloudStack
Getting Started with Apache CloudStackGetting Started with Apache CloudStack
Getting Started with Apache CloudStack

This document provides an overview of Apache CloudStack, an open source cloud computing platform. It describes CloudStack's key characteristics including on-demand self-service, broad network access, resource pooling, rapid elasticity, and API access. It outlines CloudStack's support for different cloud service models including SaaS, PaaS, and IaaS and discusses its hypervisor support, zone, pod, and cluster architecture. The document also summarizes CloudStack's management server, high availability features, networking, security groups, and usage accounting capabilities.

saasiaasxen
Hacking apache cloud stack
Hacking apache cloud stackHacking apache cloud stack
Hacking apache cloud stack

The document provides an agenda and overview of a session on hacking Apache CloudStack. The agenda includes introductions, a session on introducing CloudStack, and a hands-on session with DevCloud. The overview discusses what CloudStack is, how it works as an orchestration platform for IAAS clouds, its architecture and core components, and how users can consume and manage resources through it.

apache cloudstack
What Does “HA” Mean in an OpenStack Cloud?
CONFIDENTIAL
29
• Compute
• Multiple clusters
• Consider segmenting with Availability Zones, Host Aggregates, etc
• Consider ability to live migrate instances for hypervisor node
maintenance
• Ensure some capacity buffer for maintenance operations
• Storage
• Avoid single points of failure
• Multiple technologies can be used…but each has it’s own limitations
• Don’t think just Cinder here…your Glance backend and compute
storage matter too!
• Network
• Network disruptions will inevitably occur, so plan for them
• Design for control plane disruption (and pick technology accordingly)
• Control Plane
• May depend on the other things above
• Essential to keeping the cloud operational
• Data Plane
• Stuff that workloads running in the cloud depends on
High Availability Is Part of the Story….
….we need to think a bit about architecture.
(I’ll use a reference architecture from VMware Integrated OpenStack
as an example)
CONFIDENTIAL
30
VIO Architecture – Logical Topology-
Management
Notice something?
There’s a lot of stuff in there that isn’t OpenStack, but upon which
OpenStack depends.
CONFIDENTIAL
32

Recommended for you

Private cloud cloud-phoenix-april-2014
Private cloud cloud-phoenix-april-2014Private cloud cloud-phoenix-april-2014
Private cloud cloud-phoenix-april-2014

Private clouds are cloud infrastructure that resides within a company's own datacenter and is managed internally. They allow a company to have cloud-like capabilities while maintaining control and security over their own data. Private clouds work by using virtualization and a controller to provision and track physical resources like servers, storage, and networks as more capacity is needed. The document discusses the private cloud platforms Eucalyptus, OpenStack, and CloudStack, comparing their architectures, development histories, and strengths for different use cases.

cloudphoenixcloudeucalyptus
Txlf2012
Txlf2012Txlf2012
Txlf2012

This document provides an overview of how to create your own cloud using Apache CloudStack. It discusses the key characteristics of clouds, different cloud service and deployment models supported by CloudStack, and the core components that make up a CloudStack deployment including zones, pods, clusters, primary and secondary storage, virtual routers, hypervisors, and the management server. The document also touches on CloudStack's networking, security, high availability, resource allocation, and usage accounting features.

CloudStack - LinuxFest NorthWest
CloudStack - LinuxFest NorthWestCloudStack - LinuxFest NorthWest
CloudStack - LinuxFest NorthWest

This presentation provides an overview of Apache CloudStack, an open source cloud computing platform. It discusses CloudStack's history and licensing, its ability to provide infrastructure as a service across multiple hypervisors, and how it enables multi-tenancy, high availability, scalability, and resource allocation. Key CloudStack components and concepts are also summarized, such as networking models, security groups, primary and secondary storage, usage tracking, and its management architecture.

open sourcecloud computingcloudstack
VIO Architecture – RabbitMQ
• RabbitMQ is a messaging broker - an
intermediary for messaging. It gives
applications a common platform to send and
receive messages, and the messages a safe
place to live until received.
• RabbitMQ is the default AMQP server used by
OpenStack services (Qpid is also an option,
some support for 0mq). In production clouds,
this should be a highly available infrastructure
component.
• The OpenStack subcomponents (nova-
scheduler to nova-compute, for example)
communicate among themselves using this
hosted message queue service. They also
utilize the hosted Memcached services for
caching authentication tokens etc. As always,
they persist data to the Database.
• Component-to-Component communications
(Nova-> Neutron) is done via REST.
• For more details about the HA implementation
of RabbitMQ, please click here.
VIO Architecture – Database
• The database cluster is at the heart of the
infrastructure. Typically MySQL or MariaDB
are used, but other options such as
PostgreSQL are also supported.
• The VIO MariaDB implementation makes use
of a 3-node Galera cluster, which in itself is
Active-Active-Active. However, since some
OpenStack services enforce table locking,
reads and writes are directed to a single
node via the Load Balancers.
• Note that this database is for management
plane data. OpenStack services that store
data as part of their purpose may use
additional DB’s. For example: Ceilometer
may store meter data to MySQL, Mongo,
PostgreSQL, HBase, or DB2.
VIO Architecture – Load Balancers
• Most OpenStack Services run on the
Controllers, which are mirrored on each
controller VM and load-balanced. They
are accessible via the internal virtual IP.
• Some of the services, such as the
Dashboard, compute-api, glance-api,
keystone, cinder, neutron and
novncproxy are exposed to the end
users via the load balancer’s public virtual
IP.
• Likewise, the hosted Message Queue
(RabbitMQ) and Memcached services are
also load-balanced between 2 VMs.
• For the Database Service, the load
balancer is configured to use a primary
DB VM. In case of failure it will switch to
one of the two backup DB VMs.
• Load Balancers user here are HAProxy
with Keepalived for high availability
Etc, etc, etc
CONFIDENTIAL
36
• There’s a network connecting all that stuff
• It’s running (as virtual machines) on servers which have operating systems
• Things may get wonky if NTP fails and clocks are out of sync
• If DNS can’t resolve, Bad Things ™ will probably happen

Recommended for you

State of the Container Ecosystem
State of the Container EcosystemState of the Container Ecosystem
State of the Container Ecosystem

This document discusses containerization and the Docker ecosystem. It begins by describing the challenges of managing different software stacks across multiple environments. It then introduces Docker as a solution that packages applications into standardized units called containers that are portable and can run anywhere. The rest of the document covers key aspects of the Docker ecosystem like orchestration tools like Kubernetes and Docker Swarm, networking solutions like Flannel and Weave, storage solutions, and security considerations. It aims to provide an overview of the container landscape and components.

PowerPoint Presentation
PowerPoint PresentationPowerPoint Presentation
PowerPoint Presentation

- Alfresco solutions can be provisioned and deployed on Amazon Web Services (AWS) infrastructure as a service (IaaS) cloud platform. This provides advantages like infinite resources, pay as you go pricing, rapid scalability, and provisioning speed. - AWS services like EC2, S3, and EBS allow deployment of virtual servers and storage. DevOps tools like Puppet can help automate configuration and provisioning. - Deploying Alfresco on AWS provides opportunities for auto scaling, historical usage statistics from monitoring, and a potential Alfresco admin console for management of the AWS environment. However, challenges include network and storage configuration and automated provisioning of new nodes

alfresco
Openstack presentation
Openstack presentationOpenstack presentation
Openstack presentation

OpenStack is an open source cloud computing platform that consists of several components including Keystone for identity, Glance for images, Nova for compute, Cinder for block storage, and Quantum for networking. The document provides an overview of each component, describing their main functions and how they interact through messaging queues like RabbitMQ. It also describes the original "Nova networking" approach and how Quantum improved on this by decoupling logical and physical networking and providing plugins to support technologies like Open vSwitch.

openstack
• Consider whether you want active/active or active/passive
• Setup and tooling differs a bit, but I generally like active/active
• Note that docs.openstack.org has an HA Guide
• Currently undergoing lots of updates…patches welcome!
• Prioritize HA for the control plane
• That also means thinking about your database, network, and RPC bus
• Note: HA == more hardware
• Some components need at least 3 nodes
• Mitigate by virtualizing control plane
• Stuff OpenStack needs to run: message brokers
• Check out RabbitMQ clustering and mirrored queues
• Check out Galera for MySQL/MariaDB
• I often see Percona XtraDB in the wild
• Frontend with an HAProxy/Keepalived pair
• Memcached for caching
• Don’t do rabbit clustering
over a WAN
• Be aware of the SELECT…
FOR UPDATE issue
• Long story short: Neutron and some parts of Nova invoke an SQL
pattern known as “SELECT…FOR UPDATE” which Galera doesn’t
support due to issues with cross-node locking.
• Can cause deadlock-like symptoms due to locks not being
replicated.
• Neutron/nova code being refactored, but will likely not be done
soon.
• Meanwhile: use HAProxy to send writes to a single Galera node
and you should be fine
• With the obvious scalability bottleneck
• More info here, here, & here.
• Thank Jay Pipes of Mirantis & Peter
Boros of Percona for the find!

Recommended for you

Taking the open cloud to 11
Taking the open cloud to 11Taking the open cloud to 11
Taking the open cloud to 11

This document summarizes a presentation about the open source CloudStack cloud computing platform. CloudStack provides tools for provisioning and managing virtual infrastructure as a service, including APIs for self-service provisioning, distributed management of compute, storage and networking resources, and high availability features. The presentation outlines CloudStack's history and goals of multi-tenancy, broad hardware support, orchestration of resources behind firewalls, and scalability. It describes key CloudStack components and features such as the management server, domains and users, hypervisor and storage support, resource allocation policies, and networking functionality.

iaaslinuxopen source
OpenStack Deployment in the Enterprise
OpenStack Deployment in the Enterprise OpenStack Deployment in the Enterprise
OpenStack Deployment in the Enterprise

Shannon McFarland, Principal Engineer, discusses OpenStack deployment in the enterprise at Cisco Connect Toronto 2015.

openstack deploymentcisco connectenterprise
Directions for CloudStack Networking
Directions for CloudStack  NetworkingDirections for CloudStack  Networking
Directions for CloudStack Networking

Presented at the CloudStack Silicon Valley User Group in September 2015 at Nuage Networks. Discussed impact of containers, emerging software defined networking platforms, NFV, IPv6 and performance.

software defined networkingovnipv6
• Pick a highly available storage to back Glance
• Pick a highly available storage backend for Cinder too
– SAN, distributed, software defined, plethora of options here
• Use Keepalived/HAProxy to front-end multiple API servers
• Or another load balancer technology of your choice
• Can be deployed as dedicated nodes for scale, or cohabitate
• Data plane network: DVR & Provider Network Extensions
• Distributed Virtual Routers are a new experimental feature in Juno (not yet
ready for production)
• Please go test it and report/fix bugs!
• Provider networks essentially punt the availability issue to your physical
network
• Allows you to use standard tools like virtual port channels and VRRP
• Also highly performant
• Network: consider your backend technology
– Neutron offers a variety of plugins for various open source and vendor-supplied
network technology
– Physical networks need usual redundancy protections
– Overlays are popular for segmentation/isolation; some scale better than others
– Shameless plug: check out VMware NSX which has been used in some very
large OpenStack deployments!
• Actually, most of these techniques and technologies are things that
seasoned developers and sysadmins have used before.
• It doesn’t take a genius to learn lessons from the past and apply
them, tweak them, and tune them (but it’s a fair amount of work).
Simple Rules of Thumb
Planning for availability can go to extreme levels, so start simple:
– Can I take any one [server | switch | storage unit] out of service in my control
plane and still be operational?
– For all of the above, what’s the impact?
• Performance hit?
• Capacity loss?
• World is broken?!?
– For all the above, how easy is it to reintroduce a repaired/replaced $thing?
• Is there a recovery period that will further impact performance?
• Is it a complex procedure?
• Does the procedure cause more $things to be temporarily unavailable?
– For all of the above, how can I monitor & alert for failure?
Once you have that down, dig deeper to your heart’s content.
CONFIDENTIAL
44

Recommended for you

Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxBetter, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux

Software is eating infrastructure. Migrating reliability and scalability responsibilities up the stack from specialized hardware to software, cloud orchestration platforms such as Apache CloudStack (ACS) and object stores such as Riak CS increase the utilization and density of compute and storage resources by dynamically shifting workloads based on demand. Together, these platform can saturate compute and storage of 1000s of commodity hosts with strong operational visibility and end-user self-service. This presentation explores cloud design strategies to achieve high availability and reliability using commodity components. It then applies these strategies using Apache CloudStack and Riak CS.

cloudstack cloud objectstorage riak riakcs basho
Cloud Native Camel Riding
Cloud Native Camel RidingCloud Native Camel Riding
Cloud Native Camel Riding

Using apache camel for microservices and integration then deploying and managing on Docker and Kubernetes. When we need to make changes to our app, we can use Fabric8 continuous delivery built on top of Kubernetes and OpenShift.

camelfabric8distributed systems
Stackato v2
Stackato v2Stackato v2
Stackato v2

Presentation of ActiveState Stackato to Copenhagen Ruby Brigade More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato

stackatoactivestateruby
Recover Quickly
CONFIDENTIAL
45
Rule 1: Assume You Will Need to Change Stuff
• Change can be a lot of things:
– Hardware or software upgrades/patches/replacements
– Configuration tweaks
– Adding or subtracting capacity
– All systems change over time; OpenStack clouds are no exception.
CONFIDENTIAL
46
“Change in OpenStack?
Yeah, I’m good with
that…”
Rule 2: Assume You Can’t Manually Log In To All
The Nodes To Make Those Changes
• OpenStack is a series of cooperating distributed systems
– That means you could (potentially) have a lot of nodes
– Software & config must often be placed on many machines
– Manual changes == slow changes != quick recovery
CONFIDENTIAL
47
“I guess multitasking
only speeds things up
to a certain extent…”
Rule 3: Assume You Will Need To Test Stuff
• It’s a good idea to have a small test cloud where you can examine the
impact of changes
• When possible, roll out changes to a portion of your cloud and evaluate
before rolling out the rest
– Note: this means you need tests and monitoring…otherwise you don’t know
what “ok” looks like
CONFIDENTIAL
48
“It’s 3am and I’m still
debugging in
production…maybe I
should have taken the
time to set up a test
environment and
automate some testing
after all…”

Recommended for you

Chicago Microservices Integration Talk
Chicago Microservices Integration TalkChicago Microservices Integration Talk
Chicago Microservices Integration Talk

This document discusses integration in the age of DevOps. It describes how microservices help solve the problem of decoupling services and teams to move quickly at scale. Apache Camel is presented as a solution for integration that allows for reliable and distributed integration through mechanisms like messaging. Kubernetes and Docker are discussed as platforms that help develop and run microservices locally and at scale by providing automation, configuration, isolation and service discovery capabilities.

apache camelintegrationmicroservices
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OS

Introduction to Apache Mesos and DC/OS present at Los Angeles UUAC and Los Angeles Mesos users group meetup on Jan 11, 2018

mesos dc/os kubernetes containers docker cloud
Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWSMigrating enterprise workloads to AWS
Migrating enterprise workloads to AWS

This document provides an overview of migrating applications and workloads to AWS. It discusses key considerations for different migration approaches including "forklift", "embrace", and "optimize". It also covers important AWS services and best practices for architecture design, high availability, disaster recovery, security, storage, databases, auto-scaling, and cost optimization. Real-world customer examples of migration lessons and benefits are also presented.

legacytoolsmicrosoft
Pile of
Bash
Scripts
• Software developers and operators are increasingly the same
people.
– Agile development
– Automate (almost) everything
– Treat config & changes as you would code
– Continuous integration, testing, deployment
– Incremental change & iteration
– Unified tooling & versioning
– Critical approach to working at scale
– Really useful for building resilient clouds
Image courtesy of Rajiv Pant (http://en.wikipedia.org/wiki/File:Devops.png)
How Configuration Tools Management Help
• Can orchestrate deployment….and re-deployment.
• Most can idempotently check configuration (no-op if everything is ok)
• Can touch many nodes in parallel
• Can type much faster and more accurately than you
• Are a great way to collaborate amongst teams of operators
• Most have strong communities within the OpenStack universe
– Using a commercial OpenStack? Most vendors are using one of these tools
to deploy and manage your cloud, whether you know it or not.
– Rolling your own? Check out StackForge for tons of Ansible/Puppet/Chef
modules you can use today
• Allow you to manage other things besides OpenStack itself
CONFIDENTIAL
51
OpenStack: Toward a More Resilient Cloud

Recommended for you

Operating OpenStack on a Budget
Operating OpenStack on a BudgetOperating OpenStack on a Budget
Operating OpenStack on a Budget

There is a growing trend today of enterprises leveraging both Amazon Web Services (AWS) and on-premise OpenStack-based private clouds. However, the default networking option in OpenStack remains broken and the plethora of confusing plug-ins makes networking in OpenStack mysterious and difficult to manage. Enter MidoNet, the open source network virtualization solution from Midokura favored by DevOps cultures in web scale enterprises and service providers around the world. This session will present case studies from several end user deployments, showing how they use MidoNet to build, run and manage large-scale virtual networks in OpenStack clouds. The session will also discuss how transitioning from a public to private cloud enables organizations to accomplish much more with the same resources, without over-simplifying the inherent complexity of running an OpenStack cloud.

openstack cloud cdn operations midonet
Open Clouds: The New Primitives in Enterprise IT & Mobile Networks
Open Clouds: The New Primitives in Enterprise IT & Mobile NetworksOpen Clouds: The New Primitives in Enterprise IT & Mobile Networks
Open Clouds: The New Primitives in Enterprise IT & Mobile Networks

This talk was presented in the "101" track at All Things Open 2019. In this talk, we’ll explore open source projects like OpenStack and Kubernetes that form the basis for many cloud use cases today. This talk will serve as a general introduction to the new “primitives” of IT infrastructure, such as server, VM’s, containers, functions, API’s, etc. By examining the primitives that popular modern tools provide, it's substantially easier to make sense of the overall technology landscape and discern how to make the best technology choices for your use cases. We’ll also take a brief look at open source tools used to interact with open infrastructure, and examine some more concrete use cases describing how enterprise IT and mobile networks are using open source to build their next generation infrastructure.

openstackkubernetescontainers
Open Source in the Era of 5G - All Things Open 2018
Open Source in the Era of 5G - All Things Open 2018Open Source in the Era of 5G - All Things Open 2018
Open Source in the Era of 5G - All Things Open 2018

1) 5G networks will enable much faster speeds, lower latency, and more connections compared to previous generations of wireless technology. 2) For 5G to be fully realized, network infrastructure will need to become more distributed through small cell sites and edge computing deployments. Open source software and network function virtualization will be critical to manage this complex infrastructure. 3) Initiatives like ONAP are developing open source platforms to automate the deployment and management of virtualized network functions across distributed infrastructure as defined by standards bodies like the O-RAN Alliance.

openstackopen sourceonap
OpenStack: Toward a More Resilient Cloud
• I’ve worked on a lot of OpenStack clouds and almost everyone has
their own preferred monitoring toolset.
• One possible exception: almost everybody seems to love Graphite.
• The golden rule is: use the tools that work for you!
• Very often this will be whatever you’re using in the rest of your infrastructure.
• Break it down into at least two buckets:
• Up/down and alerting (ex: Nagios or it’s derivatives…yes, there are
OpenStack plugins out there on NagiosExchange)
• Trending data collection/plotting (ex: collectd/statsd feeding graphite)
• Don’t forget logging
• LogInsight, Logstash, etc.
• Also: use your peers!
• Operators often willing to share, so ask on the openstack-operators list.
• Ok, this could take another hour, so I’ll just hit a few highlights…
• Make use of OpenStack’s segregation features
– Availability zones, host aggregates, regions, server groups for compute
– Regions and zones for Swift
• Plan to make infrastructure maintenance less impacting
– Put API servers behind load balancers
– Virtualize tenant-facing parts of the control plane for greater scale and mobilty
– Use host evacuation and live migration to reduce impact
– OpenStack is extremely pluggable…choose your backends wisely
• You should know how to operate, monitor, and troubleshoot them
• Understand how their drivers interact with OpenStack
• You should be comfortable with their failure and recovery modes
Questions?
@marktvoelker
http://openstack.org/
http://www.vmware.com/products/openstack/

Recommended for you

OpenStack & the Evolving Cloud Ecosystem
OpenStack & the Evolving Cloud EcosystemOpenStack & the Evolving Cloud Ecosystem
OpenStack & the Evolving Cloud Ecosystem

OpenStack has come a long way since 2010. What started as a collaboration on compute and storage between NASA and Rackspace has changed dramatically and grown into a large, successful open source project that meets the needs of thousands of organizations. But OpenStack hasn’t evolved in a vacuum over the past seven years: the technology landscape around it has been changing as well. Join VMware’s chief OpenStack architect and longtime community member Mark Voelker for a look at the new technology landscape around OpenStack, how we got here, and where we might go next. We’ll discuss how what started as an IaaS platform ending up being a winning platform for Network Functions Virtualization and telco applications, how OpenStack came to be selected as a common underpinning for container orchestration systems like Kubernetes, how OpenStack governance influenced other open source communities, and how OpenStack changed the way companies looked at Open Source. We’ll consider the role IaaS might play in a future that includes options like functions-as-a-service, containers, and the internet of things. We’ll consider OpenStack as a common foundation for a variety of new technologies, and discuss OpenStack’s lasting impact in the cloud ecosystem. We’ll also discuss how OpenStack is changing and adapting to shifts in the technology landscape, both as an open source community and in terms of product offerings. Learn about new interoperability programs targeted at use cases that didn’t exist seven years ago, and new initiatives from the OpenStack technical community and Foundation.

openstackcloudvmware
Interoperable Clouds and How to Build (or Buy) Them
Interoperable Clouds and How to Build (or Buy) ThemInteroperable Clouds and How to Build (or Buy) Them
Interoperable Clouds and How to Build (or Buy) Them

This document summarizes a presentation about interoperability between OpenStack and other open source infrastructure projects like Kubernetes. It discusses how the OpenStack Interoperability Working Group defines guidelines for compatibility between different OpenStack distributions. While no distribution will be identical, following these guidelines ensures core capabilities are supported across approved implementations. It encourages selecting technologies based on common interfaces like APIs, Terraform, or Kubernetes rather than any single vendor implementation to avoid lock-in and allow flexibility.

openstackkubernetesopnfv
InteropWG Intro & Vertical Programs (May. 2017)
InteropWG Intro & Vertical Programs (May. 2017)InteropWG Intro & Vertical Programs (May. 2017)
InteropWG Intro & Vertical Programs (May. 2017)

The document summarizes the OpenStack Interoperability Working Group's efforts to promote interoperability across OpenStack distributions and products. It discusses how the group develops guidelines specifying required capabilities and tests. Products must pass these tests to be considered interoperable and qualify for the OpenStack logo program. The guidelines aim to ensure a consistent user experience while allowing flexibility in implementations. The document also outlines the group's governance process and opportunities for participants to provide feedback to help improve interoperability standards over time.

nfvopenstackinteroperability
Thank you!

More Related Content

What's hot

(SCALE 12x) OpenStack vs. VMware - A System Administrator Perspective
(SCALE 12x) OpenStack vs. VMware - A System Administrator Perspective(SCALE 12x) OpenStack vs. VMware - A System Administrator Perspective
(SCALE 12x) OpenStack vs. VMware - A System Administrator Perspective
StackStorm
 
CERN Data Centre Evolution
CERN Data Centre EvolutionCERN Data Centre Evolution
CERN Data Centre Evolution
Gavin McCance
 
The Future of SDN in CloudStack by Chiradeep Vittal
The Future of SDN in CloudStack by Chiradeep VittalThe Future of SDN in CloudStack by Chiradeep Vittal
The Future of SDN in CloudStack by Chiradeep Vittal
buildacloud
 
What's New in Grizzly & Deploying OpenStack with Puppet
What's New in Grizzly & Deploying OpenStack with PuppetWhat's New in Grizzly & Deploying OpenStack with Puppet
What's New in Grizzly & Deploying OpenStack with Puppet
Mark Voelker
 
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...
Edureka!
 
Build public private cloud using openstack
Build public private cloud using openstackBuild public private cloud using openstack
Build public private cloud using openstack
Framgia Vietnam
 
2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)
2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)
2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)
Mirantis
 
OpenStack Explained: Learn OpenStack architecture and the secret of a success...
OpenStack Explained: Learn OpenStack architecture and the secret of a success...OpenStack Explained: Learn OpenStack architecture and the secret of a success...
OpenStack Explained: Learn OpenStack architecture and the secret of a success...
Giuseppe Paterno'
 
OpenStack + VMware: Everything You Need to Know (Kilo-edition)
OpenStack + VMware: Everything You Need to Know (Kilo-edition)OpenStack + VMware: Everything You Need to Know (Kilo-edition)
OpenStack + VMware: Everything You Need to Know (Kilo-edition)
Dan Wendlandt
 
OpenStack Scale-out Networking Architecture
OpenStack Scale-out Networking ArchitectureOpenStack Scale-out Networking Architecture
OpenStack Scale-out Networking Architecture
Randy Bias
 
Introduction to Apache CloudStack by David Nalley
Introduction to Apache CloudStack by David NalleyIntroduction to Apache CloudStack by David Nalley
Introduction to Apache CloudStack by David Nalley
buildacloud
 
An Introduction to OpenStack
An Introduction to OpenStackAn Introduction to OpenStack
An Introduction to OpenStack
Scott Lowe
 
OpenStack Framework Introduction
OpenStack Framework IntroductionOpenStack Framework Introduction
OpenStack Framework Introduction
Jason TC HOU (侯宗成)
 
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander DibboOpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
OpenNebula Project
 
Successfully Deliver and Operate OpenStack in Production with VMware VIO
Successfully Deliver and Operate OpenStack in Production with VMware VIOSuccessfully Deliver and Operate OpenStack in Production with VMware VIO
Successfully Deliver and Operate OpenStack in Production with VMware VIO
Arraya Solutions
 
VMUG22 Filip Verloy VIO
VMUG22 Filip Verloy VIOVMUG22 Filip Verloy VIO
VMUG22 Filip Verloy VIO
Filip Verloy
 
Bridging The Gap: Explaining OpenStack To VMware Administrators
Bridging The Gap: Explaining OpenStack To VMware AdministratorsBridging The Gap: Explaining OpenStack To VMware Administrators
Bridging The Gap: Explaining OpenStack To VMware Administrators
Kenneth Hui
 
Introduction to openstack
Introduction to openstackIntroduction to openstack
Introduction to openstack
Yaniv Zadka
 
Cloud Computing Open Stack Compute Node
Cloud Computing Open Stack Compute NodeCloud Computing Open Stack Compute Node
Cloud Computing Open Stack Compute Node
Palak Sood
 
Oct meetup open stack 101 clean
Oct meetup open stack 101   cleanOct meetup open stack 101   clean
Oct meetup open stack 101 clean
benrodrigue
 

What's hot (20)

(SCALE 12x) OpenStack vs. VMware - A System Administrator Perspective
(SCALE 12x) OpenStack vs. VMware - A System Administrator Perspective(SCALE 12x) OpenStack vs. VMware - A System Administrator Perspective
(SCALE 12x) OpenStack vs. VMware - A System Administrator Perspective
 
CERN Data Centre Evolution
CERN Data Centre EvolutionCERN Data Centre Evolution
CERN Data Centre Evolution
 
The Future of SDN in CloudStack by Chiradeep Vittal
The Future of SDN in CloudStack by Chiradeep VittalThe Future of SDN in CloudStack by Chiradeep Vittal
The Future of SDN in CloudStack by Chiradeep Vittal
 
What's New in Grizzly & Deploying OpenStack with Puppet
What's New in Grizzly & Deploying OpenStack with PuppetWhat's New in Grizzly & Deploying OpenStack with Puppet
What's New in Grizzly & Deploying OpenStack with Puppet
 
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...
OpenStack Cloud Tutorial | What is OpenStack | OpenStack Tutorial | OpenStack...
 
Build public private cloud using openstack
Build public private cloud using openstackBuild public private cloud using openstack
Build public private cloud using openstack
 
2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)
2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)
2 Day Bootcamp for OpenStack--Cloud Training by Mirantis (Preview)
 
OpenStack Explained: Learn OpenStack architecture and the secret of a success...
OpenStack Explained: Learn OpenStack architecture and the secret of a success...OpenStack Explained: Learn OpenStack architecture and the secret of a success...
OpenStack Explained: Learn OpenStack architecture and the secret of a success...
 
OpenStack + VMware: Everything You Need to Know (Kilo-edition)
OpenStack + VMware: Everything You Need to Know (Kilo-edition)OpenStack + VMware: Everything You Need to Know (Kilo-edition)
OpenStack + VMware: Everything You Need to Know (Kilo-edition)
 
OpenStack Scale-out Networking Architecture
OpenStack Scale-out Networking ArchitectureOpenStack Scale-out Networking Architecture
OpenStack Scale-out Networking Architecture
 
Introduction to Apache CloudStack by David Nalley
Introduction to Apache CloudStack by David NalleyIntroduction to Apache CloudStack by David Nalley
Introduction to Apache CloudStack by David Nalley
 
An Introduction to OpenStack
An Introduction to OpenStackAn Introduction to OpenStack
An Introduction to OpenStack
 
OpenStack Framework Introduction
OpenStack Framework IntroductionOpenStack Framework Introduction
OpenStack Framework Introduction
 
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander DibboOpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
 
Successfully Deliver and Operate OpenStack in Production with VMware VIO
Successfully Deliver and Operate OpenStack in Production with VMware VIOSuccessfully Deliver and Operate OpenStack in Production with VMware VIO
Successfully Deliver and Operate OpenStack in Production with VMware VIO
 
VMUG22 Filip Verloy VIO
VMUG22 Filip Verloy VIOVMUG22 Filip Verloy VIO
VMUG22 Filip Verloy VIO
 
Bridging The Gap: Explaining OpenStack To VMware Administrators
Bridging The Gap: Explaining OpenStack To VMware AdministratorsBridging The Gap: Explaining OpenStack To VMware Administrators
Bridging The Gap: Explaining OpenStack To VMware Administrators
 
Introduction to openstack
Introduction to openstackIntroduction to openstack
Introduction to openstack
 
Cloud Computing Open Stack Compute Node
Cloud Computing Open Stack Compute NodeCloud Computing Open Stack Compute Node
Cloud Computing Open Stack Compute Node
 
Oct meetup open stack 101 clean
Oct meetup open stack 101   cleanOct meetup open stack 101   clean
Oct meetup open stack 101 clean
 

Similar to OpenStack: Toward a More Resilient Cloud

Cloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: OpenstackCloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: Openstack
Microsoft
 
OpenStack 101
OpenStack 101OpenStack 101
OpenStack 101
All Things Open
 
Getting Started with Apache CloudStack
Getting Started with Apache CloudStackGetting Started with Apache CloudStack
Getting Started with Apache CloudStack
Joe Brockmeier
 
Hacking apache cloud stack
Hacking apache cloud stackHacking apache cloud stack
Hacking apache cloud stack
Nitin Mehta
 
Private cloud cloud-phoenix-april-2014
Private cloud cloud-phoenix-april-2014Private cloud cloud-phoenix-april-2014
Private cloud cloud-phoenix-april-2014
Miguel Zuniga
 
Txlf2012
Txlf2012Txlf2012
Txlf2012
Joe Brockmeier
 
CloudStack - LinuxFest NorthWest
CloudStack - LinuxFest NorthWestCloudStack - LinuxFest NorthWest
CloudStack - LinuxFest NorthWest
ke4qqq
 
State of the Container Ecosystem
State of the Container EcosystemState of the Container Ecosystem
State of the Container Ecosystem
Vinay Rao
 
PowerPoint Presentation
PowerPoint PresentationPowerPoint Presentation
PowerPoint Presentation
lalitjangra9
 
Openstack presentation
Openstack presentationOpenstack presentation
Openstack presentation
Sankalp Jain
 
Taking the open cloud to 11
Taking the open cloud to 11Taking the open cloud to 11
Taking the open cloud to 11
Joe Brockmeier
 
OpenStack Deployment in the Enterprise
OpenStack Deployment in the Enterprise OpenStack Deployment in the Enterprise
OpenStack Deployment in the Enterprise
Cisco Canada
 
Directions for CloudStack Networking
Directions for CloudStack  NetworkingDirections for CloudStack  Networking
Directions for CloudStack Networking
Chiradeep Vittal
 
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxBetter, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
John Burwell
 
Cloud Native Camel Riding
Cloud Native Camel RidingCloud Native Camel Riding
Cloud Native Camel Riding
Christian Posta
 
Stackato v2
Stackato v2Stackato v2
Stackato v2
Jonas Brømsø
 
Chicago Microservices Integration Talk
Chicago Microservices Integration TalkChicago Microservices Integration Talk
Chicago Microservices Integration Talk
Christian Posta
 
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OS
Steve Wong
 
Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWSMigrating enterprise workloads to AWS
Migrating enterprise workloads to AWS
Tom Laszewski
 
Operating OpenStack on a Budget
Operating OpenStack on a BudgetOperating OpenStack on a Budget
Operating OpenStack on a Budget
Samir Ibradzic
 

Similar to OpenStack: Toward a More Resilient Cloud (20)

Cloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: OpenstackCloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: Openstack
 
OpenStack 101
OpenStack 101OpenStack 101
OpenStack 101
 
Getting Started with Apache CloudStack
Getting Started with Apache CloudStackGetting Started with Apache CloudStack
Getting Started with Apache CloudStack
 
Hacking apache cloud stack
Hacking apache cloud stackHacking apache cloud stack
Hacking apache cloud stack
 
Private cloud cloud-phoenix-april-2014
Private cloud cloud-phoenix-april-2014Private cloud cloud-phoenix-april-2014
Private cloud cloud-phoenix-april-2014
 
Txlf2012
Txlf2012Txlf2012
Txlf2012
 
CloudStack - LinuxFest NorthWest
CloudStack - LinuxFest NorthWestCloudStack - LinuxFest NorthWest
CloudStack - LinuxFest NorthWest
 
State of the Container Ecosystem
State of the Container EcosystemState of the Container Ecosystem
State of the Container Ecosystem
 
PowerPoint Presentation
PowerPoint PresentationPowerPoint Presentation
PowerPoint Presentation
 
Openstack presentation
Openstack presentationOpenstack presentation
Openstack presentation
 
Taking the open cloud to 11
Taking the open cloud to 11Taking the open cloud to 11
Taking the open cloud to 11
 
OpenStack Deployment in the Enterprise
OpenStack Deployment in the Enterprise OpenStack Deployment in the Enterprise
OpenStack Deployment in the Enterprise
 
Directions for CloudStack Networking
Directions for CloudStack  NetworkingDirections for CloudStack  Networking
Directions for CloudStack Networking
 
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxBetter, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
 
Cloud Native Camel Riding
Cloud Native Camel RidingCloud Native Camel Riding
Cloud Native Camel Riding
 
Stackato v2
Stackato v2Stackato v2
Stackato v2
 
Chicago Microservices Integration Talk
Chicago Microservices Integration TalkChicago Microservices Integration Talk
Chicago Microservices Integration Talk
 
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OS
 
Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWSMigrating enterprise workloads to AWS
Migrating enterprise workloads to AWS
 
Operating OpenStack on a Budget
Operating OpenStack on a BudgetOperating OpenStack on a Budget
Operating OpenStack on a Budget
 

More from Mark Voelker

Open Clouds: The New Primitives in Enterprise IT & Mobile Networks
Open Clouds: The New Primitives in Enterprise IT & Mobile NetworksOpen Clouds: The New Primitives in Enterprise IT & Mobile Networks
Open Clouds: The New Primitives in Enterprise IT & Mobile Networks
Mark Voelker
 
Open Source in the Era of 5G - All Things Open 2018
Open Source in the Era of 5G - All Things Open 2018Open Source in the Era of 5G - All Things Open 2018
Open Source in the Era of 5G - All Things Open 2018
Mark Voelker
 
OpenStack & the Evolving Cloud Ecosystem
OpenStack & the Evolving Cloud EcosystemOpenStack & the Evolving Cloud Ecosystem
OpenStack & the Evolving Cloud Ecosystem
Mark Voelker
 
Interoperable Clouds and How to Build (or Buy) Them
Interoperable Clouds and How to Build (or Buy) ThemInteroperable Clouds and How to Build (or Buy) Them
Interoperable Clouds and How to Build (or Buy) Them
Mark Voelker
 
InteropWG Intro & Vertical Programs (May. 2017)
InteropWG Intro & Vertical Programs (May. 2017)InteropWG Intro & Vertical Programs (May. 2017)
InteropWG Intro & Vertical Programs (May. 2017)
Mark Voelker
 
Considerations for Operating An OpenStack Cloud
Considerations for Operating An OpenStack CloudConsiderations for Operating An OpenStack Cloud
Considerations for Operating An OpenStack Cloud
Mark Voelker
 

More from Mark Voelker (6)

Open Clouds: The New Primitives in Enterprise IT & Mobile Networks
Open Clouds: The New Primitives in Enterprise IT & Mobile NetworksOpen Clouds: The New Primitives in Enterprise IT & Mobile Networks
Open Clouds: The New Primitives in Enterprise IT & Mobile Networks
 
Open Source in the Era of 5G - All Things Open 2018
Open Source in the Era of 5G - All Things Open 2018Open Source in the Era of 5G - All Things Open 2018
Open Source in the Era of 5G - All Things Open 2018
 
OpenStack & the Evolving Cloud Ecosystem
OpenStack & the Evolving Cloud EcosystemOpenStack & the Evolving Cloud Ecosystem
OpenStack & the Evolving Cloud Ecosystem
 
Interoperable Clouds and How to Build (or Buy) Them
Interoperable Clouds and How to Build (or Buy) ThemInteroperable Clouds and How to Build (or Buy) Them
Interoperable Clouds and How to Build (or Buy) Them
 
InteropWG Intro & Vertical Programs (May. 2017)
InteropWG Intro & Vertical Programs (May. 2017)InteropWG Intro & Vertical Programs (May. 2017)
InteropWG Intro & Vertical Programs (May. 2017)
 
Considerations for Operating An OpenStack Cloud
Considerations for Operating An OpenStack CloudConsiderations for Operating An OpenStack Cloud
Considerations for Operating An OpenStack Cloud
 

Recently uploaded

How RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptxHow RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptx
SynapseIndia
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
Emerging Tech
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
Mark Billinghurst
 
Cookies program to display the information though cookie creation
Cookies program to display the information though cookie creationCookies program to display the information though cookie creation
Cookies program to display the information though cookie creation
shanthidl1
 
The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
Larry Smarr
 
20240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 202420240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 2024
Matthew Sinclair
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Chris Swan
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Bert Blevins
 
7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf
Enterprise Wired
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
What’s New in Teams Calling, Meetings and Devices May 2024
What’s New in Teams Calling, Meetings and Devices May 2024What’s New in Teams Calling, Meetings and Devices May 2024
What’s New in Teams Calling, Meetings and Devices May 2024
Stephanie Beckett
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024
BookNet Canada
 
20240702 Présentation Plateforme GenAI.pdf
20240702 Présentation Plateforme GenAI.pdf20240702 Présentation Plateforme GenAI.pdf
20240702 Présentation Plateforme GenAI.pdf
Sally Laouacheria
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides
Safe Software
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
ishalveerrandhawa1
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 

Recently uploaded (20)

How RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptxHow RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptx
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
 
Cookies program to display the information though cookie creation
Cookies program to display the information though cookie creationCookies program to display the information though cookie creation
Cookies program to display the information though cookie creation
 
The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
 
20240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 202420240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 2024
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
Understanding Insider Security Threats: Types, Examples, Effects, and Mitigat...
 
7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
What’s New in Teams Calling, Meetings and Devices May 2024
What’s New in Teams Calling, Meetings and Devices May 2024What’s New in Teams Calling, Meetings and Devices May 2024
What’s New in Teams Calling, Meetings and Devices May 2024
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024
 
20240702 Présentation Plateforme GenAI.pdf
20240702 Présentation Plateforme GenAI.pdf20240702 Présentation Plateforme GenAI.pdf
20240702 Présentation Plateforme GenAI.pdf
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 

OpenStack: Toward a More Resilient Cloud

  • 1. © 2014 VMware Inc. All rights reserved. OpenStack: Toward a More Resilient Cloud Or: “How I Learned to Stop Worrying & Embrace My Inner Cloudiness” Mark T. Voelker OpenStack Architect Percona University Smart Data Event February 12, 2015
  • 2. • Who is this guy? • A little background on OpenStack • Building more resilient clouds – Withstanding failures – Quickly recovering from failures • Questions?
  • 3. @marktvoelker • OpenStack Architect @ VMware, OpenStack ATC, Ex-StackForge Puppet core dev, Triangle OpenStack Meetup founder, OS Foundation Member #54 • Fact: can be bribed with doughnuts • Currently works in VMware’s Software Defined Datacenter R&D group • In copious (hah!) spare time: OpenStack solutions, Big Data, Massively Scalable Data Centers, Devops, making sawdust with extreme prejudice
  • 4. • Tech lead, manager, software developer, architect • Started in OpenStack in 2011 at the Diablo Design Summit
  • 5. ….I’ve built a few clouds.
  • 6. Today’s talk won’t be overly formal….
  • 7. …because I tend to get excited by this stuff.
  • 8. • Who is this guy? • A little background on OpenStack • Building more resilient clouds – Withstanding failures – Quickly recovering from failures • Questions?
  • 9. “OpenStack is a global collaboration of developers and cloud computing technologists producing the ubiquitous open source cloud computing platform for public and private clouds. The project aims to deliver solutions for all types of clouds by being simple to implement, massively scalable, and feature rich. The technology consists of a series of interrelated projects delivering various components for a cloud infrastructure solution.” -- openstack.org Basically, it’s software to run cloud services—including compute, network, storage, and security—and the community behind that software.
  • 13. • IRC Channels and Mailing Lists • User/Meetup Groups • Social Networking – Twitter – LinkedIn – Facebook – Ohloh • Code in cgit, mirrored on GitHub, Bugs/Milestones in Launchpad • For now…may move to StoryBoard in future • Over 20 million lines of code by over 1,419 contributors • Two Annual Design Summit/Conferences (coinciding roughly w/releases) • Want to contribute? Start here.
  • 14. • Don’t be intimidated. • HolycrapthingsmovereallyreallyfastinOpenStack • Jump in feet first: be agile and flexible. • This is going to feel a little different for some of you.
  • 16. Horizon Nova Neutron Swift (Object Storage) Cinder (Block storage) Glance (VM Image Service) Keystone (Identity Service) AWS Management Console EC2 VPC S3 EBS Ceilometer (Telemetry Service) Trove (Database Service) Heat (Orchestration Service)Sahara (Data Processing Service)
  • 17. Library Projects Supporting Projects Documentation Oslo (common code libraries) Client libraries Incubated Projects (may become core components in the future) Designate (DNS service) Zaqar (queuing service) Gating Projects CI & Infrastructure DevStack (deployment script) Tempest (integration test) Barbican (key management) Manila (shared FS as a service)
  • 19. • Who is this guy? • A little background on OpenStack • Building more resilient clouds – Withstanding failures – Quickly recovering from failures • Questions?
  • 20. What’s a “resilient” cloud? re·sil·ient /rəˈzilyənt/ (adjective) Able to withstand or recover quickly from difficult conditions.
  • 21. • Today we’ll primarily focus on the cloud itself • Workloads running *in* clouds are another story…but we’ve only got one hour!
  • 22. 8am: “Uh-oh. Something tells me it’s going to be an interesting day in the datacenter….”
  • 23. • Hardware Failures • OpenStack software bugs (yep, those exist) • Underpinning software failures (database, message queue, etc) • Operating system failures • Network/storage/power failures • Planned maintenance windows • Hackers and malcontents • Upgrades • Automation failures • “Whoops, did I do that?”
  • 24. Some causes of outages in the past year ….did you plan for these? CONFIDENTIAL 24
  • 25. Sometimes things break (in *any* system). 25 Withstand what you can. Quickly recover from the rest. Because you don’t look this cute when your cloud is down.
  • 28. General Premise: Assume hardware and software fail. (because, shockingly, that actually happens) CONFIDENTIAL 28
  • 29. What Does “HA” Mean in an OpenStack Cloud? CONFIDENTIAL 29 • Compute • Multiple clusters • Consider segmenting with Availability Zones, Host Aggregates, etc • Consider ability to live migrate instances for hypervisor node maintenance • Ensure some capacity buffer for maintenance operations • Storage • Avoid single points of failure • Multiple technologies can be used…but each has it’s own limitations • Don’t think just Cinder here…your Glance backend and compute storage matter too! • Network • Network disruptions will inevitably occur, so plan for them • Design for control plane disruption (and pick technology accordingly) • Control Plane • May depend on the other things above • Essential to keeping the cloud operational • Data Plane • Stuff that workloads running in the cloud depends on
  • 30. High Availability Is Part of the Story…. ….we need to think a bit about architecture. (I’ll use a reference architecture from VMware Integrated OpenStack as an example) CONFIDENTIAL 30
  • 31. VIO Architecture – Logical Topology- Management
  • 32. Notice something? There’s a lot of stuff in there that isn’t OpenStack, but upon which OpenStack depends. CONFIDENTIAL 32
  • 33. VIO Architecture – RabbitMQ • RabbitMQ is a messaging broker - an intermediary for messaging. It gives applications a common platform to send and receive messages, and the messages a safe place to live until received. • RabbitMQ is the default AMQP server used by OpenStack services (Qpid is also an option, some support for 0mq). In production clouds, this should be a highly available infrastructure component. • The OpenStack subcomponents (nova- scheduler to nova-compute, for example) communicate among themselves using this hosted message queue service. They also utilize the hosted Memcached services for caching authentication tokens etc. As always, they persist data to the Database. • Component-to-Component communications (Nova-> Neutron) is done via REST. • For more details about the HA implementation of RabbitMQ, please click here.
  • 34. VIO Architecture – Database • The database cluster is at the heart of the infrastructure. Typically MySQL or MariaDB are used, but other options such as PostgreSQL are also supported. • The VIO MariaDB implementation makes use of a 3-node Galera cluster, which in itself is Active-Active-Active. However, since some OpenStack services enforce table locking, reads and writes are directed to a single node via the Load Balancers. • Note that this database is for management plane data. OpenStack services that store data as part of their purpose may use additional DB’s. For example: Ceilometer may store meter data to MySQL, Mongo, PostgreSQL, HBase, or DB2.
  • 35. VIO Architecture – Load Balancers • Most OpenStack Services run on the Controllers, which are mirrored on each controller VM and load-balanced. They are accessible via the internal virtual IP. • Some of the services, such as the Dashboard, compute-api, glance-api, keystone, cinder, neutron and novncproxy are exposed to the end users via the load balancer’s public virtual IP. • Likewise, the hosted Message Queue (RabbitMQ) and Memcached services are also load-balanced between 2 VMs. • For the Database Service, the load balancer is configured to use a primary DB VM. In case of failure it will switch to one of the two backup DB VMs. • Load Balancers user here are HAProxy with Keepalived for high availability
  • 36. Etc, etc, etc CONFIDENTIAL 36 • There’s a network connecting all that stuff • It’s running (as virtual machines) on servers which have operating systems • Things may get wonky if NTP fails and clocks are out of sync • If DNS can’t resolve, Bad Things ™ will probably happen
  • 37. • Consider whether you want active/active or active/passive • Setup and tooling differs a bit, but I generally like active/active • Note that docs.openstack.org has an HA Guide • Currently undergoing lots of updates…patches welcome! • Prioritize HA for the control plane • That also means thinking about your database, network, and RPC bus • Note: HA == more hardware • Some components need at least 3 nodes • Mitigate by virtualizing control plane
  • 38. • Stuff OpenStack needs to run: message brokers • Check out RabbitMQ clustering and mirrored queues • Check out Galera for MySQL/MariaDB • I often see Percona XtraDB in the wild • Frontend with an HAProxy/Keepalived pair • Memcached for caching
  • 39. • Don’t do rabbit clustering over a WAN • Be aware of the SELECT… FOR UPDATE issue
  • 40. • Long story short: Neutron and some parts of Nova invoke an SQL pattern known as “SELECT…FOR UPDATE” which Galera doesn’t support due to issues with cross-node locking. • Can cause deadlock-like symptoms due to locks not being replicated. • Neutron/nova code being refactored, but will likely not be done soon. • Meanwhile: use HAProxy to send writes to a single Galera node and you should be fine • With the obvious scalability bottleneck • More info here, here, & here. • Thank Jay Pipes of Mirantis & Peter Boros of Percona for the find!
  • 41. • Pick a highly available storage to back Glance • Pick a highly available storage backend for Cinder too – SAN, distributed, software defined, plethora of options here • Use Keepalived/HAProxy to front-end multiple API servers • Or another load balancer technology of your choice • Can be deployed as dedicated nodes for scale, or cohabitate • Data plane network: DVR & Provider Network Extensions • Distributed Virtual Routers are a new experimental feature in Juno (not yet ready for production) • Please go test it and report/fix bugs! • Provider networks essentially punt the availability issue to your physical network • Allows you to use standard tools like virtual port channels and VRRP • Also highly performant
  • 42. • Network: consider your backend technology – Neutron offers a variety of plugins for various open source and vendor-supplied network technology – Physical networks need usual redundancy protections – Overlays are popular for segmentation/isolation; some scale better than others – Shameless plug: check out VMware NSX which has been used in some very large OpenStack deployments!
  • 43. • Actually, most of these techniques and technologies are things that seasoned developers and sysadmins have used before. • It doesn’t take a genius to learn lessons from the past and apply them, tweak them, and tune them (but it’s a fair amount of work).
  • 44. Simple Rules of Thumb Planning for availability can go to extreme levels, so start simple: – Can I take any one [server | switch | storage unit] out of service in my control plane and still be operational? – For all of the above, what’s the impact? • Performance hit? • Capacity loss? • World is broken?!? – For all the above, how easy is it to reintroduce a repaired/replaced $thing? • Is there a recovery period that will further impact performance? • Is it a complex procedure? • Does the procedure cause more $things to be temporarily unavailable? – For all of the above, how can I monitor & alert for failure? Once you have that down, dig deeper to your heart’s content. CONFIDENTIAL 44
  • 46. Rule 1: Assume You Will Need to Change Stuff • Change can be a lot of things: – Hardware or software upgrades/patches/replacements – Configuration tweaks – Adding or subtracting capacity – All systems change over time; OpenStack clouds are no exception. CONFIDENTIAL 46 “Change in OpenStack? Yeah, I’m good with that…”
  • 47. Rule 2: Assume You Can’t Manually Log In To All The Nodes To Make Those Changes • OpenStack is a series of cooperating distributed systems – That means you could (potentially) have a lot of nodes – Software & config must often be placed on many machines – Manual changes == slow changes != quick recovery CONFIDENTIAL 47 “I guess multitasking only speeds things up to a certain extent…”
  • 48. Rule 3: Assume You Will Need To Test Stuff • It’s a good idea to have a small test cloud where you can examine the impact of changes • When possible, roll out changes to a portion of your cloud and evaluate before rolling out the rest – Note: this means you need tests and monitoring…otherwise you don’t know what “ok” looks like CONFIDENTIAL 48 “It’s 3am and I’m still debugging in production…maybe I should have taken the time to set up a test environment and automate some testing after all…”
  • 50. • Software developers and operators are increasingly the same people. – Agile development – Automate (almost) everything – Treat config & changes as you would code – Continuous integration, testing, deployment – Incremental change & iteration – Unified tooling & versioning – Critical approach to working at scale – Really useful for building resilient clouds Image courtesy of Rajiv Pant (http://en.wikipedia.org/wiki/File:Devops.png)
  • 51. How Configuration Tools Management Help • Can orchestrate deployment….and re-deployment. • Most can idempotently check configuration (no-op if everything is ok) • Can touch many nodes in parallel • Can type much faster and more accurately than you • Are a great way to collaborate amongst teams of operators • Most have strong communities within the OpenStack universe – Using a commercial OpenStack? Most vendors are using one of these tools to deploy and manage your cloud, whether you know it or not. – Rolling your own? Check out StackForge for tons of Ansible/Puppet/Chef modules you can use today • Allow you to manage other things besides OpenStack itself CONFIDENTIAL 51
  • 54. • I’ve worked on a lot of OpenStack clouds and almost everyone has their own preferred monitoring toolset. • One possible exception: almost everybody seems to love Graphite. • The golden rule is: use the tools that work for you! • Very often this will be whatever you’re using in the rest of your infrastructure. • Break it down into at least two buckets: • Up/down and alerting (ex: Nagios or it’s derivatives…yes, there are OpenStack plugins out there on NagiosExchange) • Trending data collection/plotting (ex: collectd/statsd feeding graphite) • Don’t forget logging • LogInsight, Logstash, etc. • Also: use your peers! • Operators often willing to share, so ask on the openstack-operators list.
  • 55. • Ok, this could take another hour, so I’ll just hit a few highlights… • Make use of OpenStack’s segregation features – Availability zones, host aggregates, regions, server groups for compute – Regions and zones for Swift • Plan to make infrastructure maintenance less impacting – Put API servers behind load balancers – Virtualize tenant-facing parts of the control plane for greater scale and mobilty – Use host evacuation and live migration to reduce impact – OpenStack is extremely pluggable…choose your backends wisely • You should know how to operate, monitor, and troubleshoot them • Understand how their drivers interact with OpenStack • You should be comfortable with their failure and recovery modes

Editor's Notes

  1. 1
  2. Messaging enables software applications to connect and scale. Applications can connect to each other, as components of a larger application. Messaging is asynchronous, decoupling applications by separating sending and receiving data. Advanced Message Queuing Protocol (AMQP) is an open standard and it enables conforming client applications to communicate with conforming messaging middleware servers
  3. Image courtesy of Rajiv Pant (http://en.wikipedia.org/wiki/File:Devops.png)