OpenStack Mitaka Design Summit

Keystone supports four different types of tokens, UUID, PKI, PKIZ, and Fernet. Let’s take a deep dive into: 

• Understanding token formats


• Pros and Cons of each format in Production


• Performance across multiple data centers


• Token revocation workflow for each of the formats


• Horizon usage of the different token types

We previously deployed UUID and PKI in Production and are now moving towards the latest format, Fernet. We would like to share our lessons learned with different formats and help you decide on which format is suitable for your cloud.

OpenStack is a collection of services that use messaging to communicate. Understanding the messaging layer is important for a successful OpenStack deployment.

In this talk, we will cover best practices in running RabbitMQ, things you should avoid doing, lesser known features, recent operations improvements and a bit of what's ahead.

Both of our speakers are devops who have been contributors to two key eNovance OpenStack platforms called "eNoCloud", since Essex and are located in Paris and Montreal Data-Centers. Along with eNoCloud, both have been driving various customer platforms. While working on these production platforms, they have faced a variety of challenges ranging from design, to build out, contribution, upgrading, recovering and of course dealing with users!

OpenStack has evolved a lot since the Essex release but the life of a devops remains a daily roller-coaster, a lot of sweat & tears with a lot of work to automate and sometimes get a little peace of mind! In order to increase 'peace in the mind' within the ops community, we would like to share our knowledge and open up a discussion with you.

In this talk, we will share our experience about being OpenStack-dedicated devops. Both RedHat and Debian setups will be presented. Our talk will cover operational pitfalls and how we have dealt or avoided them.

We will share our knowledge on our designs and how to recover from outages (Galera cluster, RabbitMQ, Neutron, etc.) or storage issues along with side effects (degraded Ceph cluster). We will also address topics related to monitoring (which key values you should monitor), which alerts you should expect (or “How I learned to stop worrying and love the alerts”) and our overall monitoring system. End users topic will also be approached, as we will present our (2) opensourced tools to allow users to backup their VM by themselves and to display a nice status board 'à la Amazon' (using our monitoring system).

Finally we will talk about the common issues we encountered in our experience with different architectures and share our tricks and tips :)

Join us to learn and share about the (thug) life of an osops!

While every new release of OpenStack offers improvements in functionality and the user experience, one thing’s for certain: troubleshooting is hard if you don’t know where to start.

Join us as we cover some common and not-so-common issues with Nova and Neutron that lead to some of our favorite error messages, including “No valid host was found”. Participants will learn basic troubleshooting procedures, including tips, tricks, and processes of elimination, to get their cloud back on track.

PayPal, Inc. has proven that OpenStack is well suited for large-scale production workloads. Most recently, we tirelessly built the world’s largest private OpenStack cloud to also support our non-production workloads. Now, our developers have the same capabilities that our existing Production Operations teams have been utilizing for several years.

In this talk we will share the scale and scope of our latest deployment, its common architecture, lessons learned at deploying at such a large scale, and other details to help others take a similar journey.

* Scale and Scope
* Real-world Cell usage (because we could not this large-scale without)
* Considerations for non-production workloads (it's not easier than production, we will tell you why)
* Lessons learned
* Q&A

The term quality of service (QoS) traditionally refers to the users reservation, or guarantee of a certain status of service quality. Running a cloud under the assumption that there should be unlimited resources out there for applications to consume, cloud admins are consistently looking for ways to observe user-perceived quality, and to dynamically tune the service level. Both of them should be based on real-time data generated by the cloud.

To address QoS in OpenStack we look much beyond networks. Any resource where there exists contention, there goes the need of QoS. For example:

• CPU


• Memory


• Disk IO


• Network IO


• System bus

In this talk we will explore the current status of QoS in OpenStack. we will be looking at both what is possible now and what is right happening in each project space. we will also share what we did in WebEx OpenStack deployment. How we built up the native QoS as Services framework inside Openstack, how we successfully tackled this problem systematically, and finally the most important, what we have learned along this long journey.

What happens when environments are at capacity for resources? How do we work with customers to move into different environments without live migration options? Detailed metrics regarding performance and capacity is needed to watch the various environments. At Comcast, we continue to develop and implement additional monitoring and environmental metrics in order to be successful with our capacity planning. 

Another thing to think about: What is the capacity of new environments in terms of memory, CPU, and disk? Are there any plans to add capacity to the existing environments? Or are the plans only around building out new environments?

Our panel discussion will cover topics such as resource management and capacity planning at Comcast. We look forward to sharing our lessons learned, our pain points and covering our user story and topics with fellow stackers!

Everyone wants their OpenStack upgrades to be as painless for themselves  and their customers!   We'll cover our experiences in upgrading to Kilo the problems and solutions we discovered.  This talk will cover all aspects of the upgrade with a focus on Neutron.

This talk will include:

• How to keep Neutron from dropping all your traffic during the upgrade if you're using overlay networking


• How to measure and reduce the impact of the upgrade


• Testing in development and deploying to production


• Specific problems and the solutions we used


• Orchestration of upgrading in a multi-node, multi-region environment

In this talk, Megan Rossetti and Shilla Saebi will discuss what it takes to grow an OpenStack operations team that runs a production OpenStack cloud, which is constantly growing. An important aspect of our teams roadmap is to run both the existing infrastructure while implementing the future vision for the Comcast private cloud. Building a diverse and inclusive team is no easy task and it takes dedication, time and determination. 

We will take you through the entire process from interviewing for an OpenStack engineer, to the OpenStack training we put our new hires through. We will cover topics such as qualifications for the team, operations team responsibilities, day to day tasks as well as knowledge maintenance.

OpenStack is a fast paced, ever changing, growing community and sometimes it can be challenging. We would like to share our story and let others know what works for us and what challenges we have faced. 

We hope that this talk can help provide a road map for companies and organizations to move beyond discussion to real and actionable when it comes to diversity, growth and inclusion.

CloudKitty is now part of the Big Tent, come join us and learn more about this rating
component and its features.

CloudKitty helps cloud providers define rules and charge cloud usages of their
users. Using a highly modular rating engine to interface directly with your
OpenStack cloud, application or everything else.

In this  talk we will introduce CloudKitty, an open source chargeback and
rating component for OpenStack.

It will includes:

- A global description of its architecture and capabilities
- Newest changes since last summit
- Introduction to built-in plugins (collectors, rating, storage)
- Demonstration of Horizon integration and configuration scenario
- Description of future changes and features for the Mitaka cycle

CloudKitty is getting integrated by various OpenStack clouds around the world,
so why not yours?

Scaling OpenStack is hard.  Sometimes the challenges come from gaps or issues in the software itself.  Often, however, they stem from the mere fact that growth introduces complexities.  As the number and type of various "nodes" expands, Operators are faced with several challenges around how they are deployed, tracked, interact with each other, are monitored and dealt with when they become problematic.  There is no one-size fits all solution, or one "right" way to do it, but many of us are working to overcome the same obsticles.  This talk will focus on Rackspace's approach to fleet management as we have continued to scale.

It's as much (if not more) about philosphy as it is technology.  How we choose to view various "nodes", the level of expendability we assign to each and the amount of care (or not) we'd like to focus on them in the future.  When thinking about a fleet at 10x the size it is today, we look at how we manage the inventory, provision the nodes and deal with any of the downstream issues that might arise.  Most of our tooling has been built/assembled with those things in mind.  In this talk we'll dive into:

- Our inventory management aproach - it's all about data aggreation
- Host provisioning - look Ma, no hands! 
- Remediation services - Auto correction of monitoring alerts and anomoly detection/correction
- Where we go from here - How we think about the future, What we still have to build, and yes, Virginia - we'd like to upstream the whole suite

Over the past 18 months there have been several security vulnerabilities discovered in the Xen kernel, which powers some of the largest public cloud OpenStack implementations. How do you address a security vulnerability in a timely manner for your customers whilest minimizing the impact as much as possible? What do you do when this happens again 6 months later? On the Rackspace public cloud team we had to tackle this problem. This talk will aim to address how we addressed the first (XSA-108), and what we learned to make subsequent issues (XSA-123, VENOM, etc.) easer to handle. As well as what we are currently working on to make the process even more graceful moving forward.

We used a combination of tools, driven by Ansible, to apply the fixes in a timely manner for our customers. Ultimately this is a simple patch and reboot procedure, however operating at this scale provides unique challenges that have to be accounted for.

With increasing enterprise adoption of OpenStack, Docker, and hybrid solutions, it is a continued challenge to keep operating system baselines consistent across environments. Functional, procedural, security and other mandates all need to be considered when creating images. We will discuss practical lessons learned from creating images for OpenStack, Docker, AWS, and other platforms.

We will review the current challenges and risks facing enterprises adopting these technologies, the processes and procedures for planning image creation, and the tools to achieve consistent and manageable results.

* Bake or Fry?
* Immutable Image Benefits and Challenges
* Security Lockdown
* Foundational Image Creation
* Component Reuse
* Image Build Automation

This lightning round talk will present various operators horror stories on weird things needed to keep OpenStack running.  Audience participation is almost assured as we share feats of strength and airing of grievances.


 

This talk will highlight the puppet-swift OpenStack module.  Puppet-swift has seen more attention and development recently.  The puppet OpenStack modules are now part of the OpenStack "bigtent" and have an active and diverse devops community contributing to them. This talk highlights one open source approach to automating the deployment of your swift cluster.  Starting off with a brief walk through of a multi region swift cluster running on Ubuntu at TWC that backs many critical business apps.  This talk is meant to be a blueprint for those considering a transition to or initial deployment of swift using puppet.

• We will examine key deployment points such as managing/distributing the swift rings and using hiera to map devices to drives, useful if you are migrating from another tool-set to manage swift.


• Next is a highlight of the current capabilities and recent enhancements of the puppet-swift module.


• Considerations in deployment strategy around a multi region - multi puppet master cluster.


• Along with the puppet-swift module, there are other important automation/deployment considerations that should be addressed when deploying swift with puppet.  We will touch on the areas that should be part of any deployment such as: performance tuning, log rotation, log collection and cluster telemetry using Monasca.


• The top 5 icinga checks to consider when monitoring your swift cluster health.

In very large scale, heavily used Openstack deployment like the ones we have in Paypal, resource leaks happens from time to time and the content in the Openstack databases becomes inconsistent across different components due to the distributed nature of Openstack.

The resource leaks and inconsistent data cause capacities shortage and operation failures in our cloud insfrastructure.

Please note that it is important to find and fix the underlying issues in the code. In a production environment, there are third-party services could cause the openstack into inconsistent states, for example, hardware failures in hypervisor/switch, backend storge issues, loadbalancer issues, database cluster out-of-sync, and rabbitmq issues, etc. As a cloud provider for enterprise, we also need to resolve customer issues ASAP through the quick hack.

We would like to share our experiences and lession learned on how to detect resource leaks and keep Openstack consistent.

Just like fsck for filesystem, we deployed a set of cleanup tools to check/repair the Openstack cloud.

The tool set cleans up leaking resources and fix inconsistent data not only for Openstack alone, but also other services used by Openstack (DNS server, and NSX controller, etc)

Here are the list of items being cleaned up:

1. zombie VMs. instances marked as deleted in Nova DB but still running on hypervisors.
2. zombie disk files on the hypervisor. The huge disk files left on hypervisor for deleted VMs.
3. in consistent cinder volume states acrossing five different modules: Nova DB in API cell, Nova DB in compute cells, ciner DB, the libvirt.xml of the instance on the hypervisor, and the iscsi sessions on the hypervisor.
5. Unused the DNS entries for deleted VMs, and duplicated DNS entries for the same IP.
6. Orphan ports in Neutron DB which are no longer used by VMs or 
7. Resources leaks in NSX controller, for example: virtual ports, virtual switch, virtual router and security groups.
8. nova quota out of sync and cinder quota out-of-sync
9. inconsistence caused by staled RPC message

“Someday, walls will be here.”

Les Nessman of the 70’s hit WKRP in Cincinnati fantasized that someday his open floor plan desk would be surrounded by walls, giving him greater cred among his co-workers. Everyone knew it would never happen, but they humored Les’ delusion.

Your data center is kinda like this.

Google, AWS, Microsoft, Rackspace, Facebook and a few others build data centers using lessons that can only come from massive scale-outs. It starts below the concrete and goes all the way to the staffing model—and how you buy electricity and bandwidth—all before you even start talking about pets and cattle.

Rather than living in a hyperscale delusion, embrace the reality that you will never run infrastructure at this scale, not even by half. And even if you could, it’s equally unlikely that your bosses will get on board with the Draconian ops philosophy that underpins hyperscale.

Instead, let’s glean what we can from those who actually need hyperscale and do it well. This session will highlight several specific examples of how OpenStack operators can learn and benefit from the hyperscale experiment. Commodity servers and homogeneous stack design? Yes! Evaporative cooling and 1:10,000 admin:server ratios? No.

If you’re interested in talking about real, tractive and actionable lessons of hyperscale that you can apply without pouring new concrete, then let’s gather in Tokyo and talk about it.

Monitoring large scale production OpenStack environments presents new challenges for public, managed, and private cloud providers. An effective cloud monitoring  strategy will lower your mean time to resolution, decrease TCO, and increase the stability of the OpenStack platform. In this session we will go over production operations use cases and demonstrate Monasca analyzing OpenStack performance, health, and event data to address the following use cases:

• Can my projects access compute?


• Is Horizon performance slow?


• Is my ELK stack running?


• Can my tenants create VMs?  


• And more... 

Monasca is a multi-tenant, highly scalable, performant, fault-tolerant, monitoring-as-a-service solution for OpenStack that supports storing and querying metrics, alarming/thresholding and notification methods such as email, webhooks, PagerDuty, and more. We will cover the current and future state of Monasca as it evolves from a metrics based monitoring system to supporting events and logging.

Come and learn why companies like HP, TWC, Cisco, Fujitsu and others are adopting Monasca as their OpenStack monitoring solution.

https://wiki.openstack.org/wiki/Monasca

As OpenStack becomes easier to deploy the next frontier of deploying one large cloud across regions, or multiple smaller clouds that are federated, has been a keenly followed topic of interest and an architecture that is becoming a reality.

Having just worked through implementing and automating a multi-region/federated solution for openstack-ansible, we'd like to answer the question 'What does it take to run and deploy a multi location cloud?'.

We'll cover the benefits of a multi-region/federation cloud and what these terms mean within OpenStack. As well as challenges we've run into and solutions to overcome these. This will cover in more detail, specific topics such as Authentication and Federation, as well as Global Clustering for Object Storage.