OpenStack Mitaka Design Summit

Ironic currently supports provisioning of  bare metal deployments on a flat network. While this may be acceptable in small or test deployment scenarios, it is not a desirable solution for larger deployments where multi-tenancy support is needed. Flat networks does not provide isolation of tenant traffic. Therefore, operators end up creating extensive infrastructure to provide isolation for tenants for their deployments. Ideally, Operators desire to utilize same tenant isolation for their bare metal deployments that is available for virtual machine deployments (i.e VLANs or VxLAN based isolated networks). 

We propose utilizing Neutron networking in Ironic for bare metal deployments in a similar manner as these networks are available to Nova for virtual deployments. This streamlines and simplifies the bare metal deployments while providing full multi-tenancy support. 

Ironic and Neutron teams have been working together to make this a reality. We will present the details and deep dive of this implementation in this session. 

Come and learn how you can take advatage of this framework for your deployment scenarios.

OpenStack Image Service (aka Glance) has been around from the earliest days of OpenStack and has been evolving ever since. 

It's been three years since the last major update of its API - the v2 - went live with the Folsom release, and it is time now to move forward. With the recent introduction of new large features, such as Meta Definitions and Artifacts, the time has come to introduce a new version of Glance public API - V3. 

In this session, Glance driver Brian Rosmaita and Artifacts driver Alexander Tivelkov will talk about the history of Glance API, the way it made since the initial release and the challenges it now has. The attendees will learn about the new experimental version of Glance API, the plans to deprecate the v1 and the new amazing features which are available for Glance users.

This presentation reviews the development of a reference architecture for High Performance Compute services on OpenStack. We will examine:
 
• The performance impact of Nova and KVM versus bare metal deployments 
• Cells vs Regions for segregation 
• Storage choices and the pros and cons of each. 
• Networking components and the challenges and choices to be made. 
• NUMA tuning and its impact 
• huge page tuning 
• DPDK and SR-IOV 
• GPU Pass-through.

Bare metal provisioning is inevitable for cloud. Especially in multi-vendor large environment, it is a daunting task which is difficult to be fully automated because failure could happen easily in many ways.

This session will present new Ironic features we contributed in Kilo and Liberty, and is going to contribute in M and later based on the vision that we propose "OpenStack for mission critical platform".

Fujitsu sustains many mission critical systems of social infrastructure, such as baking, stack exchange, factory automation, and government agency system.

Based on those experiences, we believe that the most important customer values are:

• Truly open, no vendor lock-in system to provide customer with freedom to switch any vendor anytime


• Reliable, robust, highly available system to operate customer's business continuously


• Responsive, responsible, competent support to resolve customer's incident quickly and accurately.

As the first step towards the final goal, we contributed the following features:

• Virtual Media Deployment for large scale multi-vendor environment


• Bare metal Graceful Shutdown for better maintenance


• NMI dump for better support


• Ironic Network Neutron SG/FW packet logging feature

As the next step, we plan to contribute:

• Ironic-Nova Integration for unified operation


• Ironic-Neutron Integration for multi-tenant support


• Ironic-Cinder Integration for N+1 redundancy support


• OpenStack logging improvement

This presentation will also show Ironic demo of implemented feature above.

This presentation starts with a brief introduction to OpenStack's Nova project, including a description of Nova's mission and scope.

Then we will take a whistle stop tour at some of the big things the Nova project has been working on during Liberty. Nova is currently doing a lot of architectural evolution work. Learn about how Nova is evolving its public API. Discover what Nova is doing in its drive towards zero downtime upgrades. Learn how Cells v2 is likely to enhance every Nova deployment. Find out about the Feature Classification effort that is happening, including looking its work to describe how some technology choices can limit the features that are available to you, and our push to try and plug those gaps.

We will finish with some pointers on how to get more involved with the Nova project, and how to find out more about what has been happening during Liberty.

What exactly happens when you click Launch Instance in OpenStack’s Dashboard?

The basics are pretty well understood, but how about if we look a level deeper than the simple, abstract narrative? What are the technologies involved below OpenStack? How does OpenStack coordinate those technologies to get you a running VM that you can SSH into?

In this session, Mark will cover as much of much of the high-level and low-level details of the story of your “Launch Instance” request right up until the time that you get a shell on your VM. By the end of the session, even the most seasoned expert will hopefully have learned some surprising facts and be eager to learn more about some obscure detail of how this crazy cloud thing works!

Compute node high availability means when hardware or network fails, or the host operating system crashes, the node should be fenced and shut down, and instances on the node get relocated and rebooted on other compute nodes. In a vanilla OpenStack deployment, it demands the tenant workloads provide falt-tolerance and failover abilities. While this assumption is true for modern clustered applications, there are still many IT solutions in traditional industries relying on compute node high availability. This makes a barrier of OpenStack deployment in traditional enterprise IT.

In small deployment, it's tempted to setup a monitoring service for compute nodes, and call Nova host-evacuate when a compute node fails. However the monitoring service itself becomes the single point of failure and hot point. There are also proposals or maybe implementations to use ZooKeeper and Pacemaker (with Pacemaker-remote) because they provide heartbeat and membership service. The basic idea is that we have compute nodes register as ephemeral znode, and ZooKeeper maintains a heartbeat. We can also run Pacemaker-remote on the compute node to achieve similar effect.

The problem is that the heartbeat usually runs on the OpenStack management network. However if a host has good storage network connectivity but just failed management network connectivity, we should not consider it failed and perform fencing and evacuation. Because failed management network connectivity just means we can not boot new instance on the server, and it does not affect the running instances, so evacuation will cause unnecessary downtime for the tenant workloads. On the other hand, if the management network is good for a host, but storage network fails, we should fence and evacuate the host. The ZooKeeper and Pacemaker-remote type of solution also suffers from the scalability problem, because the heartbeats happens between a few ZooKeeper/Pacemaker server nodes and many compute nodes.

Hence we propose a distributed health checking mechanism for compute nodes. It can deal with compute node power failure, host os crash, memory going bad, disk failure, interrupt of management/storage/tunnel network and so forth.

We use Gossip protocol for distributed heartbeat checking. The Gossip implementation comes from the consul project (consul.io). The main idea is to run an agent on compute node, and have them probe each other. The agent on the compute node can also check and monitor many types of things like OpenStack services and hardware status.

We run distributed heartbeat checking on all the OpenStack logical networks, usually management, storage and tunnel network, and report the network connectivity and other monitored status to the controller node, and let the controller node decide if we should fence and evacuate the node based on all these information. We present and discuss an example decision matrix. It's also possible to create a plugin for Ceilometer to report the data and events gathered from the distributed health checking, so admin can register alarms and add handlers, and decide what to do in a highly flexible way.

We also propose a fence mechanism based on custom Gossip query to complement IPMI remote powering off. In case of host power failure, it's not possible to distinguish IPMI network failure and actual power failure, the symptoms are the same. If we fail to ensure the power state of the failed host via IPMI, we send a custom Gossip query to the targeted node. Upon receiving the query, the target node sends ack and stops feeding the hardware watchdog and have the watchdog shutdown the host. If the node does not receive Gossip heartbeat from all the other nodes, it should fence itself and shutdown the host. In these way, the fence request either reaches the target host, or the host fences itself in case of network connectivity failure, or the host actually experienced power failure. From the perspective of controller, given a reasonable time, it can be sure that the failed host has been powered off, thus it will be safe to perform host evacuate.

Virtualization drivers (e.g. libvirt, QEMU/KVM) are the core part of OpenStack Compute layer. An OpenStack environment is challenging to debug as is -- more so when multiple Compute nodes and thereby multiple libvirt daemons and QEMU instances are involved. A good grasp of Virtualization debugging mechanisms is vital for effective root cause analysis. To that end, libvirt and QEMU provide a rich set of debugging controls that allow us to query (or modify) the state of virtual machines in distress.

This talk focuses on providing an in-depth view of aforementioned techniques. Topics include: debugging Nova Compute process crashes; gathering specific patterns from libvirt log filters, libvirt environment variables, and systemd journal fields; live querying the VM (and QEMU) state through `virsh` and QEMU Machine Protocol (QMP) commands; tuning the libvirt daemon logging; monitoring events emitted by QEMU, etc.

Audience would include OpenStack infrastructure operators, Virtualization (libvirt/QEMU/KVM) administrators, developers, tinkerers, or any one interested in understanding the Virtualization layer in OpenStack to help equip yourself with better debugging techniques.

OpenStack Compute provides a number of facilities for moving instances around, but it's not always immediately obvious how they differ from each other. In this session learn the differences between each of the available options including evacuations, cold migrations, and live migrations as well as the internal mechanics of each including some of the ways they can differ when using different hypervisor backends. You will also learn about the pre-requisites for enabling each method and the optimal configurations for ensuring the right combination of security and performance.