OpenStack Mitaka Design Summit

Swift makes an ideal storage solution for web applications that need to store large volumes of data, be it photos, videos, or any other larger type of media data. Application developers no longer need to care about
the storage and possible growth because data, metadata, and application logic can now be clearly separated. This simplifies the development process and at the same time ensures a high scalability of the whole system.

In this talk, you’ll learn about an overview of Swift itself with a focus on some very useful features inside Swift for (web) application developers. Then, you’ll learn how to make use of these features with 2 very popular web frameworks, AngularJS and Django. When you leave this session, you should have a good overview how to start your own Swift-powered applications.

Shingled magnetic recording (SMR) drives have the potential to significantly reduce storage costs. However, they have very different characteristics than standard drives. In this talk we will give background information on SMR drives and talk about performance of both device-managed (use firmware to hide differing characteristics) and host-managed (rely on the application to manage the differing characteristics) SMR drives under various simulated Swift workloads modeled after public cloud access patterns observed at SoftLayer.
 
We also present a simulation of what performance would be possible if Swift's DiskFile abstraction was modified to be SMR aware by performing sequential writes, a key requirement for utilizing SMR disks. We then discuss methods to optimize Swift deployments for SMR drives.

The session will highlight:
- Background information on SMR drives and their management models
- Experiments to characterize SMR drive performance
- How to tune Swift to improve performance with SMR drives
- Potential Swift changes to enable sequential writes that would enable better usage of SMR drives and expected performance

One of the unique features of an object store is the ability to associate user defined metadata attributes and values with containers and objects.

However, in order to leverage the full potential of valuable metadata, efficient metadata search is essential. Although this feature is currently missing in Swift, it is now gaining significant interest in the community. 

In this talk, we describe the components of design and implementation of a metadata search capability integrated with OpenStack Swift. This facility has a rich metadata search API, and enables key applications that were not previously feasible, such as bio-informatics applications and analysing a media repository. In the public transportation domain, we demonstrate a smart city IoT service which collects bus trip history for EMT Madrid and stores it in Swift, Geo-spatial and time series metadata search enables efficiently retrieving the data of interest - for example it supports queries such as retrieving the bus trip history for a certain bus line close to a major sporting event which took place last Saturday. This enables efficient analysis of the data, for example comparing it with data for the same event in previous years. 

At Rackspace we have pushed swift to its limits. While it has performed amazingly well over the years there have been some pretty hairy moments and there are still some unresolved issues. We will share some horror stories about features gone wrong and some larger issues that we are in part trying to solve by rewriting the object server layer in golang.

Swift is an object storage system designed for data that needs to be instantly accessible, stored forever, and accessible from multiple devices. In short, it is the engine that runs the biggest storage clouds on the planet. This session will cover Swift's architecture, its technology capabilities, and real-world infrastructure-as-a-service use cases. This OpenStack Swift introduction is great for attendees who want to understand the design goals of Swift and how they can best make use of this OpenStack component. It will be an informative introduction for those interested in running Swift or contributing to the Swift project.

Ceph is a fully open source distributed object store, block device and file system. The Ceph community and itself has become enough matured. It is one of the most common storage solution in OpenStack deployment. Meanwhile, Server SAN storage come up these years and become more and more popular in the openstack deployment.

The most well-known Server SAN are Solidfire and ScaleIO. Both of these storage can be used as cinder backend to provide block storage, just as the same of ceph. However, in addition to providing block storage, what are the differences of them. Faced with these options, the user will be confused to choose a suitable storage.

In this session, we make a comparison between ceph and the above-mentioned commercial Server SAN from storage function, deployment, operation and maintenance, and performance. Hope these content can provide some help in your OpenStack storage selection.

Continuing the conversation from Vancouver, meet a number of Cinder developers including Core team members.  Hear thoughts from active developers on the state of Cinder, where things are going, and some of the current challenges with the block storage project.

This session will also provide the audience time to "ask the experts" in an interactive Q&A format.  This will be a great opportunity to get your questions answered and voice your ideas, opinions, and concerns about block storage within OpenStack.

One of the biggest barriers  for enterprises interested in deploying OpenStack today is the inability to leverage existing assets - including infrastructure, workloads and their inter-relationships. Committing to an OpenStack deployment typically forces IT to start  from scratch, ignoring any important workloads built up over time. This turns out to either be a non-starter for some organizations, or results in them having to run separate silos of existing, critical workloads, alongside a new OpenStack-based private cloud. However, OpenStack can be taught to learn - and leverage - existing enterprise infrastructure, and incorporate it seamlessly into a live private cloud. This enables users to get up and running with a fully functional private cloud, already plumbed with their existing assets. 

Such learning involves not only an initial pass, but also periodic incremental updates to synchronize the enterprise infrastructure view, with OpenStack’s current state model.

In this talk, we will describe how we, at Platform9, moved our existing dev-test workloads, and infrastructure to an OpenStack-based private cloud, running on vSphere, using a set of such changes for Nova, as well as Glance, which entailed: 

- Enabling Nova compute to discover already existing cloud instances
- Enabling Nova Network  to discover and to integrate existing networks
- Enabling Nova Network to dynamically discover host IP addresses
- Enabling Glance to discover already existing VM images and import them into the Glance repository
- Adding capabilities in Openstack to deal with out of band changes in the cloud environment such as deletion of VMs, network and storage reconfigurations

The benefits that these additions provide to enterprises, in terms of being able to leverage their existing infrastructure within a modern self-service private cloud, are immeasurable. So much so that we made this part of the core feature set of the Platform9 cloud, and believe that it would add great value to the OpenStack project.

Moving Enterprise applications into OpenStack isn’t always easy. Many organizations rely on hundreds and thousands of applications to run their businesses. Migrating those applications one by one can seem like a never-ending journey. Rewriting all of them is not a viable option either. In this session, we will explore a few approaches, from nested-virtualization, containers, and API portability to Orchestration and PaaS, which provide different levels of abstraction, making this transition more seamless. The session will also include real life use cases from large Enterprises (such as TD Bank) and well known Telcos to teach listeners the lessons they learned from their experiences.

PayPal has successfully deployed an OpenStack environment over several availability zones with  300,000+ cores and petabytes of block storage. It is one of the largest OpenStack Cloud hosting ~100% Web, Mid and API workloads for PayPal production and is in operational for 3 years. One of the keys to success has been a tiered application architecture that allowed many front and mid-tier modules to be moved into the OpenStack private cloud, while back-end functions continued to reside on the non-cloud infrastructure. It provided agility and flexibility without compromising on high levels of reliability and availability.

PayPal being one of the early super-users of OpenStack for building Enterprise Grade Cloud, it met mutiple milstones to run its mission crtiical production workloads along with Developer/QA/Stage/Pre-Prod. Recently, PayPal announced that they are running ~100% of production for web, mid and API on OpenStack Cloud. 

Come and join this session for key strategies and guiding princples that PayPal adopted to meet their goal.

Main Topics to be covered:

1. OpenStack Deployment Architecture with Multi-Cells


2. Key Learnings in operating largest AZ with 1000s of hypervisors


3. PayPal PaaS deployment Architecture in OpenStack Cloud


4. Dealing with day-to-day Large Scale issues

As the OpenStack ecosystem experiences consolidation of OpenStack providers, the landscape of tools, platforms, and vendor-provided hardware/software solutions above and below the stack continues to grow. While the variety of OpenStack-compatible technologies means that users and consumers have a tremendous amount of choice, it also presents an ever-expanding set of decisions that must be made by both users and providers of commercial OpenStack Solutions. In this session, panelists will discuss:

• The balance between flexibility/choice and vendor lock-in




• Building loosely coupled/independent services versus fully supported/validated full-stack solutions




• The impact of PaaS, containers, APIs, and application deployment/management methods   




• Barriers to exit and switching costs 




• Emerging technologies and what the community can do to ease adoption and impact for users, operators, and providers 

This is a must-attend session for those building OpenStack clouds, whether you’re building your own or building commercial solutions.

Moderator: Niki Acosta, Cisco Cloud Evangelist. 

Panelists include:

• Shamail Tahir, Cloud Architect, EMC Office of the CTO




• André Bearfield, Senior Director of Product, Blue Box




• Aaron Delp, Technical Solutions Director, SolidFire

Ironic is released as OpenStack integrated project from kilo release. Many users, in particular enterprise users, want to use bare metal servers for DB, HPC etc. and so Ironic gets a lot of attention recently. To provide bare metal servers, Ironic needs to manage all the server life cycle. Typical server life cycle is following:
 
 - Register
 - Discovery a node
 - Register a node
 - Introspection a node
 - Ready
 - Deploy
 - Turn on a node
 - Install OS to a node HDD
 - Connect a node to appropriate network
 - Attach volume to a node
 - Boot a node from volume
 - USE
 - Turn on/off a node
 - Access to a node via ssh/console
 - Cleanup
 - Clean up a node HDD
 - Initialize node firmware/BIOS setting
 - Initialize RAID configuration
 - Turn off a node

Althorugh Ironic developers put in great effort on its improvement, there are serveral issues on each stage. For example, there is no network isolation per tenant. That means all Ironic users need to use same flat network. Also, current Ironic supports neithor attaching external storage volume or booting from volume. In enterprise area, users commonly uses external storage to better throughput, latency, lower overhead back up etc. and so this is one of a key feature for them to use Ironic.

To use external storage volume in Ironic, Ironic needs to work with Cinder (and Nova). In this session, we explain ironic current status first and then introduce our effort to realize Ironic and Cinder integration.

OpenStack users deploying Ceph for block (Cinder) and object (S3/Swift) are unsurprisingly looking at Manila and CephFS to round out a unified storage solution. Ceph is based on a low-level object storage layer call RADOS that serves as the foundation for its object, block, and file services. Manila's file as a service in OpenStack enables a range of container-based use-cases with Docker and Kubernetes, but a variety of deployment architectures are possible.

This talk will cover the current state of CephFS support in Manila, including upstream Manila support, Manila works in progress, a progress update on CephFS itself, including new multi-tenancy support to facilitate cloud deployments, and a discussion of how this impact container deployment scenarios in an OpenStack cloud.

Capabilities. Migration. Replication.

No one agrees what any of these things mean outside of their own context. Each vendor has their own definition of these, and it simply brings every discussion to a standstill. This is neither productive nor helpful for the Cinder project or the vendors themselves. We like to wrap Cinder up in a nice clean box and call it just "Software Defined Storage" but that, too, is pretty meaningless.

Cinder has seemingly been stuck for several cycles with the same topics coming up each summit. The Cinder development community needs to answer a few very hard questions or this state will continue. What are these questions? I'm glad you asked.

• What should Cinder do?


• What shouldn't Cinder do?


• What is interoperability and how much makes sense?


• Why do we think our operators are unable to configure their cloud?


• Why should every capability (sorry) of a backend be exposed through the Cinder API?

Swift Erasure Code support is finally here and the big question now is “How does it perform?”  There has been much speculation about when it should be used and where the tradeoffs will be for cost versus performance.  This talk will end the speculation with a carefully designed performance study carried out by Intel and SwiftStack on multiple clusters with various configurations of CPU, memory, network and media.  We will compare some basic workloads for EC versus replication with all other factors the same and see exactly how things stack up.  In addition to the data, we will talk about some experiments in tuning Swift and how various parameters affect the performance per policy.

Image management takes a lot of time to upload and download the large disk image to Glance. By using rich features such as copy-on-write, backup, etc. provided by storage arrays, we can make the image management more efficient.

We are proposing improvement of Cinder volume backend for Glance. This realizes:

• Storing image data in Cinder volumes (upload/download).


• Sharing volume data among tenants.


• Quick volume-boot of virtual and bare metal machines using copy-less volume creation from images.


• Copy-less image creation from volumes for backups, image sharing etc.

To fully enable auto-scaling for applications, we need to boot instances as quickly as possible to respond the change of demands. By levareging the copy-on-write based volume creation, instances can be booted at once without copying image data. We also introduce quick boot for baremetal servers without copying images by combining with Ironic-Cinder support.

The Block Storage project (Cinder) is often overlooked but can be critical in an OpenStack deployment. In this presentation, we'll have OpenStack experts from the deployment side as well as the development side join forces and walk through not merely the basics of Cinder, but show how Cinder is being deployed today and provide some use case examples and demos!!  Here's your chance to see a live demo go horribly wrong, or go extremely well... either way you WILL be entertained and learn some new things.

This presentation will focus on the vendor neutral use cases for block storage in OpenStack but the demo will provide a specific example of how Cinder can be deployed in production.

You have your single Swift cluster up and running in multiple geographies...now what? What could your future self tell you that would save you lots of late nights and lost weekends?  Within Symantec we have been working through the nuances of operating a single Swift cluster spanning multiple geographies and would like to share our experiences in what we have learned, and wish we knew, during the process.

In this talk, we will be sharing our insight on:

• Meeting customer demands: storage patterns and anti-patterns


• Impact of high network latencies


• Performance considerations and trade-offs


• Considerations for Erasure Coding


• Areas for Optimization


• Essential Monitoring

Block storage replication has been available in the Cinder code since Juno, but has experienced many challenges.  An effort is underway for Liberty to make replication more feature full, working, stable and usable.  This session will explain the state of replication as of the Liberty release and where future enhancements will focus. A demo will be shown and a feedback taken for M release planning.

Key Take-aways for Attendees:

• Understand the use-cases for replication in OpenStack


• Awareness of any limitations in replication for the Liberty release


• Configuration options and requirements for replication


• Walk through an example configuration of block replication on at least one storage platform

Monitoring of infrastructure and applications is key to success. Your Swift Object Storage cluster is no exception. You need to know what your environment is doing and how it is doing at all times. You also need to be able to learn from it and be able to predict issues coming up. Setting up continuous monitoring of your infrastructure and being able to alert in real-time on failures or conditions is a requirement. 

In this session Martin Lanner with SwiftStack will go through an ELK stack setup and provide hands-on examples for how to monitor an existing Swift cluster. In the past few years organizations have started to move from monitoring solutions that detect a certain state and alert on it, to more dynamic solutions like the Elasticsearch, Logstash, Kibana (ELK) stack.

Manila is a community-driven project that presents the management of file shares (e.g. NFS, CIFS, HDFS) as a core service to OpenStack. Manila currently works with a variety of storage platforms, as well as a reference implementation based on a Linux NFS server.

Manila is exploding with new features, use cases, and deployers. In this session, we'll give an update on the new capabilities added in the Liberty release:

• Integration with OpenStack Sahara
• Migration of shares across different storage back-ends
• Support for availability zones (AZs) and share replication across these AZs
• The ability to grow and shrink file shares on demand
• New mount automation framework
• and much more…

We'll demo how to install Manila using popular distributions, and discuss the proposed blueprints for the M release and will have several core team and community members present to answer any questions you might have.