OpenStack Mitaka Design Summit

Moving from an era of vertical scaling and infrastructure resiliency to a world where reliability in the infrastructure is not assumed and application owners must start thinking about horizontal scaling and fault tolerance at higher layers can be challenging. A deep dive into what are the best practices for deploying distributed, scalable applications on cloud, where are the challenges and what solutions scale well.

Hear real world examples of how Comcast stood up an internal private cloud service leveraging OpenStack and Ceph. In less than 3 years the cloud is supporting over 500 internal customers including large scale, high performance applications such as our X1 platform and Residential Email.

A review of what architectures, tools and services work well for large scale stateless applications as well as how smaller apps tackle their deployments. Answers to where and how our apps securely manage state, store data, and plan for failure.

Manila, at its core, provides basic provisioning and management of file shares to users of an OpenStack cloud. The Sahara project provides a framework to expose Big Data services such as Spark and Hadoop. Together these two projects create a solution that is greater than the sum of its parts.

Natural synergy and popular demand led these three teams to develop a joint solution to expose Manila file shares within the Sahara construct to solve real Big Data challenges.
  
This talk serves as a brief introduction to what these two projects encompass as well as a detailed look at the joint integration work that was created involving:

 o      Sahara Data sources
 o      Manila File Shares
 o      Horizon Integration
 o      Sample Workflows
     
 Some administrative enhancements to Manila will also be covered including:

 o   Manila snapshots
 o   NFS Connector for Hadoop
 o   HDFS driver for Manila
 
Finally, a demo will be presented showing a Sahara data processing job running with binaries, data sets, and results hosted in Manila file shares mounted on a Sahara cluster.

Attendees will leave this session with an understanding of Manila and Sahara integrations and tangible use cases where it can be leveraged as a key component of a Big Data application deployment.

Learn about the newer features of OpenStackClient:

* simplify the experience of communitcating with multiple clouds
* operate as a long-running process to eliminate repeated loads/forks as a subprocess
* explore the plugin interface and capabilities exposed
* what changes with the adoption of the Python SDK?

Let's talk about the application workloads and scenarios you can envision on a truly open cloud. What do you think about owning your health data or fitness data? How about running the infrastructure on an OpenStack cloud?

After learning that her eleven-year-old son was diagnosed with Type I diabetes in 2014, Anne began investigating technology to help with the daunting task of living with the day-to-day management of the condition. True story, she found a way to use OpenStack resources to monitor data from his continuous glucose monitor (CGM) to gather not just five data points a day but nearly 300 data points in a 24-hour period.

Hear the story of finding Nightscout, a collection of open source projects that can run real-time monitoring applications on OpenStack clouds.  While the project's original approach to collect the data required an Android phone attached to the CGM receiver, the project now has a REST API for collecting and displaying data. Justin has a family member with Type I diabetes also, and we can compare and contrast the proprietary approach with the open source approach. We can demonstrate the Nightscout web application and iOS app running on a Rackspace Cloud server with node.js connected to a MongoDB database.

While the CGM use cases are medical devices with medical data, we also want to explore consumer data collection such as fitness trackers. Let's discuss the full spectrum of open source and open data in an open cloud.

In this session, we will talk about stuff that's going to make developing Swift easy by offloading the hard problems of storage. OpenStack Swift makes your application better. We can find dozens of storage systems that can do read write and delete but the fact that you can use swift for these advanced features means that you're able to focus on making your app awesome rather than worrying about the hard problems in storage.

We will cover how application developers can take advantage of some of the more advanced features in Swift, including... TempURL, StaticWeb, FormPost, and Static or Dynamic Large Objects among others.

IoT is one of the biggest topics in IT system today.  In this session, we will discuss how we can achieve an effective IoT system on OpenStack.

Firstly we'll describe IoT use cases, and summarize some generic requirements for IoT backend.  Secondly, we'll present our reference design of IoT backend on OpenStack IaaS.  Finally, we'll discuss the result of fit and gap analysis of OpenStack itself as a platform for IoT backend.

This session includes following items.

What kind of components we need to enable IoT backend
How to design and create network model to gather up all data from distributed sources
How to support flexible data gathering, storing and processing of massive data
How to archieve multi-tenanty required for IoT platform

I'm slim shade yes I'm the slim shade all you other slim shades are just imita ... ok, so the joke only makes it so far before the rhyming breaks horribly.

The OpenStack Infra project is one of the largest and most active OpenStack End Users in existence. As a result, we've learned just about everything about how OpenStack operates for consumers, the ways it can fail and the way different clouds diverge. We've encoded all of that learning in a library called 'shade' which wraps the OpenStack clients in the business logic needed to get your work done. shade is currently at the heart of Infra's nodepool project, which is the program that provides a pool of nodes across multiple public clouds so that everyone's OpenStack test jobs can run, as well as the new OpenStack modules that are in the Ansible 2.0 release.

In this talk, we'll talk briefly about the motivation for and challenges of writing shade - but we'll mostly talk about how to use it to get stuff done. In the process, there may be some ranting about things which could work better but don't.

I do not promise to not rap.

You have got your IaaS cloud up and running but are your developers happy? Has your IaaS forced your developers to also become agile sysadmins? At Symantec, we want out cloud developers to remain developers and this means abstracting away complexities of compute, storage and networking details and instead focus on actual software development.

After we built our Openstack cloud at Symantec, our next logical step was to build a Platform as a Service that to provide even more automation and streamlining to our customer’s experience. In this talk, we will share our experience and lessons learned in building a unified customer facing PaaS solution around our Openstack cloud.

Some of the topics we will cover are:

• The results of our evaluation of different PaaS platforms that are out there like Cloudfoundry, Openshift and Deis


• Potential design architectures to use when building your own PaaS from scratch


• Adding multi-tenancy to your PaaS


• Benefits of running on both containers and VMs

In this session, we will show you how to use OpenStack's Heat, Monasca, and Neutron, to autoscale your application! Autoscaling helps you maintain application reliability and intelligently scales your infrastructure while managing infrastructure costs. 

In the Liberty release you will be able to use performance and health metrics to trigger scaling policies in Heat. This powerful feature enables the following use cases and much more:

• Dynamically remove VMs when load is low


• Automatically scale up to larger VMs when load is high


• Increase reliability and automatically recover from failed VMs


• Automatically rebalance application traffic via the Neutron LBaaS

As part of this presentation the audience will learn more about how we have implemented autoscaling, some of the challenges faced during the implementation, and application autoscaling best practices. We will also demonstrate horizontal autoscaling a sample web application while rebalancing incoming traffic with 0 down time. 

Do you know C#? Want to learn how to write a scalable cloud application
using an OpenStack SDK? Come to this presentation!

We will discuss how you can use OpenStack.NET to
*    Create and destroy compute resources.
*    Scale available resources up and down.
*     Use Object and Block storage for file and database persistence.
*    Make cloud-related architecture decisions such as turning functions
into micro-services and modularizing them

On one hand, OpenStack is quite promising with open APIs, open-source reference implementation with extensible plug-in architecture to support different implementations. On the other hand, it is a nightmare for software developers building enterprise-grade distributed applications on-top-of OpenStack i.e., using only APIs available to users/tenants.  Such applications need to be elastic (scale-out and scale-in as loads fluctuate), highly-available (several 9's of availability), and support high throughput (several Gbps of traffic). Unfortunately, many of the primitives available in the physical infrastructure to be able to build such services are either non-existent or available via ad-hoc extension APIs at the virtual infrastructure layer in OpenStack implementations.


The Good (examples):

• Nova's CRUD API for VMs: The basic VM creation and deletion with several options are pretty solid APIs in the OpenStack and this allows an application orchestration engine to easily scale-out and scale-in the app VMs as needed. The placement options exist albeit bit restricted.



• Neutron's CRUD API for networks, ports, router, subnets: Neutron abstractions of the networks, ports, routers, and subnets makes it easy for an application orchestration engine to connect VMs to right networks for creating multiple app tiers with proper isolation.

The Bad (examples):

• No notification APIs from OpenStack services such as Nova, Neutron, Glance, and Keystone: There is no way for the application orchestrator process to timely detect rebooted/deleted/dead application VMs or ports. The only option is to periodically check.



• Security groups API mess: Depending on whether Nova is implementing the security groups or Neutron, the semantics of the APIs are different!!



• Network performance: The reference implementation's stack is too complex (layers of different bridges connected by veth pairs, iptables with resource intensive nf_conntrak) with multiple bottleneck locations. And different vendors' plugins have myriad of other limitations.

The Ugly (examples):

• Too much restriction on network connectivity without exposing proper APIs to customize: For example, source IP address spoofing is not allowed even in a local network, where as that is an essential primitive for building highly available services. Ad-hoc neutron extension APIs such as allowed-address-pairs and port-security APIs alleviate this but not being part of core APIs means it is not no guarantee that they are available everywhere



• Semantics of APIs left to the interpretation of plugins: for example, in a physical world, all servers connected to a switch can communicate at L2 without any restriction. However, with all of the neutron plugins, there is one or another restriction imposed that is not explicitly indicated (for example, can't have the multiple fixed IPs on a port, can't have the same IP on two ports, etc.)

OpenStack gives you a non-proprietary and extensible cloud. Microservices and Docker allow for extensible app architecture, and a vendor-agnostic, scalable infrastructure. While Microservices simplify app deployments, they come with a price: because it is so fragmented, it is more difficult to track and manage all the independent, yet inter-connected, components of the app.

With a combination of Docker, OpenStack, and an end-to-end orchestration layer you can have a Microservices architecture, while supporting easy deployments across Build, QA, and Production environments, with a scalable, centrally managed Openstack infrastructure.

This seamless integration between the application and the infrastructure simplifies and accelerates your DevOps processes and software delivery pipeline, while maximizing compute resources.

Using real examples and a live demonstration (JIRA, Jenkins, Chef, Selenium), this talk would cover best practices and tips for enabling a robust, scalable and extensible DevOps infrastructure to support today’s modern app delivery – all the way from architecture, pipeline design, build, test, and deployment.

Terraform is an open source tool that enables users to declaratively design infrastructure and have those designs materialize into working components. It is useful for developers who want to create or manage IT architectures across multiple clouds.

OpenStack support was added to Terraform earlier this year, enabling users to deploy cloud-based infrastructure inside the OpenStack clouds.

This session will cover the following topics:

• What Terraform is


• How to write a Terraform configuration


• Using Terraform with OpenStack


• Terraform and OpenStack internals

No prior knowledge of Terraform is required to attend. By the end of the session, you'll be ready to start deploying cloud infrastructure inside OpenStack using Terraform.