OpenStack Mitaka Design Summit

We have split alarming function to new service "Aodh" successfully, let's check further development plans for alarimg.
(cross project topic)
- Reify Aodh Independence
(aodh internal topics)
- aodh client or openstack client?
- (potential) new evaluation for combination alarms?
- check work items; event-alarm/multi-worker, event-alarm/rbac (event-alarm only?), quota and tempest/rally support (to identify performance problems)

https://etherpad.openstack.org/p/mitaka-telemetry-alarms

without a UI of some kind ceilometer can seem rather opaque. If the integration in horizon was both more interesting and more
performant that would be helfpul. What can we do to improve it?
- horizon attempts to show meters (and slowly is trying to show events) -- a lot of the feedback in survey is that it's unusable.
- how to visualise events
- how to leverage existing gnocchi+graphana integration
- there are other consumers such as cloudkitty which uses ceilometer distinctly for chargeback, how do we create an interface for them?

https://etherpad.openstack.org/p/mitaka-telemetry-ui

this talks about using versioned queues to handle upgrades rather than oslo.versionedobjects. the idea is to transmit data to another
queue each time we change. the receiving service would then need to listen to multiple queues.
- does it cover all uses cases?
- how do we handle udp? with versionedobjects?
- how does versionedobjects handle db?

https://etherpad.openstack.org/p/mitaka-telemetry-upgrades

Componentisation Part 2

- how did the last one go
- what should we do next
- how can we do it better
- should we bail and just focus on bug fixing, performance and tech debt?

https://etherpad.openstack.org/p/mitaka-telemetry-split

In this small working group we'll be planing changes needed to add gnocchi support in CloudKitty.

We plan on using gnocchi as a new collector backend, and possibly as a storage one to store the ratings back with the input data. Some changes might be needed on the gnocchi part to be able to store new data types.

For more informations, please see:
https://etherpad.openstack.org/p/mitaka-cloudkitty-gnocchi

Functional and Integration Testing

We now have reasonable infrastructure in place to do functional and integration testing across ceilometer, aodh and gnocchi. However the actual level of testing being done is not really complete. In general we confim that the APIs are working but end-to-end full service tests that get into the corners are not really present (a notable exception is the new integration gate job, which is awesome let's have more of them).
Some specific concerns:
- In ceilometer, when the functional/unit test split was done, coverage provided by the unit tests took a pretty big drop. It's not yet clear if a combined run of the unit and functional tests provides robust coverage.
- The aodh functional tests check the api but do not check the evaluator, notifier or event listener.Until recently the gnocchi functional tests did not check that gnocchi-metricd was doing its job.
- Ceilometer now has its grenade testing in tree but very little is tested, really just sanity checks. We can do whatever we like here so it would be useful to more complex investigations. One thing to consider is what's being called partial-upgrade-testing (in line with rolling upgrades above): Upgrading some of the pieces of the system to the new version and leaving others behind and confirming that behavior is okay.
- For those projects which are not on the release cycle, do we need some kind of upgrade testing for them?
- We need more complete tests for those systems which use the messagaing bus and are primarily driven by notifications. This has proven a bugbear in testing because latency can throw things out of wack.
- We also need better tests of transformers and other members of the measure sampling pipeline as corner cases are constantly revealed or situations where a change elsewhere in the system violated an untested assumption in the pipeline. The new testing tool "hypothesis" might be useful here.
- We need tempest plugins for Ceilometer, Gnocchi and Aodh. (right? or the new integration gate job can cover the same level of testing?)-

https://etherpad.openstack.org/p/mitaka-telemetry-testing

Events - Business Intelligence

we stopped providing volume=1 samples so we need to fill that gap it'd be good to start generating statistics from events such as:
- lifespan of resource (duration of a resource)
- event count
- what else?

there are many possibilities to derive data from events:
- track time difference between events (compute.instance.create.start and compute.instance.create.end) to derive meters
- other stuff?

https://etherpad.openstack.org/p/mitaka-telemetry-bi

Refining Polling

instance metadata refinement
each poll for instance related meters per host, polls the nova api for metadata details and subsequently polls hypervisor for actual metric
values. this means for each host, we hit the nova api. the idea here is to separate the metadata polling from metric polling so that we poll
nova-api only once. this does mean that the metadata associated with metric potentially can be unsync'd.

healthcheck events
we are blocking volume=1 samples so now the polled volume=1 samples are useless. they do offer ability to function as resource
healthcheck which is still useful as events. this is discuss what we need to do build events from polling.
- do we just switch the output to build events rather than samples?
- how do we define/name healthcheck events

creating a new polling definition
we currently leverage the pipeline definition to define polling. we probably shouldn't. if we define new interface, we can truly separate polling from processing.
- we can also consider how to handle more granular polling.. ie polling frequency per project.
- Stevedore as configuration is a bit problematic too: http://lists.openstack.org/pipermail/openstack-dev/2015-September/074203.html
- polling logic to handle same multiple interval polling
- ie. one pollster set for 5, another set for 10, we should really only poll once.
- Do we want to continue with batching the polling tasks as a single task per interval. Isn't it more logical to have a single task per poll at whatever interval it wants?
- this works but we need to redefine what a 'task' is. currently a 'task' is a request to retreive a single meter. using this definition we will have a redundant calls as we often are able to get multiple data from a single api req.
- small requests are more scalable (horizontally) and things which do just one thing are easier to debug

https://etherpad.openstack.org/p/mitaka-telemetry-polling

Giving power back to the projects

As a system Ceilometer violates the DRY principle in a pretty horrible way: It keeps, within itself, all kinds of knowledge about the projects that it monitors and measures. This is a technical and social error which makes it difficult for the system to evolve quickly and address its own shortcomings because it is constantly looking to add and maintain support for those other tools. Ceilometer is already stuctured to make this way of doing things not a requirement. We should considering pushing functionality back to the monitored for the sake of both sides of the arrangement. Areas to consider:
- We now have declarative meters in yaml that could be broken up into files kept by the projects.
- Pollster plugins and notification listeners are stevedore plugins. These do not need to live in the ceilometer namespace, thus just need to use the right entry points.
These kinds of changes mean that the projects will be able to fix issues more quickly and have greater control over data that is published. It will also make it more clear for end users that they have the power to add meters and plugins under their own steam without needing to wait on the highly frictioned OpenStack process.

Pros:
- Projects can manage and maintain their own definitions
- less code in ceilometer to worry about
- less review and maintainance time for ceilo devs
Cons:
- Coordinating effort with other projects and getting them on board

https://etherpad.openstack.org/p/mitaka-telemetry-cross-project

A shared session with Zaqar and potentially other interested projects to discuss mechanisms for providing notifications to users asynchronously.

Event alarms

currently, if we have multiple event alarm evalutors, there might be some race conditions in the following scenario:
1. ceilometer-notification-agent gerated 2 events of the same type in sequence, but with different traits, the 1st one would trigger the alarm, and the 2nd one won't, it then send the 2 events to aodh alarm evaluators
2. if there're multiple aodh alarm evaluators, it's possbile that the 2nd event would be processed before the 1st one, in that case, the alarm status in the DB might not be consistent, depends on the sequence of how the events are being evaluated

https://etherpad.openstack.org/p/mitaka-telemetry-alarms

The Ceilometer contributors meetup is a informal gathering of the project contributors, with an open agenda.