Openstack Summit / Opendevconf 2018

Vancouver in May 2018

Vancouver Harbour

I attended Openstack Summit in Vancouver, BC, Canada on May 21st - 23rd. Mostly my interest was in attending the colocated Opendev event that was focused on CI/CD.

As a contributer to the CentOS and Fedora projects and as someone who maintains CI systems for my job, I was interested in hearing about how large scale projects like Openstack coordinate test strategies, and communicate results across all of the different subprojects.

OpenStack Summit Day 1 (May 21st)

I spent most of the first day wandering around Openstack Summit, meeting some of the infra team members and attending sessions. Some highlights from Summit overall, since I spend the rest of the week at OpenDev:

  • Zuul v3 made its debut as a toplevel project in the Openstack Foundation

  • There was a rather controversial Keynote sponsored by Canonical, that sparked some discussion about the relationship amongst vendors at project-focused events.

OpenDev Day 2 (May 22nd)

Using CI in an Ecosystem link

This was an open discussion about how to do tests on components of a "release" but which are themselves individual projects. This is a common pattern for Linux distributions that take in software projects from 1000s of sources and package them individually for inclusion in the distribution. Openstack has similar problems because of its dependence on upstream python projects, that have separate lifecycles and goals.

We found that lots of projects are tackling this in a similar pattern:

  1. Generate a graph of dependencies
  2. Run as many tests as you can from the component projects whenever something changes
  3. Have integration tests that run separately
  4. Promote the components from testing -> known good

Cross-Community CI/CD link

This session was mostly focused on helping individual projects under the Openstack foundation talk to each other. Openstack, and the peripheral projects didn't have a good forum to talk to one another, and share best practices between folks who are on Openstack's Zuul, and folks using Jenkins-based solutions from the Linux Foundation and other providers.

We decided to collaborate using the OpenCI "Community of Practice".

Ansible Community Meetup link

We began the Ansible community meetup with a demo from dmsimard about ARA, which is an ansible plugin that presents a friendly web interface describing ansible runs. ARA is something I've been meaning to investigate ARA for my own projects and seeing a live demo makes it that much easier.

We spent the last half of the meetup talking about scaling up, many of the participants were managing 10s of thousands of of machines and looking for management strategies. The most common way to deal with this was to use ansible-pull, but that requires careful management of secrets.

Day 2 Summary

If I had to summarize day 1 in a sentence it would be this: Openstack acts like a Linux Distribution, we can learn from one another.

Participants in each of the discussion sessions, and in the hallway track, were primarily interested in testing and delivering Openstack, but they all used experience taken from their roles as participants in the distribution ecosystems (Debian, Ubuntu, Fedora, CentOS, Mageia, etc.) as lessons.

OpenDev Day 3 (May 23rd)

OpenCI: What, Why, How? link

OpenCI is a non-governed, non-product community of people working generically on the CI/CD problem for integration projects. Key members come from the NFV, and Openstack communities and are looking for a forum to discuss best practices and collaborate/agree on standard methods of communication between projects that care about each-others' CI results.

Operationally the folks in the room were looking at methods and a specification for triggering CI runs in a separate infrastructure based on their dependencies. For example, if the OpenDaylight project wants to run its tests once a 'good' test of a component in OPNFV completes, how do we do that?

One solution that we brought from the distro space is to wrap an existing proposed message specification and deliver it via Fedmsg.

Fedmsg is used in the Fedora and Debian projects, and it would make sense for openci to occupy the io.openci namespace so that others can subscribe to topics from various infrastructures they're interested in.

We talked about strategies to include and rebroadcast event streams from existing infrastructures as well. For example, we have a Fedmsg structure emitted from the Fedora CI Pipelines; it could be interesting to also rebroadcast those events using Fedmsg so projects that rely on Fedora or CentOS as distributions could consume those events in their own format.

Productizing Open Source Through CI link

This was a good discussion about strategies that teams (even outside openstack, and OSS projects) use to test products based on information from the people that consume the product. I'll admit the subject matter was interesting to listen in on, but this was not part of my focus for attending the event.

Day 3 Summary

1 Sentence: Because the Openstack family and distributions are so similar (see day 2's summary), we can collaborate on tools too!

One action I'd like to take from day 3 is to continue participating in the OpenCI discussions on the mailing lists. It seemed that Fedmsg was interesting as a touchpoint for a group of engineers who are desperate for a single event stream that they can act on in their own infrastructures. Luckily there aren't many changes/updates/features needed in Fedmsg to get started quickly

My "To Watch" List

It seems like no matter what conference you go to, there's always a choice you have to make between 2 sessions scheduled at the same time. Luckily we have the Internet for this, here's what I would have gone to if I was able to clone myself:

Copyright © 2012-2015 Brian Stinson. All Rights Reserved