Highlights of OpenStack PTG, Denver 

The OpenStack foundation organizes “Project Teams Gathering” every six months, mostly in Q1 and Q3, at the beginning of the development phase of a release cycle. Previously known as the “Design Summit”, the main motive for this event is to discuss the upcoming cycle, solutions for complex issues, and innovative solutions to critical items. I was privileged to attend the Denver PTG this September.

It was my first time joining the Project Teams Gathering (PTG) since it started in Atlanta. The location of the event was pretty unique in its own way. I had never been to this part of the USA, and I could feel the difference. The event was held in Renaissance Denver for five days between September 11 and 15. I arrived here on Sep. 10 and I could already see the active contributors in the hotel lobby discussing stuff related to the latest technology trends in the space. I met some of my friends and took a rest for the rest of the day.

Day-1, September 11

The day started with registration for the event. I joined the openstack-ptg channel to get updates about the day, and there I was introduced to ptgbot. Initially, many people, including me, were a bit confused with how it works, but as we got familiar with it, we got more used to it for tracking events. As per schedule, the first two days of the event were dedicated to inter-project discussions.

I headed directly to the infra/stable/release/requirements room for discussions with the requirements team. We had a discussion around topics to be worked on in Queens, which included the per-project/independent/divergent requirements, OpenStack client testing, and Python 3. The discussion was pretty good with insights provided by Tony Breeds, Matthew Thode, Dirk Mueller, Monty Taylor, and John Dickinson. Post lunch, I joined Kolla team with discussions around collaboration across different deployment tooling in OpenStack. We had discussions around architecture, health monitoring, the role of containers, Kubernetes, and security. I also attended the Technical Committee (TC) meeting for Rebooting the Stewardship WG and Onboarding new community members.

The day ended with an unofficial PTG happy hour at the elevated lounge in Renaissance Denver.

Day-2, September 12

The day started with meeting people at the registration desk hallway. I joined the TC discussion for Q & A around new projects in OpenStack. The discussion included emerging projects like Blazar, Glare, Gluon, Masakari, Stackube, Cyborg, and Mogan. Each project representative presented TC members with a general project overview, objectives, and current status and tried to answer any queries related to the Requirements for new OpenStack Projects applications. TC members also asked for details related to the maturity of projects with respect to contributor’s diversity, the potential to increase collaboration, and identifying any overlap with current/existing projects in OpenStack. We also had a discussion about potential project removals from official OpenStack projects. This is due to very low contributor activity during the cycle, no project goals achieved, or changes in the focus of the participating organizations. The particular projects highlighted in the area are Searchlight, Solum, Designate, CloudKitty.

Michal Jastrzębski and I were chasing the infra/TC members for a resolution related to publishing kolla images on the public registry like Quay with OpenStack-kolla namespace. We did not get any direct solution, and I think there is a need to initiate a discussion on openstack-dev mailing list for further discussion.

I met with Saad Zaher (Project Technical Lead) of Freezer regarding the current status of the project. I had a very informative discussion and got a clear idea of things to do.

We had a few hallway discussions related to kolla-kubernetes, openstack-helm, kolla with Ironic, and kolla with neutron.

In the meantime, we also had team photos clicked for Kolla and the requirements team. This was the end of the first section of the schedule of inter-project discussions. The day ended with IBM sponsored happy hour in the hotel, where random discussions happened regarding projects, OpenStack, etc.

Day-3, September 13

Day 3 of Queens PTG started with project-specific design discussions; I joined the Kolla team, where we started off with the design summits we had and a very important thing for the community, the "Documentation." We broke down the discussion in the documentation for quick-start with Kolla contributors, operators, and reference documentation. The documentation available currently is scattered across projects after the project split, and it’s essential to have a common landing page on OpenStack Deployment guides that everyone can refer to. We had representatives from the Documentation team Alexandra Settle, Doug Hellmann, and Petr Kovář, who are working on improving the doc experience by migrating the docs to a common format across the community. They understood the problem Kolla is facing and we had a discussion where we created the table of contents for all available and required documentation for Kolla.

Kolla team then joined the TripleO team which is consuming the kolla images for OpenStack deployment for a discussion about collaboration of efforts. The teams will work together to improve the build and publish pipeline for kolla images, improving & adding more CI jobs for the kolla/kolla-ansible/kolla-kubernetes, configuration management post-deployment of containers. The tripleo team has come up with basic health-checks for containerized deployment; the Kolla team will help get those checks in the current kolla images and improve on those to better monitor the containerized OpenStack deployment. The teams will also collaborate on improving the orchestration steps, container testing, upgrades, and creating metrics for OpenStack deployment.

During lunch, we had an extended discussion with Lars Kellogg-Stedman and Kevin Fox about “Monitoring OpenStack with Prometheus and other monitoring tools.”

Post lunch, Kolla team started with key discussion to the heart of operators, the OpenStack plugins deployment with Kolla. There are multiple issues currently related to the plugin, like when would be an ideal time to make them available, during build/deployment, non-matching dependencies to OpenStack components for plugins, etc. The team came up with multiple permutations of options available which would need to be PoCed during the release.

Since the inception of project loci, there has been discussion around kolla-images size, and the team had an interesting discussion on how to reduce that. The important part is to remove the things like apt/yum cache, removing the fat base image, etc. The team also discussed utilizing alternate container build tooling to write own image build tool. The team will hack on Friday, September 15, removing the fat base images and see if that improves the image size.

External tools like Ceph are common pain points when we are doing OpenStack deployment. When Kolla community evaluated the options for Ceph as storage backed for containerized OpenStack deployment, there wasn’t a thing like containerized ceph. The team built it from scratch and got it working. The ceph team has currently come up with ceph-docker and ceph-ansible. It would be useful for operators that Kolla uses the tools directly available from vendors. We had a discussion with representatives from ceph to initiate the collaboration to deprecate current ceph deployment in Kolla and use the combination of ceph-docker & ceph-ansible. It will help both the communities and will benefit exchange of things at each end.

I got a surprise gift of vintage OpenStack swag from the PTG team

The day ended with hanging out with Kolla teammates at “Famous Dave's.”

Day-4, September 14

Day 4 of PTG started with the next Kolla discussions related to kolla-ansible. Discussion started with kolla dev-mode effort by Paul Bourke. The discussion was about current missing pieces in dev_mode like installing clients, libs, and virtualenv bindmount. The goal of the cycle was to fill in the missing pieces, verify options for multi-node dev_mode, investigate options for remote debugging, and also consider using PyCharm.

One of the important topics in Kolla is gating. Currently, Kolla has around 14 different gates for deployment testing, and it has to be improved by testing the deployment for sanity with Tempest. This will help to validate the entire deployment in the gates. Upgrade testing is also one key requirement, Kolla team will model something like grenade testing for it. The key is to maximize the testing of scenarios that Kolla supports in gate, but since we are restricted with openstack infra resources as well as the time each test takes to validate. It was agreed that team members would create a list of scenarios and assign them to everyone to verify and record the results in a central location. This will also help to evaluate the stability of Kolla deployment in each release.

Skip-level upgrades were one of the major talking points in the current PTG. Kolla team will evaluate fast-forward upgrades for each service deployed with Kolla to decide on skip-level upgrade support in Kolla. This would be a PoC in the current cycle.

The second half of the discussion was around the kolla-kubernetes, where the team discussed the roadmap for the current cycle. That will include upgrade prototyping for z stream & x stream services, validating the logging solution with fluent-bit, automated deployment, removing deprecated components, and improving documentation.

Most of the teams had wrapped up their design discussions on Thursday and were gearing up for “the hackathons” on the last day.

Day-5, September 15

Day 5 of PTG started as the day for hackathons and general project/cross-project discussions. Most project teams were preparing for their travel plans or site-seeing in Colorado. The Kolla team started the day with an alternate Dockerfile build tool review. Later on the day, there was something everyone in OpenStack and the containers community was looking forward to - the OpenStack - Kubernets SIG with Chris Hodge. Some key targets for the release were identified, including contributors interested. We then had the most pressing issue for all deployment projects based on containers - the build and publishing pipeline for Kolla images with openstack-infra team. Most of the current requirements, needs, and blocking points were identified for rolling this feature. The Kolla team and OpenStack infra team will work together to get this rolling in the starting phase of this cycle once zuul v3 rollout becomes stable. The Kolla team ended the day early for some much needed buzz for the whole week's work at “Station 26”.

It was a very insightful session this time around in Denver. And with the kind of work Opcito is coming up within OpenStack space, I am already looking forward to the Dublin edition of OpenStack PTG.