A presentation at German Testing Day in in Munich, Germany by Max Körbächer
How to Release a new Kubernetes Version Max Körbächer
Max Körbächer Co-Founder & Manager Platform Engineering @ Liquid Reply Part of the Kubernetes Release Team & Release Engineering since v1.17 mkoerbaecher mkoerbi
Next Stops 01 Kubernetes A project, a community, a whole universe 04 From the Merge into main 7060 tests later 02 From a Commit to a Release 03 The First PR Overview of the release process System side tests 05 06 The Release Lessons Learned beta.1, RC.1, a new major version What you can take away from the K8s release!
01 Kubernetes
Kubernetes Kubernetes supports the deployment and operation of applications with a micro service architecture. This is done through an abstraction layer across a group of nodes which handles containers. Developers can deploy their containerized applications towards K8s while K8s takes care about basic operations and keep alive & scaling mechanisms. Source: https://newrelic.com/blog/how-to-relic/what-is-kubernetes
Kubernetes Projekt
02 From Commit
Release Cycle an d eC The cycle ends with an Enhancement Freeze Re le This cycle ends with the major Release . SIGs have to opt-in their to be released feature as Within the final cycle the new release getting prepared, a new release branch gets created and required jobs will be implemented. Enhancement Cycle - 3 Weeks ida te Release Preparation & Testing - 2 Weeks se s The cycle ends with a Test Freeze Re l ph a ea Final tests needs to be adjusted and implemented. l Re Beta Testing - 1 Weeks Development Cycle - 10 Weeks Al ta Be ea se s Ca. 4 month per release Within 10 weeks the SIGs have to merge the KEPS into kubernetes/kubernetes The cycle ends with an Code Freeze
03 The First
From development to PR The development happens locally, all test can and must be executed once before committed. As soon as the developer decides to move on with an PR, PR pre-submit tests and checks are automatically executed. K8s test infrastructure: https://github.com/kubernetes/test-infra
Taking over a commit into the planned release If a commit is merged into the main branch, it gets labeled with a milestone. Milestones create a mapping between the opt-in KEP and the truly implemented feature.
A KEP can be one or many PR KEPs are not obviously visible. That’s also caused by the fact that a KEP normally consist of multiple PRs. In an enterprise this maybe comes close to an Epic. KEP PR 1 In addition to KEPs, we also handle fixes, patches, tests and cherry picks, which are integrated through an fast forward push. PR 2 PR 3 Fix 2.1 Security Patch 3.1
04 After the
Continuous Testing As soon as the PR is closed and the code is within the main branch - it will get tested within the test infrastructure. Test are executed within different cycles, but at least once per day. Tesgrid: https://testgrid.k8s.io/sig-release-master-blocking
Testgrid Tesgrid is a system that continuously scrapes the logs of Prow (CI/CD) and shows the results. Tests are running specifically for the various 3 major versions, the master branch and are differentiated between informing and blocking. Each branch has therefore two tabs, within each tab are multiple jobs, clustering Branch Job Test Test Test sometimes hundreds of tests. 7044 tests at the moment!
Testgrid Testgrid is a pool of information: ● ● ● ● ● Which test was executed when? Was the test successful or did it fail? Is it a flaky test? After which changes did the failure happen? Build number, K8s code commit hash, infrastructure hash Each box links to the prow & spyglass dashboard for exactly this test
A short tour through Testgrid
Tests are Multi-dimensional The test is executed in different jobs. Therefore, a test can be seen multi-dimensional. That’s because a job clusters a functionality, while for a functionality multiple components are required. Testgrid is job oriented, to see the test perspective we have Triage tool in which you can research for the different test. Example: Scaling Test - executed at the Replica & load test, as well as for different rollout strategies. Here you can play around: https://storage.googleapis.com/k8s-triage/index.html
05 The
Preparation and Execution Documentation Branching Split off of the release branche & e2e test of the new release version Publishing release notes, K8s documentation and communication are prepared Release Cut A Release Manager runs the krel tooling to create a release
A own set of tools was developed for the release cut Kubernetes release Toolbox - KREL Krel is the central and critical component for the release cuts. The release cut is the step where within kubernetes/kubernetes main Branch a new release is created and packaged.
The Release Process User perspective
Adding some Noise Canceling
You need to know what you are looking for
Every build runs in the cloud
Lessons 06
Take Away Strong Automation Every single step within the release is automated and tested Never Trust a Commit “Everyone” can commit, every commit & PR is tested and reviewed Without a Release Team it will not work Even with a high automation you need someone to ensure the quality and drive further development Own Processes … require sometimes own tools Tests must be consistent Sometime a commit is just a few lines long, sometimes they are tens of thousands of lines Did I mention Atomization?
Thank you! Max Körbächer - Co-Founder CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik
for free. You
can too.