Many people forget about patching the worker node OS believing that it is handle by Microsoft. This is not the case.
In this article I am going to go over a very important topic, updates. When I say updates, I am referring to not only Kubernetes updates, but also the worker node OS updates. Many people forget about patching the worker node OS believing that it is handle by Microsoft. This is not the case. It is currently the customers responsibility to keep the cluster up to date. Below I will answer a question I get asked a lot: why update my cluster? I will also go into detail about what you need to update.
Why update my cluster?
Kubernetes is developed fast. A new minor release comes out every three to four months. Until recently each version was only officially supported for 9 months, with Kubernetes version 1.19 this has increased to 12 months. Within these minor releases you can expect new features and improvements and sometimes breaking changes, like API version changes, which could stop your deployment from working. Patches are released more frequently, sometimes weekly, to fix critical bugs and security vulnerabilities. There is also no Long-Term Service (LTS) release of Kubernetes.
Sounds like a nightmare and perhaps you are thinking why update if it could break things. Well for one, Support, Azure support only the last 3 versions of Kubernetes that AKS has made generally available (GA). What this means is once AKS has released a version of Kubernetes to all supported Azure regions it is classed as GA. So, say version 1.19 has gone GA for AKS, only version 1.19, 1.18, and 1.17 minor versions are supported. AKS only supports two patch versions. So, for version 1.19 you can only use the latest two patches, 1.19.3 and 19.1. 1.19.0 was supported until patch 1.19.3 came out. Hopefully that’s clear.
To recap, AKS only supports the last 3 minor versions and last 2 patch versions it has made GA. If you require support from Microsoft and are not using the supported versions, the first thing they will ask you is to update the cluster.
AKS control plane update
In AKS you have something called the control plane. This is really just the master nodes and is looked after by Microsoft, well apart from updating. There is no supported automatic way to update the AKS control plane and that’s probably a good thing. You should always test your application continues to work and is deployable on a new version before you go ahead and update your production systems.
You can update the control plane via the Azure portal or Azure CLI. Of course, it is a lot easier via the portal as it is just a drop-down list.
When performing an update to the control plane Microsoft handles everything by updating the control plane components of your cluster. Not much is known about what they do you just wait a short while and it is updated.
AKS Node update
Worker nodes on the other hand have different types of updates required. In this section I am going to talk about the Kubernetes version updates. In the next section I will talk about the node OS updates.
Like with the control plane you can use both the Azure Portal and Azure CLI to update the nodes. In AKS each node sits inside something called a node pool (think Virtual machine Scale Set). AKS supports multiple node pools too and they can be windows or Linux, just not both in the same node pool.
You can only ever upgrade a node pool to the version of the control plane or lower. So, say your control plane is running Kubernetes version 1.18.10. You can only upgrade your node pools to 1.18.10, but you could upgrade them to 1.18.8. You just will not be able to upgrade them to 1.19.3 unless you upgrade the control plane first.
When you perform an upgrade AKS will add something called a buffer node. This buffer node, normally one, but can be configurable using a feature called max surge, is created in your cluster. This buffer node is running the new version of Kubernetes. The cluster will then cordon and drain one, or more of the older nodes, depending on max surge setting, to help minimize disruption to running applications. When the older node is fully drained it will be reimaged with the latest VM image from Microsoft with the selected Kubernetes version. This reimaged node will then become the buffer node. This will continue until you have one node left, once it is fully drained it will be deleted, maintaining the existing worker node count.
This whole process can take some time depending on your workloads. Each node has a total allowed time for upgrades which is 10 minutes, so when doing the upgrade just times 10 by the number of nodes you are upgrading to ensure you have enough time.
AKS node OS updates
You now know about control plane updates and node updates for Kubernetes versions, but you still have the operating system that needs patching, just like any server you would normally run. Luckily you don’t actually need to install the patches yourself. Microsoft takes care of this, for Linux nodes at least. All you have to do is reboot the Linux node for the updates to take effect. Windows on the other hand is a bit different. Let us take a look at Linux nodes first.
The Linux nodes are configured to check for updates every night. If a security or kernel update is available, it will automatically download and install the update. Some of these updates, like kernel updates will require a reboot. When a node requires a reboot, a file named reboot-required under /var/run/ is created. You Can create your own solution to monitor for this file or you could use an open source tool called KURED (KUbernetes Reboot Daemon) by Weaveworks. Just follow the instructions from their GitHub repo to install it on to your cluster. It even lets you do schedules and send a message to Slack or Teams.
For Windows node pools you will need to upgrade the node OS image. Each week Microsoft make available a new node image available for both Windows and Linux. You can then use the Azure CLI to upgrade the nodes in your cluster. You are able to do either the full cluster or just a node pool. Currently there is no way to automate this, hopefully it will come one day, but until then you could look at using logic apps or Azure automation to do this.
The last option you have is by updating the Kubernetes version on your node pools. As mentioned in the section above, when you upgrade Kubernetes it creates new nodes with the latest image version of your selected Kubernetes version, so you will get a new node OS image with all the security patches.
In the next few weeks, we will publish more articles related to AKS. Don’t want to miss out? Sign up here for our Intercept Insights and we’ll keep you updated with the latest articles.
Sign up here for our Intercept Insights and we’ll keep you updated with the latest articles.
Vist our AKS workshop
Learn even more about AKS through our interactive AKS workshop. In 1.5 hours you will receive the benefits and best practices to make your environment more efficient. Through common AKS challenges you will be ready for AKS. Click here for dates and register!
Tags
Azure Kubernetes Services
Containers
Possibly interesting as well:
Windows containers in a Linux world
A common question when someone is looking into transforming / containerizing their solution is “I’m now running on a Windows based system; do I use Windows containers, or do I go with Linux containers?”. And that is a very valid question.
Kubernetes is “hot & happening”, almost every company already uses or wants to use Kubernetes. Creating an AKS-cluster seems easy enough, but what if you want to create an AKS-cluster for serious production workloads? In this article we will give you some pointers and best-practices which will help you create an AKS-cluster ready for production environments.
The evolution of AKS: from Containers to Kubernetes to Azure Kubernetes Service (AKS)
For some time now [Azure] Kubernetes is thé platform to orchestrate containers. Why is this and what are the benefits of using Kubernetes and managed Kubernetes (AKS)?
One of the motivations for using containers and Kubernetes I come across are “We want to be multi-cloud”. But how easy is it to achieve a hybrid or multi-cloud scenario?
This e-book gives you all the information you need to decide if Azure Kubernetes is a good match for your software architecture. We also tell you more about our best practices on security, update scenarios and monitoring and managing of AKS cluster (and many more!).
Many people talk about Microservices and Service Mesch, but what is this? In this article we'll look at what Microservices are, what a Service Mesh is and what you can do with it, and finally whether you really need it now.
You have a deadline and promised your customers to launch your new platform on a specific date, everyone is working hard and you’re barely meeting your target date. And then someone asks.. “What about security?”
Kubernetes is “hot & happening”, almost every company already uses or wants to use Kubernetes. Creating an AKS-cluster seems easy enough, but what if you want to create an AKS-cluster for serious production workloads? In this article we will give you some pointers and best-practices which will help you create an AKS-cluster ready for production environments.
