Recently I have been setting up a Kubernetes on Hetzner Cloud. The reason of choosing Hetzner Cloud in this instance was purely an economical one, plus the service is good and I have never had any major issues.
One of the most critical aspects of running a cost efficient Kubernetes cluster is to have autoscaling of your cluster nodes. Whats the point in running nodes that have no running pods? It’s just a waste of money and resources. This is where using Kubernetes Cluster Autoscaler comes in.
Cluster Autoscaler monitors pod scheduling to decide if there is enough capacity on your cluster to schedule the pods on a node. If there is not enough capacity to run the pods it will call the Hetzner Cloud API and launch new instances and add them to your cluster. When the pods are no longer needed and are removed from the cluster, Cluster Autoscaler will remove the Kubernetes nodes which are no longer needed and delete the associated instance.
One of the really useful features on Cluster Autoscaler is the ability to have different pools. Pools are distinct sets of nodes, you can chose to have nodes of different instance types, different regions or even have pools which schedule nodes of the same type in the same region but distinctly group them so you can schedule specific pods on specific pools.
This is where the fun started for me, how to schedule specific pods onto specific pools? Having this ability was important for me as I have some workloads which are heavy and also have some security concerns, I did not want these pods running alongside sensitive pods on the same node just incase an attacker was able to escape from the container.
First of all I thought that to schedule the pods on a specific node pool each instance in the node pool would need a label to specify which node pool it belongs to. However the nodes had no such node pool specific labels, so how the heck do we schedule pods on specific node pools? Read on for the explanation.
How to schedule pods on specific node pools
To schedule pods on a specific node pool with Cluster Autoscaler on Hetzner is actually really simple using the following nodeAffinity example.
Note within nodeSelectorTerms, the key being used hcloud/node-group and the value being a list of node pool names to schedule on, in this case I used pool1 as an example. The node pool name comes from the –nodes flag passed to Cluster Autoscaler.
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: hcloud/node-group
operator: In
values:
- pool1
So at this point you might be wondering, without node pool specific labels on the nodes? How on earth does Cluster Autoscaler know how to schedule pods on specific node pools? Cluster Autoscaler before actually launching a real instance will first simulate what a real instance would look like, from this simulation it determines if the current scheduled pods would actually fit. You can find the block of code responsible for this process here
It took me some trial and error and code digging to finally figure out how this feature works, so I thought I would share my findings so hopefully no one else has to scratch their head for hours on end trying to figure it out.
If you found this article useful and are yet to sign up with Hetzner Cloud, please consider using my referral link to do so, by using the referral link you will get 20 euros of credit to help you start building your projects on Hetzner Cloud.
Referral Code: https://hetzner.cloud/?ref=OkdP7lCBirsn