Before deploying workloads (pods) onto Kubernetes, it is important to know what can and needs to be configured in order to not suffer downtime or unreliable performance.

Kubernetes’ Pod concept offers several handy abstractions utilizing cloud’s basic building blocks, such as zero downtime deployments of containers using loadbalancing and update strategies. It also offers configuration for defining the resource usage of your workload, allowing it to allocate and scale out underlying compute resources (node pools). The ability to configure a Pod’s node pool “affinity” guarantees containers will be scheduled to run on the node pool with the best fitting hardware specs. When planned and configured well, these features can not only minimize operational cost, but also ensure optimal workload “fit”. From a security perspective you should also “know your workload” and make sure that verified images are used. 

Let’s zoom into these concepts to better understand what is involved and expected. I recommend working on them in the order of presentation, as each next step will benefit from the learnings of the previous. Note that we took a stateless workload as an example, as we will address state in another article later down the line.

Resource allocation

Before you can utilize the resources on the cluster optimally, you should determine the behavior of your workloads over time. You should know or predict how it will behave when idling, peaking or during office hours, etc. Workloads may be configured to get guaranteed cpu and memory resources, which will be used to calculate scaling of the underlying node pool. It is advisable to give enough memory to be sure the workload starts, but also to be sure it does not run out of memory (OOM). Continuously monitoring resources and learning how workloads behave should be part of a feedback loop to get control over, finetune them and bring costs down.

Learning the optimal requests and limits settings for a container can be done by observing its behavior during load testing. You could then choose the least restrictive to start with.

Autoscaling

Now that you have configured the resource usage of the container, you can choose how you want to autoscale them. There are two ways to do so: Vertical and Horizontal Pod Autoscaler (VPA & HPA). Each has their advantages and limitations. The VPA analyzes the memory and cpu needs over time, and automatically fine tunes requests and limits accordingly, whilst staying under the max that the node has available. It is generally considered to be the best solution for situations where:

• Containers can’t be scaled horizontally: it might be a stateful or legacy app
• Resources are limited (fixed node size cluster) and scaling out is not an option

If a container can be scaled horizontally and the cluster autoscaler can add more nodes to accommodate placement of x copies, it is recommended to use the HPA. It also offers some extra advantages like placement across failure zones, which adds to the resiliency of your application.

Node pools

A cluster can have one or more node pools, where each is a group of identical nodes as defined by a NodeConfig. This allows for creating different pools that can accommodate  your workload types. By now you should know that some of your workloads need more cpu, others more ram, while worker jobs may land on Spot VMs to minimize cost. When node pools exist to satisfy those needs you can be sure the workloads won’t behave erratically. If considered up front, fitting node pool configurations are likely to lead to less friction down the line, because future behavior is anticipated and planned for. Fortunately it is quite easy to add and label node pools to running clusters and relabel workloads’ node affinity to get them scheduled there.

Artifact attestation

Every time that a container is deployed, it uses its image as binary to start from. And because any sane networking architecture should assume intrusion as inevitable, kubernetes should check whether the image has been tampered with. It can do so by verifying image signatures are still the same as those they were registered with. Unfortunately this process is not yet part of core Kubernetes, but is well documented.

One solution is to use cosign for signing and verification with image registries. When you make this process part of a policy controller automatic verification happens just in time for every workload deployed.

Conclusion

By now you have a firm grasp of your workload from a Kubernetes perspective. You may think that managing these processes may be well suited for AI, and this is correct. We recommend you keep an eye out for smart automation emerging in this area, as a lot is currently being built and tested and even offered commercially (albeit expensive perhaps).

Of course you should also know an app’s intended behavior from a business perspective, so be sure to generate the metrics and logs to observe over time so you can react to changes. But that is all up to you to implement, and beyond the scope of this article.

‍