Inherent Problems of Kubernetes
2021 Dec 13Inherent Problems of Kubernetes
I have been using Kubernetes professionally since 2018 and I have to admit it is the most challenging technology I have ever had the opportunity to work with, and at the same time, the one I am most enthusiastic about regarding the future.
As I work essentially with dockerized data pipelines on Kubernetes, from my experience I think that in the near future this will be the standard for architectural and practical reasons with the increasing use of micro/nano-batching.
However, Kubernetes is a technology that is easy to sell but hard to operate, and depending on a technology team’s maturity, using K8s is even unadvisable due to the increase in complexity.
I am currently translating an article by Anna Geller called “Why Kubernetes is becoming so popular in Data Engineering”. However, I leave here today an interesting counterpoint by Itamar Turner-Trauring called Let’s use Kubernetes! Now you have 8 problems which, in short, is summarized as:
- Multiple instances/machines
- A lot of code
- Architectural Complexity
- Operational Complexity
- Configuration Complexity
- Conceptual Complexity
- Development Complexity
- Microservices (are generally a bad idea)
I agree with all these points and can even report from my experience that it took me at least a year to understand its implications in a more holistic way. Especially the conceptual complexity part (i.e., you end up thinking about the architecture and design of your code to support K8s’s limitations and advantages) and the configuration complexity part (e.g., managing configmaps, helm charts, etc.), which makes deployment much more challenging and, without a good dose of processes and automation, can lead to minor disasters in production.
This is a great article for anyone who wants to dissuade someone from using Kubernetes in production without understanding the risks and limitations.