Not mentioned in this writeup, is the added complexity of not using IPv6 inside.
IPv6 has an on-demand, probabalistic-unique unroutable address assignment model called ULA and it was made for something like this. Instead of NAT and the like, this framework could have been using semantic address models in the bits of the IPv6 address space, and delivered V6 to the edge.
But they went v4 only, and acquired all the growing pains of making that work.
If Kubernetes were IPv6 it would just add complexity for newcomers. Its adoption is actually impressive, but still not really something a junior dev or admin knows. And they do know basics of IPv4.
The whole article is kind of nice, but I don't think Kubernetes itself is that hard. It's large, but after initial shock it's also very logical and well documented.
I think people are afraid of k8s because it forces them to actually implement concepts that they ignored and neglected. Suddenly it occurs that they did not understand how storage works (and k8s has nothing new to offer). Or what their network admins were doing when they set up their "VIP" addresses.
Watched a large webinar on "kubernetes" just yesterday. It was aimed at developers, had a huge positive response. Presenter basically did not say more then 2-3 sentences on Kubernetes itself. It was all regular concepts that were with us all the time. State/stateless, synchronous/async, tracing, monitoring. It even ignored storage completely yet still people were shocked "how much usefull kubernetes they learned".
Kubernetes exposes how much we knew about system architecture, as our poor designs are exposed quickly.
There's a tendency in common distributions to pick complex base components to support seldomly used features at the expense of turning K8s from understandable to magic.
I can tell you the main modes of operation and how everything fits together with flannel. Ask me to compare Canal to Calico or explain Cilium and I have no idea anymore. These systems also don't explain themselves well either. I have to dig into the code and docs pretty deeply to see it's BGP and eBPF, two technologies I wouldn't expect your average network-literate admin to know.
Istio has lots of support for retrying and recovering from network errors, which is fortunate because adding it to our infra introduced a lot of transient network errors
I think that's the other thing that's understated too because a lot of these underlying choose-your-own-adventure pieces are also in wildly different levels of maturity when it comes to observing and debug. You can spend a while pursuing one solution only to find one of the pieces simply doesn't have the functionality or it is not achievable or the big thing for me is not able to be secured in any meaningful way or it's a TODO with no sign of completion.
That's still the main thing for me that gives me pause, it ends up being very unreliable when you are trying to secure it or simply put walls up around things. Finding out that your Network plugin doesn't support the thing you need is often not obvious until you have put in a massive amount of resources and work and sometimes it's not reconcilable. The idea of k8s is good, and the model is good, but the observability of features and working through the operational costs are still really raw right now and still has a lot of room for growth.
I think this complexity is part of the business model. You either pay the cloud vendor to solve this issue, or you pay enterprise vendors for storage/network licenses, etc.
J2EE was similar, from what I saw. In theory, you could run the open source stuff. In practice, IBM and Oracle made $$$ selling and supporting a proprietary version that supposedly worked out of the box.
It is the whole ecosystem around it that is complex, especially as there are no 2 setups done the same way. Also each third party operator has its own set of pros/cons/limitations/learning curve.
This is the reason when you follow a tutorial about how to publish application X using helm on your local kubernetes minikube/k3s whatever setup on your own laptop usually works fine but just doesn't necessarily translate quickly to deploy a production ready working app of your own on your company's managed k8s setup.
It's very specific to the company/environment you are in. Plain Kubernetes can be clean and pleasure to work with until you hit level that "sascha_sl" is describing. On conferences I've met a few guys who considered themselves "junior/mid admins" and did all infra (k8s based) AND desktop support for devs in a 40-50 people software houses. Sounds crazy, but some people just don't complicate their lives, like larger companies love to.
Some make it a silo and a complex, huge piece of internal IaaS, some just do "fly my dear bird" like on the popular comic [1]
You can manage your cluster externally and have an k8s API ready for you with (some of...) popular clouds with barely any knowledge, including ingress on a public facing ip and a DNS pointing to it. A junior can surely do it. "Does it have backups? YES. So it's production ready".
It's "How to implement microservice applications on Kubernetes". It's a cool run through many topics, but it barely touches Kubernetes or even clear microservices as I understand it. I find those topics quite universal, thus I chose that example.
Kubernetes isn't IPv4-only. It makes very few assumptions about the network, and can easily be configured to use IPv6 (either directly or in an overlay).
Also you don't have to use NAT even with IPv4, it's just that most CNIs implement Kubernetes networking with non-addressable pod addresses. It's perfectly possible to make service and pod addresses routable from outside the cluster.
> Also you don't have to use NAT even with IPv4, it's just that most CNIs implement Kubernetes networking with non-addressable pod addresses. It's perfectly possible to make service and pod addresses routable from outside the cluster.
This is the default on Google Cloud now. Also an option on Azure.
At first I found it to be a strange decision, but then I realized the beauty of it, having pod as first class citizens of the cloud.
> But they went v4 only, and acquired all the growing pains of making that work.
Because the major clouds didn't speak IPv6 well enough, back in the time - Kubernetes dates back to 2014 after all. And from what I've read on various threads here on HN and on Reddit, all three major clouds still have weird bugs and IPv4-only services...
Might be for ease of use (fitting MAC addresses with spare bits) and/or security, like residential subnets are supposed to be one (/64) suffix each (with from /56 to /48 per customer).
IPv6 has an on-demand, probabalistic-unique unroutable address assignment model called ULA and it was made for something like this. Instead of NAT and the like, this framework could have been using semantic address models in the bits of the IPv6 address space, and delivered V6 to the edge.
But they went v4 only, and acquired all the growing pains of making that work.