Kubernetes Networking··13 min read
Your Cluster Has 5,000 Services and kube-proxy Is the Bottleneck. Welcome to the iptables Cliff.
Every Service create rewrites your entire iptables chain. At small scale you never notice. At 5,000 Services kube-proxy is at 100% CPU, Service updates take 30 seconds, and your latency p99 is in the seconds. Here is the cliff and how to fall off it.
Read post