Kubernetes Networking — CNI, Pod-to-Pod, and DNS Deep Dive

Your app works locally, but the moment you deploy to Kubernetes, the frontend cannot reach the backend. You run curl from inside a pod and get nothing. DNS resolution fails. Network policies silently drop traffic. Kubernetes networking is powerful, but if you do not understand the fundamentals, you will spend hours debugging what should be a five-minute fix.

The Kubernetes Networking Model

Kubernetes enforces three fundamental rules that every networking implementation must follow:

1. Every pod gets its own IP address — no NAT between pods
2. Pods on any node can communicate with pods on any other node — without NAT
3. Agents on a node (kubelet, kube-proxy) can communicate with all pods on that node

This means that from a pod's perspective, the network is flat. A pod on Node A can talk directly to a pod on Node B using its IP address, just like two machines on the same LAN.

# Every pod gets a unique IP
kubectl get pods -o wide
# NAME       READY   STATUS    IP            NODE
# frontend   1/1     Running   10.244.1.15   node-1
# backend    1/1     Running   10.244.2.23   node-2

# Pod-to-pod communication works directly
kubectl exec frontend -- curl http://10.244.2.23:8080/health

CNI Plugins — The Network Plumbing

CNI (Container Network Interface) plugins are responsible for assigning IPs to pods and setting up the network routes between nodes. Choosing a CNI plugin is one of the first decisions you make when building a cluster.

Plugin	Networking	Network Policy	Encryption	Best For
Calico	BGP, VXLAN, IPIP	Yes (full)	WireGuard	Production clusters, enterprise
Cilium	eBPF	Yes (L3-L7)	WireGuard, IPsec	Observability, performance, security
Flannel	VXLAN	No	No	Simple clusters, learning
Weave Net	VXLAN, sleeve	Yes	IPsec	Multi-cloud, ease of setup
AWS VPC CNI	Native VPC	Yes (with Calico)	VPC encryption	EKS clusters
Azure CNI	Native VNet	Yes	VNet encryption	AKS clusters

Flannel is the simplest to set up but has no NetworkPolicy support. Calico and Cilium are the top choices for production. If you care about eBPF-based observability and L7 policies, go with Cilium.

# Check which CNI is installed
kubectl get pods -n kube-system | grep -E 'calico|cilium|flannel|weave'

# Calico example output:
# calico-node-xxxxx    1/1     Running
# calico-kube-controllers-xxxxx    1/1     Running

Service Networking and kube-proxy

While pods talk to each other directly, Services provide a stable virtual IP (ClusterIP) that load-balances across backend pods. kube-proxy makes this work.

kube-proxy runs on every node and programs the data plane rules that redirect Service IPs to actual pod IPs:

Mode	Mechanism	Performance	Connection Tracking
iptables (default)	iptables rules	Good for < 5,000 services	Yes
IPVS	Linux IPVS kernel module	Better for 5,000+ services	Yes
nftables	nftables rules (K8s 1.29+)	Modern replacement for iptables	Yes
eBPF (Cilium)	eBPF programs, bypasses kube-proxy	Best performance	Yes

# Check kube-proxy mode
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

# See the iptables rules kube-proxy creates (on a node)
iptables -t nat -L KUBE-SERVICES -n | head -20

# View Service endpoints
kubectl get endpoints my-service
# NAME         ENDPOINTS
# my-service   10.244.1.15:8080,10.244.2.23:8080,10.244.3.44:8080

CoreDNS and Service Discovery

CoreDNS runs as a Deployment in kube-system and resolves DNS names to Service ClusterIPs. This is how pods discover each other without hardcoding IPs.

DNS Naming Conventions

# Service DNS format:
# <service-name>.<namespace>.svc.cluster.local

# From any pod, these all resolve the same Service:
kubectl exec frontend -- nslookup backend-api.production.svc.cluster.local
kubectl exec frontend -- nslookup backend-api.production.svc
kubectl exec frontend -- nslookup backend-api.production

# If the pod is in the SAME namespace, just use the service name:
kubectl exec frontend -- nslookup backend-api

# Headless Service DNS (returns individual pod IPs, not ClusterIP):
# <pod-name>.<service-name>.<namespace>.svc.cluster.local
kubectl exec frontend -- nslookup postgres-0.postgres-headless.production.svc.cluster.local

Inspecting CoreDNS

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# View CoreDNS config
kubectl get configmap coredns -n kube-system -o yaml

# Check DNS resolution from inside a pod
kubectl run dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 --rm -it -- nslookup kubernetes.default.svc.cluster.local

# Check /etc/resolv.conf inside a pod (shows which DNS server pods use)
kubectl exec frontend -- cat /etc/resolv.conf
# nameserver 10.96.0.10          <- CoreDNS ClusterIP
# search production.svc.cluster.local svc.cluster.local cluster.local
# ndots:5

The ndots:5 setting means any name with fewer than 5 dots gets the search domains appended first. This is why backend-api (0 dots) gets resolved as backend-api.production.svc.cluster.local.

NetworkPolicy for Microsegmentation

By default, every pod can talk to every other pod. NetworkPolicy lets you restrict which pods can communicate — essentially a firewall for your cluster.

Default Deny All Ingress

Start by denying all incoming traffic, then allow specific paths:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}    # Applies to ALL pods in the namespace
  policyTypes:
    - Ingress

Allow Frontend to Backend Only

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend-api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Allow Backend to Database Only

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-to-database
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: backend-api
      ports:
        - protocol: TCP
          port: 5432

Deny All Egress Except DNS

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-egress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend-api
  policyTypes:
    - Egress
  egress:
    # Allow DNS
    - to: []
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # Allow traffic to database pods
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432

Debugging DNS Issues

DNS is the number one networking issue in Kubernetes. Here is a systematic troubleshooting approach:

# 1. Is CoreDNS running?
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

# 2. Can the pod resolve DNS?
kubectl exec frontend -- nslookup kubernetes.default.svc.cluster.local
kubectl exec frontend -- nslookup google.com    # External DNS

# 3. Is the DNS Service reachable?
kubectl get svc -n kube-system kube-dns
kubectl exec frontend -- nc -zv 10.96.0.10 53

# 4. Run a dedicated DNS debugging pod
kubectl run dnstest --image=busybox:1.36 --rm -it -- sh
# Inside the pod:
nslookup backend-api.production.svc.cluster.local
nslookup google.com
cat /etc/resolv.conf

# 5. Check if NetworkPolicy is blocking DNS (port 53)
kubectl get networkpolicy -n production

Network Troubleshooting Tools

When DNS is fine but connectivity is still broken, bring in the heavy tools:

# nicolaka/netshoot — the Swiss Army knife for network debugging
kubectl run netshoot --image=nicolaka/netshoot --rm -it -- bash

# Inside netshoot:
curl http://backend-api.production.svc:8080/health
tcpdump -i eth0 port 8080 -nn
traceroute backend-api.production.svc
ip route
iptables -t nat -L -n | head -30

# Test connectivity from the host node (via debug container)
kubectl debug node/node-1 -it --image=nicolaka/netshoot
# Inside:
nsenter -t 1 -n -- ss -tlnp
nsenter -t 1 -n -- iptables -t nat -L KUBE-SERVICES -n | head

---

Next, we will cover health probes — liveness, readiness, and startup probes that tell Kubernetes when your pods are actually healthy and ready to serve traffic.