Architecture

How Kubernetes actually works: one control loop, one request path

·Architecture·11 min read

Most introductions to Kubernetes start with tooling: a cluster, a kubectl binary, a first YAML file. That order is understandable, but it hides the one idea that makes the system coherent. Kubernetes is not a scheduler with an API attached. It is a store of desired state, surrounded by independent components that watch that store and move the real world towards it. Once that idea is clear, the rest of the architecture reads like a consequence rather than a list of parts.

This article answers two questions, and the diagram below answers both of them in one picture:

  1. How does an intention, such as "run four copies of version 2 of the checkout service", become running containers on real machines?
  2. How does a request from the internet find one of those containers, and only one that is actually ready to serve?

The first question is the control loop (steps 1 to 5 in the diagram). The second is the request path (steps A to D). Everything else in Kubernetes, from rolling updates to self-healing to GitOps, is a variation on these two paths.

Kubernetes – control loop and request path Kubernetes — control loop and request path Everything goes through the API server · components watch and reconcile · a Service is a rule set, not a process Engineer · CI · GitOps write or update the spec Deployment spec desired state, declarative (YAML) image: checkout:v2 · replicas: 4 LEGEND control flow (watch / write) request traffic logical API object, no process 1 control loop steps A request path steps KUBERNETES CLUSTER CONTROL PLANE API server the only door — every component reads and writes cluster state here 2 etcd stores cluster state · only the API server talks to it 3 Controller manager Deployment → ReplicaSet → 4 Pods 4 Scheduler binds each pending Pod to a node WORKER NODES Worker node 1 Worker node 2 Container runtime containerd, CRI-O · runs containers kubelet node agent · runs assigned Pods kubelet node agent · runs assigned Pods Container runtime containerd, CRI-O · runs containers CRI CRI REPLICASET checkout-v2 · owns 4 Pods · label app=checkout Pod A ready container · checkout:v2 Pod B ready container · checkout:v2 Pod C ready container · checkout:v2 Pod D not ready · no traffic container · checkout:v2 Service checkout virtual IP · selector app=checkout · ready Pods only Ingress controller Pod NGINX, Traefik … · applies routing rules kube-proxy programs the Service rules on this node kube-proxy programs the Service rules on this node Ingress controller Pod NGINX, Traefik … · applies routing rules Client browser · API consumer External load balancer cloud LB · public IP · TLS entry Ingress and Gateway API are two separate APIs; both need a controller running as Pods. Many controllers skip the virtual IP and send straight to ready Pod IPs (EndpointSlices). 1 kubectl apply · GitOps sync 5 watch · report status 5 same on every node A B C C D AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTplYjBhNGY0ZS04Mzk5LTQ1OTAtYTQ2Mi01ZGRjODk0MTRhZGUAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaPhYt09vXLDJBoBhDCk1+9cAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDphMzVmMDM5MC1hMmE4LTQ2NDQtODliOC1kODBjMDkwOTVhZWFscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNoTbcC8EX/EhgkX2J0CbKj+wAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggYArHsZEFYrC9hAssvJYVD74PJQPdd45esTDSbl50UtqkZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaEIOpmipg1oQZXvLJBGDdBUAAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRMAAAAAAAAAAAAAAAAZGhhc2hYIH3ld8Xf51QiQQb0396eqjShQ3ZQAFhCg5yVIB3kMzsaZG5hbWVuanVtYmYgbWFuaWZlc3RqZXhjbHVzaW9uc4GiZXN0YXJ0GU2AZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOmViMGE0ZjRlLTgzOTktNDU5MC1hNDYyLTVkZGM4OTQxNGFkZS9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOjQ4MmU1Y2ZhLTI0ZTktNDUwOC05OTlkLTQ2Mjc0ZTIwMjE0MXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCBgCsexkQVisL2ECyy8lhUPvg8lA913jl6xMNJuXnRS2qJjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFggtsJ8mVRucR+G1l8YOMQYr2s60+CxVyPGBMXsgicjOB+iY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFgg0IF3rkZo69r+a8nL+uGz5gRyng8ELgshEWjFkUv2PdN0Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQHExgGzJeEc4+ggT0scB5BXLR5doxesCDa5+HzsJdXpiY62XzkxgSVKcY+TCRVkep/zKk+z5jWjwTZjA+GJyt3M=

Desired state: what "declarative" really means

A Kubernetes cluster is operated by describing a goal, not by issuing a sequence of commands. The goal for our example is small:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: checkout
          image: checkout:v2

Nothing in this document says how to reach the goal. It does not name a machine, it does not say which copies to start first, and it does not say what to do when a machine disappears. It only states what should be true: four replicas of checkout:v2 should be running. This document is the desired state, and the whole cluster is organised around making the observed state match it, continuously, for as long as the document exists.

That distinction matters for everyone who plans around Kubernetes, not only for the people who type the YAML. A declarative system can be versioned, reviewed and reproduced, because the input is a description rather than a transcript of actions. It also behaves differently under failure: a system that executes a script is finished when the script ends, while a system that reconciles a desired state is never finished and keeps correcting drift.

The API server is the only door

Every arrow in the upper half of the diagram either starts or ends at the API server, and that is not a simplification for the sake of the drawing. In Kubernetes, the API server is the only component that reads from or writes to etcd, the key-value store that holds every object in the cluster. Engineers, CI pipelines and GitOps tools talk to the API server. The controller manager, the scheduler and every kubelet on every node talk to the API server as well. None of them talk to each other directly, and none of them touch etcd.

This single door is what makes a Kubernetes cluster governable. Because all changes pass through one place, that place can enforce authentication, authorisation (RBAC) and admission control uniformly, and it can write an audit log that covers every actor, human or machine. An organisation that wants to know who changed what in production, or that wants to block container images from unapproved registries, implements that once, at the API server, and it applies to every path into the cluster.

Note

etcd is deliberately boring from the outside. It is a consistent, replicated store with a watch mechanism, and that watch mechanism is what the rest of the architecture is built on. Components do not poll for changes. They subscribe through the API server and are notified when an object they care about is created, changed or deleted.

The control loop: from spec to running container

With the door established, the upper half of the diagram can be read as a sequence, even though the components involved never coordinate with each other explicitly.

Step 1: the spec is submitted. An engineer runs kubectl apply, a pipeline does the same, or a GitOps controller syncs the file from a repository. In all three cases the result is identical: the API server validates the Deployment, runs it through admission control and stores it in etcd. At this point nothing is running yet. A Deployment is a record, not a process.

Step 2: the record is persisted. etcd now holds a Deployment named checkout whose desired state says four replicas of checkout:v2. Every component that has registered a watch on Deployments is informed.

Step 3: controllers reconcile. The controller manager hosts a collection of control loops, each responsible for one kind of object. The Deployment controller notices the new Deployment and does the only thing it knows how to do: it ensures that a matching ReplicaSet exists. The ReplicaSet controller notices the new ReplicaSet and ensures that four Pod objects exist. Each controller compares desired with observed state for its own object kind and issues the smallest change that closes the gap. No controller instructs another one. They only write objects to the API server, and the next controller in the chain reacts to those writes.

This is the heart of the model, so it is worth stating plainly: a controller is a loop that watches, compares and corrects. The Deployment controller does not know how Pods are started. The ReplicaSet controller does not know which node a Pod will land on. Each loop is simple, and the system's behaviour emerges from many simple loops running side by side.

Step 4: the scheduler assigns nodes. The four new Pods exist as objects, but none of them is bound to a machine. The scheduler watches for exactly this condition, a Pod without a node, and for each one it chooses a node that satisfies the Pod's requirements: enough CPU and memory, matching labels, affinity and anti-affinity rules, taints and tolerations. It then writes that decision back to the API server as a binding. The scheduler decides, but it does not execute. It never starts a container.

Step 5: the kubelet runs the workload. On every node, a kubelet watches for Pods bound to its own node. When one appears, the kubelet pulls the image and asks the node's container runtime, such as containerd or CRI-O, to start the containers. The conversation between kubelet and runtime goes through the Container Runtime Interface (CRI), which is why Kubernetes can run on different runtimes without changing anything above the node. The kubelet then reports status back to the API server: whether the containers are running, whether they are healthy, and, as we will see below, whether they are ready to receive traffic.

Read as a whole, steps 1 to 5 show that Kubernetes never executes a plan. Each component reacts to a change in the shared state and contributes its part. If a node fails, the kubelet on that node stops reporting, the Pods are eventually marked as gone, the ReplicaSet controller sees three Pods where four are desired, and the loop runs again from step 3. Self-healing is not a feature that was added on top. It is what the control loop does when the observed state changes for any reason.

The request path: from client to a ready Pod

The lower half of the diagram answers the second question. Four Pods are now running, and a client on the internet wants to reach the checkout service.

Step A: the external load balancer. The request first arrives at a load balancer outside the cluster, typically provided by the cloud platform. It owns the public IP address, usually terminates TLS, and forwards traffic to the nodes of the cluster. Kubernetes requests this load balancer through a Service of type LoadBalancer, but the load balancer itself runs in the provider's infrastructure, not inside the cluster.

Step B: the Ingress controller. Inside the cluster, the request reaches an Ingress controller, for example NGINX or Traefik. This is a point that regularly surprises newcomers: the Ingress controller is itself an ordinary set of Pods, scheduled and supervised by the same control loop described above. It reads Ingress objects from the API server and configures its proxy accordingly, routing by hostname and path to the right Service.

Note

Ingress and the newer Gateway API are two separate Kubernetes APIs for describing how traffic enters the cluster. Neither of them does anything on its own. Both are declarations that a controller, running as Pods, must watch and implement. A cluster without such a controller accepts Ingress objects and silently routes nothing.

Step C: the Service. The Ingress controller forwards the request to the Service named checkout. A Service is the most frequently misunderstood object in Kubernetes, and the diagram's subtitle states the correction: a Service is a rule set, not a process. There is no Service daemon that receives a packet and passes it on. A Service is a stable virtual IP address plus a label selector, in this case app=checkout. A component called kube-proxy runs on every node and translates that definition into packet forwarding rules in the node's kernel, so that traffic to the virtual IP is rewritten to the IP address of one of the matching Pods. The forwarding happens in the network stack, not in a user-space process.

Which Pods qualify is tracked in EndpointSlices, and here readiness enters the picture. In the diagram, Pods A, B and C are ready and Pod D is not. Pod D exists, it is scheduled, its container may even be running, but its readiness probe has not yet succeeded. Perhaps it is still warming a cache or waiting for a database connection. As long as that is the case, Pod D is not listed as an endpoint of the Service, and it receives no traffic. This is the mechanism that makes rolling updates safe: a new Pod only starts receiving requests once it has told the kubelet that it is able to handle them.

Step D: the Pod. The request arrives at one of the ready Pods and is answered by the checkout container. The response travels back the same way.

Note

In practice, many Ingress controllers do not send traffic to the Service's virtual IP at all. They watch the EndpointSlices themselves and forward directly to the ready Pod IPs, which gives them finer control over load balancing, retries and connection reuse. The Service still matters as the declaration of which Pods belong together and which of them are ready; the controller simply reads that declaration directly instead of going through kube-proxy's rules.

What follows for architecture and operations

A mental model is only useful if it changes decisions. Four consequences of the two paths come up in almost every engagement.

Troubleshooting follows the control loop. When a workload does not come up, the cause sits at a specific step, and the steps are ordered. Does the Deployment exist and is its spec what you expect (step 1 and 2)? Did the controllers create a ReplicaSet and the Pods (step 3)? Are the Pods pending because the scheduler finds no suitable node (step 4)? Did the kubelet fail to pull the image or did the container crash (step 5)? Only when the Pods are running does the request path become the question: is the Pod ready, is it listed as an endpoint, does the Ingress route to the right Service? Teams that internalise this order stop guessing and start reading the system from top to bottom. In kubectl terms, the reading order looks like this:

# Steps 1 and 2: is the stored spec what you expect?
kubectl get deployment checkout -o yaml

# Step 3: did the controllers create the ReplicaSet and the Pods?
kubectl get replicaset,pods -l app=checkout

# Steps 4 and 5: the events tell you about scheduling, image pulls and crashes
kubectl describe pod <pod>

# Step 5: why did the previous container exit?
kubectl logs <pod> --previous

# Step C: which Pods does the Service currently consider ready?
kubectl get endpointslices -l kubernetes.io/service-name=checkout

# Step B: does the route point at the right Service and port?
kubectl describe ingress checkout

GitOps is the control loop extended to Git. If the cluster reconciles itself towards the objects in etcd, the obvious next step is to reconcile etcd towards a Git repository. That is exactly what tools such as Argo CD and Flux do: they are controllers whose desired state lives in a repository, and they continuously correct the cluster towards it. Nothing about this is foreign to Kubernetes. It is the same pattern, applied one level up, and it turns the repository into the audited, reviewable source of truth for what runs in production.

Readiness is a contract, not a checkbox. Because the Service only routes to ready Pods, a readiness probe that answers "yes" too early makes rolling updates drop requests, and one that never answers "yes" makes a deployment hang with no error in sight. The probe is where an application tells the platform the truth about itself, and it deserves the same care as the business logic.

Resilience of the control plane means the API server and etcd. Every path in the diagram runs through the API server, and every object lives in etcd. Workloads that are already running keep running without them, because the kubelet and the kernel rules on each node do not need the control plane for the request path. But nothing can change: no scaling, no rollout, no replacement of a failed Pod. When a managed Kubernetes offering is evaluated, the availability and backup story of these two components is the part worth reading closely.

The model in two sentences

Kubernetes stores what should be true and lets independent loops make it true, each one watching the API server, comparing desired with observed state and correcting the difference. Traffic reaches a workload through a chain of declarations, from load balancer to Ingress to Service, that controllers turn into forwarding rules, and only Pods that have declared themselves ready are part of that chain.

Everything else, from Helm charts to service meshes to operators, builds on these two ideas. A team that can draw this diagram from memory is in a good position to judge which of those additions it actually needs.

Comments

Write a comment

Loading the form …


Back to the index