FUNDAMENTALS · MECHANISM WALKTHROUGH

kubectl apply to Running Pod
What Really Happens, Step by Step

from a YAML manifest on your laptop to a container actually serving traffic — every control-plane hop in order.

kubectl apply→ API SERVER AUTH/ADMISSION→ ETCD WRITE→ SCHEDULER ASSIGNS NODE→ KUBELET WATCHES→ CRI PULLS IMAGE→ CONTAINER STARTS→ READINESS PROBE→ TRAFFIC SERVED

01 The Full Sequence — Worked Example

kubectl Sends a REST Request

kubectl apply -f deployment.yaml converts the YAML to JSON and sends it as an HTTPS request to the API server — kubectl itself has no cluster logic; it's purely a REST client.

Client

Authentication & Authorization

The API server authenticates the request (client cert, token, or OIDC) and then checks RBAC — does this identity have permission to create/update Deployments in this namespace? A denial stops everything right here.

AuthN/Z

Admission Controllers Run

Before persisting anything, admission webhooks/controllers can mutate the object (inject a sidecar, default values) or validate it (reject a Deployment missing resource limits, for example) — this is where policy enforcement (OPA/Gatekeeper, etc.) lives.

Admission

Checkpoint — Written to etcd

The API server persists the Deployment object to etcd, the cluster's key-value store. This is the single source of truth for desired cluster state — nothing in Kubernetes is "real" until it's here. No Pod exists yet at this point.

Checkpoint

Deployment Controller Creates a ReplicaSet

The Deployment controller (running in the control plane, watching the API server) notices the new Deployment and creates a ReplicaSet, which in turn creates Pod objects — again, only object definitions in etcd so far, nothing running.

Controller

Scheduler Assigns a Node

The scheduler watches for Pods with no assigned node, filters candidate nodes (enough CPU/memory, matching nodeSelector/taints), scores the survivors, and writes the chosen node name into the Pod's spec via the API server.

Schedule

kubelet on That Node Notices

Every node runs a kubelet that continuously watches the API server for Pods assigned to its own node. It sees the new assignment and begins actually creating the container — this is the first point any real container work happens.

kubelet

Container Runtime Pulls the Image

kubelet talks to the local container runtime (containerd, CRI-O) over the CRI (Container Runtime Interface). If the image isn't already cached on the node, it's pulled from the registry now — often the slowest single step in the whole sequence.

Pull

Container Starts

The runtime creates and starts the container's namespaces/cgroups and launches the entrypoint process. kubelet reports the Pod phase as Running — but "Running" is not the same as "Ready."

Start

Readiness Probe Determines Traffic Eligibility

If a readinessProbe is defined, kubelet polls it (HTTP/TCP/exec). Only once it succeeds does the Endpoints controller add this Pod's IP to any matching Service's endpoint list — a Running-but-not-Ready Pod receives zero traffic.

Readiness

Traffic Actually Flows

Once the Pod is an active Endpoint, kube-proxy's rules on every node route matching Service traffic to it. Only now — 10 steps after you ran the command — is the new Pod actually serving real requests.

Serve

02 How to Explain This in an Interview

Say kubectl is "just a REST client." It has zero cluster intelligence — establishing this up front frames everything that follows correctly.
Emphasize etcd as the single source of truth. Nothing is "real" until it's persisted there — every controller in the system is just watching and reacting to that state.
Distinguish scheduling from running. The scheduler only assigns a node name; it never talks to kubelet directly or starts anything itself — a subtlety interviewers specifically probe for.
Draw a hard line between Running and Ready. This is the single most practically important distinction in the whole flow, and directly explains "why is my new pod not getting traffic yet."
Name the control loop pattern explicitly — every component here (Deployment controller, scheduler, kubelet) is independently watching the API server and reconciling, not calling each other directly.

03 Follow-Up / Gotcha Questions

Q Does the scheduler talk directly to the kubelet on the chosen node?

A No — it only writes the node assignment back to the API server (and etcd). The kubelet independently discovers the assignment by watching the API server itself; there's no direct scheduler-to-kubelet call.

Q Why can a Pod show status "Running" but still receive zero traffic?

A Because Running only reflects the container process state, not application health. Until the readiness probe passes, the Endpoints controller won't add the Pod's IP to the Service, so kube-proxy never routes traffic to it.

Q What happens if admission control rejects the manifest?

A kubectl apply returns an error immediately and nothing is written to etcd — no ReplicaSet, no Pod, nothing is scheduled. The entire downstream chain never starts.

Q If I re-run kubectl apply with an unchanged manifest, does anything happen?

A The API server computes a diff (a strategic merge patch) against the existing stored object; if nothing changed, it's effectively a no-op — no new Pods, no restarts.

Q Why is an image pull often the slowest step, and how do you avoid repeating it?

A Registry pulls are network + disk I/O bound and can be large. Nodes cache pulled image layers locally, so subsequent pods using the same image on the same node skip the pull — this is one reason node affinity/image pre-pulling matters for fast scale-up.

Q Does the Deployment object itself ever run anything?

A No — Deployment is purely a declarative spec plus a controller loop. It creates and manages ReplicaSets, which create Pods; the Deployment object itself never becomes a running process anywhere.

04 Quick-Fire Glossary

Term	Meaning
API Server	The single entry point for all cluster reads/writes — everything else watches it
etcd	Distributed key-value store holding the cluster's entire desired/actual state
Admission Controller	Webhook/plugin that mutates or validates objects before they're persisted
Controller	A watch-and-reconcile loop (Deployment, ReplicaSet, etc.) driving actual toward desired state
Scheduler	Assigns unscheduled Pods to a node based on resource fit and constraints
kubelet	Per-node agent that watches for Pod assignments and manages the actual containers
CRI	Container Runtime Interface — how kubelet talks to containerd/CRI-O
Readiness Probe	Health check gating whether a Pod receives traffic via a Service
Endpoints	The controller-maintained list of ready Pod IPs backing a Service
RBAC	Role-Based Access Control — governs who can do what against the API server