How to Drain Kubernetes Nodes Running Databases Safely

A database can look healthy until the first node upgrade. The drain begins, one pod leaves, and its replacement stays Pending because the remaining nodes cannot host it. A PodDisruptionBudget may stop the next eviction, but it cannot create capacity or finish database recovery.

Safe Kubernetes database maintenance needs an eviction policy, a placement plan and an application-level success check. Treat the node drain as one step in that procedure.

TL;DR

• A PodDisruptionBudget limits eligible voluntary evictions. It does not guarantee database availability.
• Confirm replica health, storage placement and replacement capacity before draining.
• Follow the database operator's maintenance procedure and preserve its ownership of generated resources.
• Prove that reads, writes and replication recover before moving to the next node.

Technical documentation reviewed 16 October 2026. Commands and scenarios are illustrative; no production cluster was modified for this article.

What does a PodDisruptionBudget actually protect?

A PodDisruptionBudget, or PDB, sets an availability limit for a selected group of pods during voluntary evictions. A drain using the Eviction API respects that limit. A failed machine or other involuntary disruption can still take pods away, and that unavailability reduces the remaining budget. Kubernetes disruption documentation

The important distinction is between the Kubernetes signal and the database outcome. A Ready pod is useful evidence, but it does not automatically prove acceptable replication lag, a reachable writer endpoint or successful client transactions.

A PDB also does not govern every deletion path. Kubernetes documents that workload rolling updates are controlled by the workload's own update behavior, rather than being limited by its PDB. Evaluate the specific maintenance operation instead of treating the budget as a universal safety switch.

How does the budget behave with three replicas?

Consider an illustrative workload with three pods selected by one PDB and minAvailable: 2. When all three are healthy, the budget can permit one voluntary eviction. If one pod is already unavailable, a further eviction would violate that minimum.

Observed state	Maintenance implication
Three healthy selected pods	One eviction may fit the budget; still check database readiness and spare capacity.
Two healthy selected pods	Restore the missing replica before expecting another eviction to proceed.
One pod with minAvailable set to one	The policy leaves no room for voluntary eviction. Plan an outage or change the architecture deliberately.
Replacement pod remains Pending	Investigate scheduling and storage placement; the PDB cannot resolve those constraints.

Kubernetes' PDB configuration guide explains why a zero-eviction budget can prevent a drain from completing. That can be the policy working as configured, rather than a reason to bypass it.

Do not copy a generic PDB into an operator-managed database without checking the operator's documentation. It may manage separate budgets for roles, or coordinate a switchover before permitting removal. Two overlapping policies can also make the intended procedure harder to reason about.

What should be checked before maintenance?

Start with read-only inspection. Substitute the namespace and node from your approved maintenance plan:

kubectl get pdb -n database
kubectl get pods -n database -o wide
kubectl get pods -A --field-selector spec.nodeName=worker-1
kubectl get pvc -n database
kubectl get events -n database --sort-by=.metadata.creationTimestamp

Check the PDB's selected pods and allowed disruptions, not merely whether a PDB object exists. Confirm the operator is healthy, replicas are caught up within your threshold, and the application can reach its current writer or quorum.

Then inspect placement. Can another node satisfy the replacement pod's resource requests, affinity rules and volume topology? A PVC can remain Bound while the pod that needs it cannot be scheduled. Kubernetes distinguishes persistent volume ownership, access modes and node affinity; these constraints must fit the replacement location. Persistent volume documentation

Keep a tested restore path available, but do not confuse a backup with a maintenance procedure. The immediate objective is to preserve the service while replacing capacity. The restore plan covers a different failure.

What should happen during the drain?

Use the procedure supported by your database operator. It may require a planned switchover, a maintenance mode or a particular order of replica replacement. Let that procedure determine the next action.

For a test cluster where the prerequisites are satisfied, a conventional drain might look like:

kubectl drain worker-1 --ignore-daemonsets --timeout=15m

This is an example, not an instruction to run against an arbitrary production node. The timeout bounds how long the command waits; it is not a database recovery objective. The command can stop on conditions that need an explicit decision.

The kubectl drain reference makes several distinctions worth preserving. --disable-eviction uses deletion and bypasses PDB checks. --force addresses pods without an available managing controller; it is not a generic synonym for ignoring a PDB. --delete-emptydir-data permits deletion of local emptyDir data. Do not add those flags simply to make a blocked command finish.

Watch application errors, replication and replacement scheduling while the operation runs. If the remaining replicas degrade, pause the maintenance sequence and investigate before touching another node.

How do you prove maintenance succeeded?

A successful drain establishes that the targeted eviction work completed, subject to documented exclusions. It does not establish that the database has returned to its normal operating margin. Kubernetes documents both the drain procedure and its treatment of excluded system pods in its safe-drain guide.

Use an acceptance record with concrete checks:

Check	Evidence to retain
Client correctness	A uniquely identified test write is readable through the normal application endpoint.
Service impact	Error count and latency during the maintenance window remain within your agreed limits.
Replica recovery	The intended replica count and database replication health are restored.
Placement margin	No unexpected Pending pods or storage attachment errors remain.
Next-step readiness	The operator is settled and the disruption allowance is understood before another node is touched.

When a maintained node is ready to accept workloads again, follow the cluster procedure for returning it to service. Avoid combining several maintenance steps into one unobserved batch.

What changes on NKS?

Nirvana Kubernetes Service provides the cluster and integrates persistent storage through Nirvana's CSI driver. Its storage documentation explains the storage layer. The database's replication policy, readiness checks and operator runbook still need to be designed and tested for the workload.

Evaluate node replacement under representative database activity. Record the time to a useful client response and a healthy replica set, alongside storage attachment and pod readiness. Our persistent-volume expansion guide covers a separate storage maintenance operation.

FAQ

Should an unhealthy pod always block a drain?

Kubernetes recommends considering the AlwaysAllow unhealthy-pod eviction policy to avoid a drain waiting indefinitely for a misbehaving application. Check the database operator's support and recovery procedure before changing that behavior. Source

Does three replicas mean the database can lose any two?

No. Availability depends on the database's quorum or replication rules, role placement and failure domains. A pod count alone does not define fault tolerance.

What if the drain does not complete?

Keep the node in its current maintenance state while you inspect the blocking pod, budget, controller, scheduling events and storage constraints. Resolve the cause through the runbook rather than deleting the protection.

Test maintenance as a workload behavior. Start with NKS storage documentation and define the database checks that must pass before the next node leaves service.

About Nirvana Labs

Nirvana Labs is a high-performance storage cloud purpose built for blockchain, AI and databases i.e. the most demanding, real-time, stateful workloads. Accelerated Block Storage (ABS) offers 20K baseline IOPS included, no over provisioning. Nirvana Kubernetes Service (NKS) with Karpenter auto-scaling, high clock-speed compute and private networking. Backed by Jump Trading, Crucible, etc with 50+ customers live in production today.

Learn more at Nirvana Labs

Nirvana Cloud | Pricing | Blog | Docs | Changelog | LinkedIn | Twitter | Telegram | YouTube