Kubernetes Descheduler Guide: Keep Your Cluster Balanced Automatically
Introduction
Have you ever noticed that your Kubernetes cluster becomes unbalanced over time? Some nodes end up running hundreds of pods while others sit nearly idle. This happens because the Kubernetes scheduler only makes decisions when pods are created — it doesn’t rebalance existing pods when cluster conditions change.
Enter the Descheduler — a tool that identifies pods that can be moved and evicts them so they can be rescheduled to better nodes.
In this article, I’ll share my hands-on experience configuring the Descheduler on OpenShift, including the challenges I faced and the solutions that actually worked.
The Problem the Scheduler Doesn’t Solve
The default scheduler only acts when pods are created. It does not rebalance running pods.
Common imbalance scenarios:
- Nodes recover after failure, but pods don’t return
- New nodes are added, but remain underused
- Resource usage drifts unevenly
- Affinity or taint rules change
- One node runs too many replicas
Result: capacity waste + performance risk
What the Descheduler Does
The Descheduler runs periodically and:
- Scans nodes and pods
- Applies policy rules
- Selects movable pods
- Evicts them safely
- Scheduler reschedules them
Important: Descheduler does NOT place pods — it only evicts.
Useful Descheduler Strategies
Balance Strategies
Used to spread pods better:
- LowNodeUtilization — move pods away from overloaded nodes
- HighNodeUtilization — compact pods for scale-down
- RemoveDuplicates — avoid the same ReplicaSet pods on one node
- TopologySpread enforcement
Deschedule Strategies
Used to remove invalid placements:
- Violating node affinity
- Violating taints
- Anti-affinity conflicts
- Too many restarts
- Very old pods
- Failed pods
Example — Pod Count Balancing
Cluster state:
Node1 — Pods 192
Node2 — Pods 92
Node3 — Pods 146
Goal: balance near ~140 pods each.
Policy Example (Pod Count Based)
apiVersion: descheduler/v1alpha2
kind: DeschedulerPolicy
maxNoOfPodsToEvictPerNode: 20
maxNoOfPodsToEvictTotal: 50
profiles:
- name: BalancePods
pluginConfig:
- name: LowNodeUtilization
args:
thresholds:
pods: 40 # < 40% = underutilized (can receive pods)
targetThresholds:
pods: 68 # > 68% = overutilized (evict from here)
- name: DefaultEvictor
args:
nodeFit: false # Allow eviction even if fit check fails
minReplicas: 1
evictLocalStoragePods: true
plugins:
balance:
enabled:
- LowNodeUtilization
filter:
enabled:
- DefaultEvictorAfter several runs, the nodes balanced automatically.
Key Lessons Learned
1. Don’t Mix Threshold Types
Choose one:
# Option A: Pod count only
thresholds:
pods: 40
# Option B: Prometheus metrics only
thresholds:
MetricResource: 0- Pod count
- CPU/memory
- Prometheus metrics
Mixing causes policy errors.
2. nodeFit Can Block Evictions
If your pods have nodeSelector or specific tolerations, the descheduler may refuse to evict them because it can’t verify they’ll fit elsewhere.
Solution: Set nodeFit: false to let the scheduler handle placement.
- name: DefaultEvictor
args:
nodeFit: false # Trust the scheduler3. Tune Eviction Limits for Your Environment
### Default limits are very conservative:
maxNoOfPodsToEvictPerNode: 2
maxNoOfPodsToEvictTotal: 5
#### For faster balancing:
maxNoOfPodsToEvictPerNode: 20
maxNoOfPodsToEvictTotal: 504. Pods That Are Protected
Descheduler will not evict by default; these pods are protected:
- System-critical pods (kube-system)
- DaemonSet pods
- Pods with local storage
- Pods violating PodDisruptionBudget
- Static/mirror pods
Installation
helm repo add descheduler https://kubernetes-sigs.github.io/descheduler/
helm install descheduler descheduler/descheduler --namespace kube-systemBest Practices
- Exclude critical namespaces
- Test your policy before enabling actual evictions
- Running as a CronJob is the recommended approach for production
- Monitor eviction metrics
- Set Appropriate Intervals
- Monitor and Alert
Conclusion
The descheduler keeps placement healthy over time. It’s not for constant movement; it’s for correcting drift and restoring balance. If you run medium-to-large clusters, the descheduler is not optional — it’s operational hygiene.
:) Be sure to clap and follow the writer
About The Author
Suraj Solanki
Senior DevOps Engineer
LinkedIn: https://www.linkedin.com/in/suraj-solanki
Topmate: https://topmate.io/suraj_solanki
