Skip to main content
In this guide, we detail security best practices for hardening the Kubernetes Controller Manager and Scheduler—two pivotal control-plane processes responsible for maintaining your cluster’s desired state. Implementing these measures helps protect against lateral movement, unauthorized access, and data breaches.

​
Controller Manager and Scheduler Roles

A Kubernetes cluster uses these control-plane components:
  • Controller Manager
    • Monitors node and pod health
    • Maintains the desired number of pod replicas
    • Manages service accounts via controllers (e.g., ReplicationController, EndpointController, NamespaceController, ServiceAccountController)
  • Scheduler
    • Assigns pods to nodes based on resource availability and scheduling constraints
The image illustrates the architecture of Kubernetes Controller Manager and Scheduler, showing their roles in managing node health, pod health, and service accounts within a cluster.

​
1. Isolation on Dedicated Nodes

Running the Controller Manager and Scheduler on isolated master nodes prevents compromised application pods from reaching critical control-plane components.
  • Dedicate nodes exclusively for control-plane services
  • Taint master nodes to avoid scheduling regular workloads
  • Monitor and patch these nodes independently
The image illustrates the architecture of a Kubernetes cluster, highlighting the controller manager, scheduler, and nodes with pods.
By isolating control-plane components:
  • You limit lateral movement if an application pod is breached
  • You can apply updates and security patches without impacting user workloads
Use kubectl taint nodes and node selectors to keep control-plane pods off worker nodes.
The image illustrates a Kubernetes cluster with nodes, showing the controller manager and scheduler components, along with pods distributed across the nodes.

​
2. Role-Based Access Control (RBAC)

Adopt least-privilege RBAC policies so the Controller Manager and Scheduler only have access to the resources they need.
ComponentPermissions GrantedPermissions Denied
Controller ManagerManage ReplicaSets, Services, EndpointsSecrets, NetworkPolicies, ConfigMaps
SchedulerList and watch pods, nodes, bindingsCreating roles, accessing etcd directly
The image illustrates the Kubernetes Controller Manager and Scheduler, showing how RBAC manages pod replicas, service accounts, and scheduling tasks within a cluster of nodes.
Review the Kubernetes RBAC documentation when defining your ClusterRoles and RoleBindings.

​
3. Encrypting Communications with TLS

Ensure all communication between the Controller Manager, Scheduler, API Server, and etcd is encrypted:
  • Enable mutual TLS (mTLS) for client-server and server-server connections
  • Use a reputable Certificate Authority (CA) or cert-manager
  • Automate certificate renewal to avoid expired credentials
The image illustrates a Kubernetes cluster architecture, showing nodes with controller manager, scheduler, and pods, connected via SSL.
Expired certificates can silently fail, causing control-plane outages. Implement automated alerts to track upcoming expirations.

​
4. Audit Logging

Activate audit logging for both the Controller Manager and Scheduler to record every API request and response. These logs are essential for forensic analysis and anomaly detection. 
Event Type: PATCH
Timestamp: 2024-11-09T12:34:56Z
Description: ReplicaSet controller updated deployment "my-deployment" in namespace "default" to adjust replicas to 3. Request by user "system:serviceaccount:kube-system:replicaset-controller" from IP 10.10.10.10. Status: 200 OK.

Event Type: GET
Timestamp: 2024-11-09T12:35:12Z
Description: Kube Controller Manager retrieved config map "kube-root-ca.crt" in namespace "kube-system". Request by user "system:serviceaccount:kube-system:kube-controller-manager" from IP 10.10.10.11. Status: 200 OK.

Event Type: PATCH
Timestamp: 2024-11-09T12:35:45Z
Description: Horizontal Pod Autoscaler adjusted settings for "my-app-autoscaler" in namespace "default" to min replicas 2, max replicas 10, target CPU utilization 80%. Request by user "system:serviceaccount:kube-system:horizontal-pod-autoscaler" from IP 10.10.10.12. Status: 200 OK.
Use monitoring stacks like Prometheus and Grafana to set up alerts for suspicious API patterns.

​
Summary of Best Practices

The image lists seven security practices for Kubernetes Controller Manager and Scheduler, including isolating nodes, using RBAC, encrypting communications, enabling audit logging, securing settings, running the latest version, and scanning for vulnerabilities.
  1. Isolate Controller Manager and Scheduler on dedicated, tainted nodes
  2. Apply least-privilege RBAC policies
  3. Encrypt all control-plane traffic (TLS/mTLS)
  4. Enable detailed audit logging with real-time monitoring
  5. Secure default configurations and protect kubeconfig files
  6. Keep Kubernetes up to date with security patches
  7. Regularly scan for vulnerabilities and remediate promptly

Watch Video