Skip to main content

​
Introduction

Welcome to this lesson on Kafka Raft (KRaft), the next evolution of Apache Kafka’s architecture. By integrating a native Raft-based consensus protocol, KRaft streamlines cluster operations and removes the need for an external coordination layer.

​
Challenges with ZooKeeper

Apache Kafka has long relied on ZooKeeper to manage metadata, broker states, and leader elections. While reliable, this approach introduces several pain points:
The image highlights the issue of complexity with Zookeeper, showing a computer screen with various technical elements. It mentions that Zookeeper adds operational overhead and increases the complexity of managing a Kafka cluster.
Maintaining a separate ZooKeeper ensemble demands extra configuration, monitoring, and troubleshooting—making cluster operations more cumbersome.
The image discusses the issue of operational overhead with Zookeeper, highlighting the additional effort required for deployment, monitoring, and scaling. It features an illustration of a person holding a document labeled "Deployment & monitoring" next to a smartphone with gear icons.
Running ZooKeeper as its own service raises infrastructure costs, slows deployments, and increases the chance of human error.
The image discusses the issue of scaling limitations with Zookeeper, highlighting that it can become a bottleneck for large Kafka clusters, impacting performance.
At scale, ZooKeeper can become a throughput and latency bottleneck. Plus, Kafka and ZooKeeper version misalignments can complicate upgrades and feature rollouts.
ZooKeeper coordination requires careful tuning of session timeouts, leader election parameters, and quorum sizes to maintain cluster health.

​
Introducing Kafka Raft (KRaft)

Kafka Raft, or KRaft, embeds the Raft consensus algorithm directly into Kafka brokers. This eliminates the external ZooKeeper dependency and brings metadata management in-house:
  • Simplified architecture: no separate ensemble to provision.
  • Lower operational overhead: fewer components to monitor.
  • Improved scalability: built-in governance for partition metadata.

​
Benefits of KRaft

The image lists three features of Kafka KRaft: right-sized clusters, enhanced stability and simplified management, and a unified security model.
  • Right-sized clusters: Only the Kafka brokers you need—no extra ZooKeeper nodes.
  • Enhanced stability: A single control plane reduces failure points and downtime.
  • Unified security: Consistent authentication, authorization, and encryption across brokers.
The image lists features of Kafka KRaft, including simplified startup and deployment, instant controller failover, and elimination of Zookeeper dependency.
  • Simplified deployment: Start brokers in KRaft mode without bootstrapping an external service.
  • Instant controller failover: Automatic Raft leader election ensures high availability.
  • No ZooKeeper: One less moving part in your data streaming pipeline.

​
Architecture Transformation

Below is a side-by-side view of the old and new Kafka architectures: 
Before (ZooKeeper-based):
  ZooKeeper ensemble
  └── Multiple ZooKeeper nodes
Kafka brokers → ZooKeeper for metadata and leader elections

After (KRaft mode):
  Kafka brokers (Raft cluster)
  ├─ Controller broker (Raft leader)
  └─ Replica brokers
Internal __metadata__ topic replaces ZooKeeper coordination
AspectZooKeeper-based KafkaKRaft-based Kafka
Coordination ServiceExternal ZooKeeper ensembleInternal Raft consensus
Deployment UnitsKafka brokers + ZK nodesKafka brokers only
FailoverSeparate ZooKeeper electionInstant Raft leader election
ScalabilityZK bottlenecks at scaleDistributed metadata management
Security ModelTwo config domainsUnified broker configuration
Do not mix ZooKeeper-based brokers with KRaft brokers in the same cluster. Plan your migration carefully to avoid compatibility issues.

​
Next Steps

In the following lesson, we will walk through:
  1. Initializing a Kafka cluster in KRaft mode
  2. Configuring controller and broker roles
  3. Adding and removing brokers dynamically
  4. Best practices for monitoring and scaling
See you in the next lesson!

Watch Video