Managing Kafka On Kubernetes with Strimzi - Part One
In the second part of our journey with Kafka on Kubernetes, we introduce a pivotal tool that has reshaped how we manage our Kafka clusters: Cruise-control. This dynamic tool is designed to automate and optimize various aspects of Kafka cluster management.
Cruise-control is a comprehensive Kafka management system, known for its ability to monitor key metrics and maintain the health and efficiency of Kafka clusters. It's not just a monitoring tool; it's an active management system that ensures the Kafka environment is running optimally.
Cruise-control can be easily deployed by enable cruiseControl in kafka manage by strimzi.
Cruise-Control UI is a frontend application that provide GUI for cruise-control. It’s not bundled with strimzi but can be added optionally.
Cruise-control continuous monitoring various metrics of the cluster including topic/partition distributions, partition sizes, disk / CPU usages, network rate in/out rates.
Based on the observed metrics, Cruise-control can generates rebalance proposals. These proposals are tailored to enhance the overall performance and efficiency of the Kafka cluster. You can also customize the proposal by including/excluding performance/capacity goals.
Upon approval, Cruise-control can automatically execute these rebalance proposals. It will automatically queue the movements of hundreds of partitions with proper concurrency without impacting the online system, keep track of the progress. And you may interrupt any time needed.
