In this blog, we explore Auryc's journey of migrating Kafka to Kubernetes using Strimzi. Part One of the blog focus mainly on setup.

Managing Kafka On Kubernetes with Strimzi - Part Two will focus more on operation.

Introduction

VM base cluster deployment

At Auryc, Kafka serves as a fundamental component of our infrastructure. The main traffic sent to collectors directly goes into our Kafka clusters. Additionally, Kafka functions as a message queue, channeling various application streams.

Initially, our Kafka clusters were established on multiple high-performance virtual machines (VMs). This approach, while functional, presented significant challenges in terms of adaptability and operations.

The VM-based setup made it particularly challenging to implement changes, whether for scaling, updating, or modifying configurations. Operations often required meticulous planning and execution, hindering our agility in responding to evolving needs and maintaining optimal performance.

Strimzi Kafka Operator

After evaluating various Kafka operators available for Kubernetes, we narrowed down our options based on maturity, flexibility, and cost-effectiveness:

1. Banzai Cloud Kafka Operator: We found this option not mature enough for our requirements, lacking in certain aspects of stability and feature-completeness.

2. Confluent Operator: While offering a robust solution and backed by the developer of Kafka, the Confluent Operator is proprietary and not open-source. Its high cost was another factor that led us to consider other alternatives.

3. Strimzi Kafka Operator (✅ Chosen Solution):

   After a careful evaluation we chose Strimzi for Kafka management. Strimzi's an opensource project and was accepted to CNCF. Its much more muture with comprehensive documentation. Moreover, Strimzi's integration with Kubernetes through well defined Custom Resource Definitions (CRDs) streamlines the management of Kafka clusters.

The Strimzi Operator

Components of Strimzi Operators:

1. Cluster Operator: This is the central component. It manages the entire Kafka ecosystem, including Kafka clusters, Kafka Connect, Kafka MirrorMaker, and others. It's responsible for deploying and maintaining these components, guided by configurations defined in custom resources.
2. Entity Operator: A composite of two other operators:
   • Topic Operator: Manages Kafka topics. It ensures that the KafkaTopic resources in Kubernetes are in sync with the actual topics in the Kafka cluster.
   • User Operator: Handles Kafka user management. It aligns KafkaUser resources with the real Kafka users, managing authentication and authorization.

Strimzi Custom Resources

Strimzi enriches the Kubernetes ecosystem by introducing Custom Resource Definitions (CRDs) specific to Kafka. These CRDs represent various entities within a Kafka system, such as clusters, users, and topics. By defining these resources in Kubernetes, the Strimzi operators work to reconcile and materialize them into actual Kafka components.