A Kafka setup of around 150 nodes in a single cluster

Real-time ingestion for a national telecom needed a lot of Kafka capacity, and running many separate clusters would have multiplied the operational load.

I designed a setup of around 150 Kafka nodes inside a single cluster:

• Config groups gave groups of brokers their own tuned configuration.
• Separate ZooKeeper znodes kept logically separate Kafka estates isolated while they were still managed from one place.

Monitoring with Grafana and Kafka-HQ and security in line with the rest of the lake (Kerberos, Ranger) gave the team one consistent estate to run.