Apache Kafka is the backbone of modern data architecture, a streaming platform that enables real-time event driven data processing and can act as a solid integration point for microservices.
In this presentation, we share our experiences and approaches for operating, monitoring and maintaining hundreds of managed Apache Kafka clusters on behalf of Aiven customers. We touch on our architecture, tooling and automation used to maintain high uptimes, actively manage partition placement, perform live scaling and migrations as well as perform roll-forward updates for software patches and major version upgrades.