Kafka Meetup at The Shuttle

Event Information

Share this event

Date and Time



Zalando Tech Headquarters BMO

Mollstraße 1

10178 Berlin


View Map

Friends Who Are Going
Event description


For this Kafka Meetup, we welcome Ricardo De Cillo, Dmitriy Sorokin, Andrey Dyachkov, Max Schultze, Daniel Truemper, Ian Duffy, and Conor Clifford with a series of short talks highlighting tales from the trenches. Topics will be focused on operating large, critical Kafka clusters in the cloud - and hope to foster discussion and share best practices for running Kafka without losing sleep.


17:30 - Doors Open (Food and Drinks) *Please bring your ID for security
18:20 - Welcome and Intro
18:30 - Operating Kafka and Zookeeper on AWS - Ricardo De Cillo
19:45 - Bubuku - A Supervisor to Run Kafka on AWS - Dmitriy Sorokin
19:00 - Kafka and EBS - Andrey Dyachkov
19:15 - The Tricky Thing about Offsets - Max Schultze

19:30 - Break - Drinks, snacks, and discussions

19:45 - Kafka-powered Microservice - Daniel Truemper
20:00 - Lessons learned Building and Operating a new event driven Platform - Conor Clifford
20:15 - Backing up Kafka to S3 - Ian Duffy
20:30 - Networking + Drinks
21:45 - Event ends

For more details on topics and speakers, please read below:

Operating Kafka and Zookeeper on AWS
Speaker: Ricardo De Cillo

Deploying and operating Kafka and Zookeeper reliably on AWS is a non trivial task. Most teams learn it the hard way. In this talk we are going to share our experience growing a Kafka cluster to ingest 5 TB of data per day with 99,999% availability.

You can expect to hear about: Deployment, Monitoring, Recovering from various failure scenarios, Scaling, Hardware choices, Important configurations

There is no single solution to this challenge, so let's talk. What choices did you make regarding operations and why?

Bubuku - A Supervisor to Run Kafka on AWS
Speaker: Dmitriy Sorokin

Operating a Kafka cluster on a cloud platform such as AWS comes with its challenges - finding exhibitor nodes, updating the configurations, reacting to lost instances without downtime, rebalancing data across brokers, and many others). Daily operations on large clusters, with thousands of topics, are not trivial, and can take up a lot of time.

To simplify operations, we have built Bubuku, a supervisor for Kafka. In this talk, we will explore some of Bubuku's most useful features, such as rebalances and migration of data, updates of Zookeeper, and rolling restarts of Kafka brokers. We will share our experience in running a large cluster in production using Bubuku, and discuss future plans for the project.

Kafka and EBS
Speaker: Andrey Dyachkov (github)

Kafka cluster is able to grow to huge amount of data stored on the disks. Hosting Kafka, requires support of instance termination (on purpose or just because `cloud provider` decided to terminate the instance), which in our case introduces a node with no data, the rebalance of the whole cluster has to accomplished in order to evenly distribute the data among the nodes, taking hours of data copying. In this talk, we will present how we avoid rebalance after node termination in Kafka cluster hosted on AWS.

The Tricky Thing about Offsets
Speaker: Max Schultze (twitter)

The Data Lake at Zalando is all about archiving data and making it accessible efficiently. For that, we consume a lot of data from Zalando's internal event bus Nakadi, which is based on Kafka. Additionally, we run another internal Kafka that data flows through first before it gets archived. This talk will be about presenting our current infrastructure, as well as talking about the concept you can mess up the most with during manual operations: offsets.

Kafka-Powered Microservice
Speaker: Daniel Truemper (twitter, github)

Microservices have fueled an industry-wide transition toward massively distributed architectures with many more people working on the same systems. They also allow for greater individual ownership, freedom of deployment, and diverse technology choices.

In this distributed landscape, easily creating and maintaining consistency among various views of business-critical data, decoupling reads from writes, decoupling microservice go-lives, while adhering strictly to incrementality emerge as important topics.

This presentation will demonstrate how Kafka can function as a backbone for the microservice world thanks to log compaction, which makes it unique compared to other event brokers. We’ll show how log compaction is essential in delivering a consistent user experience, and how this can empower teams developing their own autonomy. We will walk through a migration path from a hypothetical monolithic shop to a set of microservices.

Lessons learned Building and Operating a new event driven Platform
Speaker: Conor Clifford (twitter)

In this talk, we present several lessons learned by our team, Buffalo, while building, and operating, the new Smart Product Platform - an event driven architecture built on top of Kafka and Nakadi.

Backing up Kafka to S3
Speaker: Ian Duffy (website, github, twitter)

In this talk, we demonstrate how we used Kafka Connect to backup topics to Amazon S3. We show how we can get a backup up and running in minutes, and how to recover data when everything goes wrong.

We look forward to seeing you there! If the status of your attendance changes, please update your registration accordingly to allow for others on the waiting list to attend. Thank you!

Share with friends

Date and Time


Zalando Tech Headquarters BMO

Mollstraße 1

10178 Berlin


View Map

Save This Event

Event Saved