Writing Custom Sink Connectors for Pulsar I/O

A presentation at Pulsar Summit Europe 2021 by Ricardo Ferreira

You currently have massive amounts of data sitting on Apache Pulsar and likely — even more data coming in every second. But part of your project requires this data to be sent to some system to explore the dataset further. You look into the Pulsar website for built-in connectors to that particular system and realize that there are none. What a bummer.

But you don’t have to give up on Pulsar because of this. You can write custom sink connectors for Pulsar I/O that work just like the built-in ones, and all you will need is a bit of Java development experience and creativity. The rest you can leave to this talk. This talk will explain how to start developing your custom connector, show how the code looks like, demonstrate how to deploy, and discuss some of the design decisions that your custom connector may need to address.

Video

Code

The following code examples from the presentation can be tried out live.

Buzz and feedback

Here’s what was said about this presentation on social media.