A presentation at Kafka Summit Americas by Robin Moffatt
This is the recording of my Kafka Summit 2021 talk about doing interesting things with real-time streams of AIS data. It covers both “standard” streaming ETL stuff to build fancy dashboards, as well as using stream processing to identify in real-time ships that may be engaged in transshipping.
If you’re interested in the topic but prefer the written word to video, the talk is based on these two blogs.
The great thing about streams of real-time events is that they can be used to spot behaviours as they happen and respond to them as needed. Instead of waiting until tomorrow to find out what happened yesterday, we can act on things straight away.
This talk will show a real-life example of one particular pattern that it’s useful to detect—ships engaged in potentially suspicious behaviour at sea. Transhipping is often used for legitimate purposes to optimise efficiencies but can also be used for nefarious purposes such as illegal fishing.
By capturing streams of maritime AIS data in real-time into Kafka and processing it with ksqlDB, it’s possible to detect the kind of characteristics that could indicate behaviour of interest, such as ships moving slowly at close proximity for a length of time.
I’ll demonstrate how the data was ingested from a raw TCP feed, unified with reference data from CSV files, and then processed to spot patterns with the resulting real-time stream of matches written to a new Kafka topic for validation and analysis.
The following resources were mentioned during the presentation or are useful additional information.
Managed Apache Kafka, ksqlDB, and Schema Registry. Use code
RMOFF200 when you sign up!
The pre-eminent resource for learning Apache Kafka. There are free training courses, event streaming patterns, deep-dive articles, and language-specific client programming guides. Check it out!
Here’s what was said about this presentation on social media.