Fight Crime with Kafka Streams and the Bintray Firehose API

Fight Crime with Kafka Streams and Bintray Firehose API

Consuming liquid software from firehose like...

Hackers? WAT? × Baruch Sadogursky × Head of DevRel @jfrog × @jbaruch × Viktor Gamov × Developer Advocate @confluentinc × @gamussa

WTF is firehose?!

Firehose api × Twitter has it, AWS has it, Bitntray has it × Streaming events from the socket × A lot of information

Bintray firehose api × × × × × Login success Login failure File uploaded File downloaded File deleted

Consumption × Jfrog cli (piping etc) × Rest api × Java api (soon)

Our process is easy ka Kaf ams e r St ka Kaf r cto e n Con L Q S K

Kafka Basics

What is a Streaming Pl atform? Producer Connectors Consumer The Log Streaming Engine Connectors

Kafka’s Distributed L og Producer Connectors Consumer The Log Streaming Engine Connectors

The log is a simple ide a Old New Messages are added at the end of the log

Consumers have a pos ition all of their own George Scan is here Old Fred Scan is here New Sally Scan is here

Only Sequential Acces s Old Read to offset & scan New

Shard data to get sca lability Producer (1) Producer (2) Producer (3) Cluster of machines Messages are sent to different partitions Partitions live on different machines

Replicate to get faul t tolerance leader Machine A msg Machine B replicate msg

Replication provides r esiliency A ‘replica’ takes over on machine failure

The Connect API Producer Connectors Consumer The Log Streaming Engine Connectors

Ingest / Output to pra ctically any data sou rce Kafka Connect Kafka Kafka Connect

What is a Streaming Pl atform? Producer Connectors Consumer The Log Streaming Engine Connectors

KSQL: continuous com putation SELECT card_number, count() FROM authorization_attempts WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3;

Join Streams and Tabl es Kafka Kafka Streams / KSQL Topic Stream Join Table Compacted Topic

USE CASE LOOKING FOR HACKERS

Bintray firehose api

Use case: looking for hackers! 1. Honeypot: secret file downloads attempts 2. Brute Force: login attempts 3. * leaked passwords: usage of the same password in multiple places

Code is on github × https://github.com/russianhackers × Kafka Connect for bintry × Docker compose and stuff

Thank ya’ll! × @Gamussa × @jbaruch × #OracleCodeOne