You can go your way: Kafka without Zookeeper

Slide 1

Ka ka without Zookeeper You can go your way | @con luentinc f f @gamussa

Slide 2

Slide 3

Slide 4

@gamussa | #JPoint | @confluentinc

Slide 5

★gamov.dev/twitter ★gamov.dev/telegram ★gamov.dev/youtube @gamussa | #JPoint | @confluentinc

Slide 6

Agenda What we are going to talk about @gamussa | #JPoint | @confluentinc

Slide 7

Agenda What we are going to talk about • Brief history of Ka ka f @gamussa | #JPoint | @confluentinc

Slide 8

Agenda What we are going to talk about • Brief history of Ka @gamussa | #JPoint | @confluentinc

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

f ka Past and Future

Slide 15

replication @gamussa | #JPoint | @confluentinc

Slide 16

replication @gamussa | #JPoint | @confluentinc

Slide 17

@gamussa | #JPoint | @confluentinc

Slide 18

streams @gamussa | #JPoint | @confluentinc

Slide 19

@gamussa | #JPoint | @confluentinc

Slide 20

exactly once @gamussa | #JPoint | @confluentinc

Slide 21

exactly once @gamussa | #JPoint | @confluentinc

Slide 22

ksqlDB @gamussa | #JPoint | @confluentinc

Slide 23

KIP 500 @gamussa | #JPoint | @confluentinc

Slide 24

KIP 500 @gamussa | #JPoint | @confluentinc

Slide 25

KIP-455: Create an Administrative API for Replica Reassignment KIP-497: Add inter-broker API to alter ISR KIP-543: Expand Con igCommand’s non-ZK functionality KIP-555: Deprecate Direct Zookeeper access in Ka ka Administrative Tools KIP-589: Add API to update Replica state in Controller KIP-590: Redirect Zookeeper Mutation Protocols to The Controller KIP-595: A Raft Protocol for the Metadata Quorum KIP-631: The Quorum-based Ka ka Controller f f f @gamussa | #JPoint | @confluentinc

Slide 26

How Ka ka uses ZooKeeper f @gamussa | #JPoint | @confluentinc

Slide 27

Clients @gamussa | #JPoint | @confluentinc

Slide 28

@gamussa | #JPoint | @confluentinc

Slide 29

@gamussa | #JPoint | @confluentinc

Slide 30

producer consumer streams connect @gamussa | #JPoint | @confluentinc

Slide 31

admin client @gamussa | #JPoint | @confluentinc

Slide 32

admin client /bin tools @gamussa | #JPoint | @confluentinc

Slide 33

quota management replica reassignment admin client /bin tools @gamussa | #JPoint | @confluentinc

Slide 34

@gamussa | #JPoint | @confluentinc

Slide 35

@gamussa | #JPoint | @confluentinc

Slide 36

@gamussa | #JPoint | @confluentinc

Slide 37

Brokers @gamussa | #JPoint | @confluentinc

Slide 38

@gamussa | #JPoint | @confluentinc

Slide 39

@gamussa | #JPoint | @confluentinc

Slide 40

@gamussa | #JPoint | @confluentinc

Slide 41

@gamussa | #JPoint | @confluentinc

Slide 42

@gamussa | #JPoint | @confluentinc

Slide 43

ISR changes event notifications broker registration @gamussa | #JPoint | @confluentinc

Slide 44

@gamussa | #JPoint | @confluentinc

Slide 45

@gamussa | #JPoint | @confluentinc

Slide 46

Client Compatibility @gamussa | #JPoint | @confluentinc

Slide 47

@gamussa | #JPoint | @confluentinc

Slide 48

@gamussa | #JPoint | @confluentinc

Slide 49

@gamussa | #JPoint | @confluentinc

Slide 50

CreateTopic @gamussa | #JPoint | @confluentinc

Slide 51

CreateTopic @gamussa | #JPoint | @confluentinc

Slide 52

CreateTopic @gamussa | #JPoint | @confluentinc

Slide 53

@gamussa | #JPoint | @confluentinc

Slide 54

AlterConfig @gamussa | #JPoint | @confluentinc

Slide 55

AlterConfig @gamussa | #JPoint | @confluentinc

Slide 56

AlterConfig @gamussa | #JPoint | @confluentinc

Slide 57

35 @gamussa | #JPoint | @confluentinc

Slide 58

@gamussa | #JPoint | @confluentinc

Slide 59

AlterConfig @gamussa | #JPoint | @confluentinc

Slide 60

AlterConfig @gamussa | #JPoint | @confluentinc

Slide 61

AlterConfig @gamussa | #JPoint | @confluentinc

Slide 62

AlterConfig @gamussa | #JPoint | @confluentinc

Slide 63

KIP-500: Current State @gamussa | #JPoint | @confluentinc

Slide 64

@gamussa | #JPoint | @confluentinc

Slide 65

Metadata Quorum @gamussa | #JPoint | @confluentinc

Slide 66

@gamussa | #JPoint | @confluentinc

Slide 67

@gamussa | #JPoint | @confluentinc

Slide 68

Leader (Controller) @gamussa | #JPoint | @confluentinc

Slide 69

Follower Leader (Controller) Follower @gamussa | #JPoint | @confluentinc

Slide 70

Follower Leader (Controller) Follower Observers @gamussa | #JPoint | @confluentinc

Slide 71

@gamussa | #JPoint | @confluentinc

Slide 72

Data Nodes Metadata Nodes @gamussa | #JPoint | @confluentinc

Slide 73

Dedicated Deployment Data Nodes Metadata Nodes @gamussa | #JPoint | @confluentinc

Slide 74

Data Nodes Metadata Nodes @gamussa | #JPoint | @confluentinc

Slide 75

Data Nodes @gamussa | #JPoint | @confluentinc

Slide 76

Data/Metadata Nodes @gamussa | #JPoint | @confluentinc

Slide 77

Mixed Deployment Data/Metadata Nodes @gamussa | #JPoint | @confluentinc

Slide 78

Mixed Deployment @gamussa | #JPoint | @confluentinc

Slide 79

Mixed Deployment @gamussa | #JPoint | @confluentinc

Slide 80

Mixed Deployment Observer Promotion @gamussa | #JPoint | @confluentinc

Slide 81

Mixed Deployment Observer Promotion @gamussa | #JPoint | @confluentinc

Slide 82

@gamussa | #JPoint | @confluentinc

Slide 83

Single Node Deployment @gamussa | #JPoint | @confluentinc

Slide 84

Single Node Deployment @gamussa | #JPoint | @confluentinc

Slide 85

Single Node Deployment MM MM MM @gamussa | #JPoint | @confluentinc

Slide 86

KRaft: Ka kaesque Raft f @gamussa | #JPoint | @confluentinc

Slide 87

https://www.kafka-summit.org/sessions/a-kafkaesque-raft-protocol @gamussa | #JPoint | @confluentinc

Slide 88

Writes Fencing Ka ka Raft Single Leader Single Leader Monotonically increasing epoch Monotonically increasing term Log reconciliation Offset and epoch Term and index Push/Pull Pull Push Commit Semantics ISR Majority Leader Election From ISR through Zookeeper Majority Fault Tolerance F+1 2F+1 f @gamussa | #JPoint | @confluentinc

Slide 89

@gamussa | #JPoint | @confluentinc

Slide 90

STOP! Demo time! @gamussa | #JPoint | @confluentinc

Slide 91

Want to learn more about Ka ka? f @gamussa | #JPoint | @confluentinc

Slide 92

Learn Ka ka. Watch full version https://gamov.dev/developer Start building with Apache Ka ka at Con luent Developer. f f f f f Con luent Developer developer.con luent.io

Slide 93

As Always Have A Nice Day @gamussa | @confluentinc