Apache Kafka DevOps with Kubernetes and GitOps

Apache Kafka DevOps with Kubernetes and GitOps @gamussa | @devnexus | @confluentinc

What are we going to talk about? @gamussa | @devnexus | @confluentinc

★gamov.dev/twitter ★gamov.dev/telegram ★gamov.dev/youtube @gamussa | @devnexus | @confluentinc

https://gamov.dev/kafka-devops

Preface

A Minimal Microservice Implementation (RPC) Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Event Streaming Paradigm To rethink data as neither stored records nor transient messages, but instead as a continuously updating Stream of Events

A Stream represents history as a sequence of Events @gamussa | @devnexus | @confluentinc

Events change the way we think Event-First Approach ●an event ●a stream ●a ‘data’ flow ●a stream processor Monolithic Approach ●a database ●a variable ●a singleton ●an RPC Orders Service Payments Service Orders Service Order Validation Service Customers Service request response event streams Email Notification Service DB @gamussa | @devnexus | @confluentinc Tax Service

Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Kafka Connect Broker Broker Broker Cluster Data Source Kafka Connect @gamussa Kafka Connect | @devnexus | @confluentinc Data Sink

Gitops

• The Git repo represents the desired state of the system. • Use existing development workflows and tools, like PRs and code reviews. • A clear history of the changes to your system in a Git. @gamussa | @devnexus | @confluentinc

Gitops with Flux @gamussa | @devnexus | @confluentinc

Stop! Demo time @gamussa | @devnexus | @confluentinc

• ccloud • kubectl • helm • fluxctl • k3d (for local testing) • kubeseal • jq • yq Prerequisites @gamussa | @devnexus | @confluentinc

Operator pattern

Gitops with Flux @gamussa | @devnexus | @confluentinc

cCloud-operator @gamussa | @devnexus | @confluentinc

Connect operator @gamussa | @devnexus | @confluentinc

Stop! Demo time @gamussa | @devnexus | @confluentinc

Let’s scale connect

Want to learn more? @gamussa | @devnexus | @confluentinc

Learn Kafka. Watch full version https://gamov.dev/developer Start building with Apache Kafka at Confluent Developer. Confluent Developer developer.confluent.io

@gamussa | @devnexus | @confluentinc

@gamussa | @devnexus | @confluentinc

Apache Kafka DevOps with Kubernetes and GitOps @gamussa | @devnexus | @confluentinc