Apache Kafka DevOps with Kubernetes and GitOps @gamussa | @devnexus | @confluentinc
A presentation at Devnexus 2021 in February 2021 in by Viktor Gamov
Apache Kafka DevOps with Kubernetes and GitOps @gamussa | @devnexus | @confluentinc
What are we going to talk about? @gamussa | @devnexus | @confluentinc
★gamov.dev/twitter ★gamov.dev/telegram ★gamov.dev/youtube @gamussa | @devnexus | @confluentinc
https://gamov.dev/kafka-devops
Preface
A Minimal Microservice Implementation (RPC) Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Event Streaming Paradigm To rethink data as neither stored records nor transient messages, but instead as a continuously updating Stream of Events
A Stream represents history as a sequence of Events @gamussa | @devnexus | @confluentinc
Events change the way we think Event-First Approach ●an event ●a stream ●a ‘data’ flow ●a stream processor Monolithic Approach ●a database ●a variable ●a singleton ●an RPC Orders Service Payments Service Orders Service Order Validation Service Customers Service request response event streams Email Notification Service DB @gamussa | @devnexus | @confluentinc Tax Service
Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Kafka Connect Broker Broker Broker Cluster Data Source Kafka Connect @gamussa Kafka Connect | @devnexus | @confluentinc Data Sink
Gitops
• The Git repo represents the desired state of the system. • Use existing development workflows and tools, like PRs and code reviews. • A clear history of the changes to your system in a Git. @gamussa | @devnexus | @confluentinc
Gitops with Flux @gamussa | @devnexus | @confluentinc
Stop! Demo time @gamussa | @devnexus | @confluentinc
• ccloud • kubectl • helm • fluxctl • k3d (for local testing) • kubeseal • jq • yq Prerequisites @gamussa | @devnexus | @confluentinc
Operator pattern
Gitops with Flux @gamussa | @devnexus | @confluentinc
cCloud-operator @gamussa | @devnexus | @confluentinc
Connect operator @gamussa | @devnexus | @confluentinc
Stop! Demo time @gamussa | @devnexus | @confluentinc
Let’s scale connect
Want to learn more? @gamussa | @devnexus | @confluentinc
Learn Kafka. Watch full version https://gamov.dev/developer Start building with Apache Kafka at Confluent Developer. Confluent Developer developer.confluent.io
@gamussa | @devnexus | @confluentinc
@gamussa | @devnexus | @confluentinc
