A presentation at Devnexus 2021 by Viktor Gamov
Apache Kafka DevOps with Kubernetes and GitOps @gamussa | @devnexus | @confluentinc
What are we going to talk about? @gamussa | @devnexus | @confluentinc
★gamov.dev/twitter ★gamov.dev/telegram ★gamov.dev/youtube @gamussa | @devnexus | @confluentinc
https://gamov.dev/kafka-devops
Preface
A Minimal Microservice Implementation (RPC) Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Event Streaming Paradigm To rethink data as neither stored records nor transient messages, but instead as a continuously updating Stream of Events
A Stream represents history as a sequence of Events @gamussa | @devnexus | @confluentinc
Events change the way we think Event-First Approach ●an event ●a stream ●a ‘data’ flow ●a stream processor Monolithic Approach ●a database ●a variable ●a singleton ●an RPC Orders Service Payments Service Orders Service Order Validation Service Customers Service request response event streams Email Notification Service DB @gamussa | @devnexus | @confluentinc Tax Service
Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service
Kafka Connect Broker Broker Broker Cluster Data Source Kafka Connect @gamussa Kafka Connect | @devnexus | @confluentinc Data Sink
Gitops
• The Git repo represents the desired state of the system. • Use existing development workflows and tools, like PRs and code reviews. • A clear history of the changes to your system in a Git. @gamussa | @devnexus | @confluentinc
Gitops with Flux @gamussa | @devnexus | @confluentinc
Stop! Demo time @gamussa | @devnexus | @confluentinc
• ccloud • kubectl • helm • fluxctl • k3d (for local testing) • kubeseal • jq • yq Prerequisites @gamussa | @devnexus | @confluentinc
Operator pattern
Gitops with Flux @gamussa | @devnexus | @confluentinc
cCloud-operator @gamussa | @devnexus | @confluentinc
Connect operator @gamussa | @devnexus | @confluentinc
Stop! Demo time @gamussa | @devnexus | @confluentinc
Let’s scale connect
Want to learn more? @gamussa | @devnexus | @confluentinc
Learn Kafka. Watch full version https://gamov.dev/developer Start building with Apache Kafka at Confluent Developer. Confluent Developer developer.confluent.io
@gamussa | @devnexus | @confluentinc
@gamussa | @devnexus | @confluentinc
Operating critical Apache Kafka® event streaming applications in production requires sound automation and engineering practices. Streaming applications are often at the center of your transaction processing and data systems, requiring them to be accurate and highly available.
In this talk, Viktor Gamov, a developer advocate with Confluent, will share the ideas around the project that weaves together Kubernetes, GitOps, and Confluent Cloud.
The following resources were mentioned during the presentation or are useful additional information.
Here’s what was said about this presentation on social media.
