Microsoft Azure <3 Open Source Tech

❤ Microsoft Azure Open Source Tech

Microsoft + Open Source ? 2 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

GitHub Octoverse 2018 3 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

GitHub Octoverse 2018 4 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

GitHub Octoverse 2018 5 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

GitHub Octoverse 2018 6 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

HDInsight Services 7 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

HDInsight Services 8 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

$ whoami ” • Hans-Peter Grahsl • living & working in Graz • technical trainer at • independent engineer & consultant • associate lecturer • 9 irregular conference speaker @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Application Needs ? • “It depends” ! • plenty of pieces & components • irrespective of concrete use cases • two architectural pieces… 10 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

operational data store 11 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Cosmos DB 12 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

messaging platform 13 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Event Hubs 14 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

15 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Cosmos DB • global distribution • high availability & elastic scaling • multi-model & native NoSQL APIs • consistency choices • leading security, compliance & SLAs 16 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Multi-Model • keys + values • documents • column families • graphs 17 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Native API Support • SQL • Table Storage • MongoDB • Cassandra • Gremlin 18 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Throughput Provisioning • request units (RUs) • abstract performance metric • combination of CPU + Memory + IOPS • backed by SLAs 19 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

exciting …in theory

Azure Docs on RUs “The cost of all database operations is normalized by Azure Cosmos DB and is expressed in terms of Request Units (RUs). The cost to read a 1-KB item is 1 Request Unit (1 RU) and minimum RUs required to consume 1 GB of storage is 40.” 21 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

22 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

daunting …in practice

Measure via API • write operation consumed ??? RUs • given 1000 provisioned RUs at most ☛ 1000 ÷ 20.19 ≅ 49 docs/sec • no parallel activity considered 24 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Measure via API • read operation consumed ??? RUs • given 1000 provisioned RUs at most ☛ 1000 ÷ 3.11 ≅ 321 reads/sec • no parallel activity considered 25 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Influences on RUs • indexing • property count • consistency level • query patterns • scripts …and any other concurrent operation 26 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Cosmos DB for MongoDB API • native implementation • wire protocol compatibility • transparent re-use (code + tools) • migration benefits: in both ways 27 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Showtime! 28 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Important Note • feature support is limited of course • carefully check what does (NOT) work • wire protocol versions: 3.2 GA, 3.4 preview 29 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

3.6 ?? 4.0 31 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Currently Missing Out On… • powerful operators from 3.2 + 3.4 - $graphLookup, $facet, $bucket(Auto) - $reduce, $zip, $switch, $replaceRoot, … • great features & major advances from 3.6 + 4.0 - views, change streams, - type conversions, schema validation, - multi-element array updates, array filters, - sessions & transactions, … 33 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

34 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Event Hubs • distributed event ingestion platform • decouple producers ⬌ consumers • source for stream processing • high availability & scalability • fully-managed 35 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Concepts: Event Hub 36 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Concepts: Partition 37 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Concepts: Partition Keys 38 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Concepts: Partition Offsets 39 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Concepts: Big Picture 40 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Event Hubs for Apache Kafka • overlay on top of Event Hubs • binary compatible with Kafka 1.0+ • transparent re-use (code + tools) • migration benefits: in both ways 41 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

The Promise… “You update the connection string in configurations to point to the Kafka endpoint exposed by your event hub instead of pointing to your Kafka cluster. Then, you can start streaming events from your applications that use the Kafka protocol into Event Hubs.” 42 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Showtime! 43 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

Unsupported Kafka Features • idempotent producers & transactions • message compression • size-based retention or log compaction • parition resize for existing topics • HTTP Kafka API support • Kafka Streams & KSQL 44 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

The devil is in the detail

THANK YOU Q&A? https://bit.ly/2W7KGn1 46 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria