Real-Time Dashboards in the Corporate World: How Apache Pinot Powers Industry Leaders

Real-Time a in the Corporate World How Apache Pinot® Powers Industry Leaders

Who Much Pizza We Sold? @gamussa | @devnexus | @apachepinot

Viktor GAMOV Head of Developer Advocacy | StarTree f THE CLOUD CONNECTIVITY COMPANY Twitter X: @gamussa Kong Con idential

«I Build Highly Scalable and Highly Available, (sometimes) Over-Engineered Hello World Applications» @gamussa

A Taxonomy of Analytics @gamussa | @devnexus | @apachepinot

REAL-TIME OBSERVABILITY/ MONITORING DASHBOARDS REPORTING EXTERNAL BATCH INTERNAL USER-FACING ANALYTICS @gamussa | @devnexus | @apachepinot REPORTING FEATURES

REAL-TIME DATADOG TRINO, PRESTO, BIGQUERY SNOWFLAKE HADOOP LEGACY DWH @gamussa EXTERNAL BATCH INTERNAL PINOT CLICKHOUSE ROCKSET DRUID | @devnexus | @apachepinot SEE ALSO (BUT WITH CACHING)

Real-Time Analytics Summit 2024 May 7-9, 2024 | San Jose, CA Use code DELIGHT35 to register at rtasummit.com

Who Does Real-Time Analytics? @gamussa | @devnexus | @apachepinot

Who Viewed My Pro ile? Total users 700 Million+ QPS 100,000s Latency SLA < 100 ms p99th Freshness Chinmay Soman Founding Engineer @gamussa | @devnexus | @apachepinot f Seunghyun Lee Senior Software Engineer Seconds

LinkedIn Feed Total users 700 Million QPS 100,000 Latency SLA < 100 ms p99th Freshness @gamussa | @devnexus | @apachepinot Seconds

Uber Eats Restaurant Manager ● Identify surge in realtime ● Detect missed/inaccurate orders in realtime Total users 500000+ QPS 100s Latency SLA < 100 ms p99th Freshness @gamussa | @devnexus | @apachepinot Seconds - Minutes

Who Much Pizza We Sold? @gamussa | @devnexus | @apachepinot

Let’s Break It All Down @gamussa | @devnexus | @apachepinot

@gamussa | @devnexus | @apachepinot

@gamussa | @devnexus | @apachepinot

@gamussa | @devnexus | @apachepinot

@gamussa | @devnexus | @apachepinot

@gamussa | @devnexus | @apachepinot

Let’s See It in Action https://stree.ai/pizza-demo @gamussa | @devnexus | @apachepinot

Pinot Architecture @gamussa | @devnexus | @apachepinot

Batch Ingestion [ , { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET /site/user_status.html HTTP/1.1”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 59.0.3071.115 Safari/537.36” }, ] . . . . . . @gamussa | @devnexus | @apachepinot

Batch Ingestion [ ] { { { { }, }, }, } TABLE . . . . . . . . . . . . @gamussa | @devnexus | @apachepinot

Batch Ingestion segments [ ] { { { { }, }, }, } 1 2 3 4 5 6 7 8 TABLE . . . . . . . . . . . . @gamussa | @devnexus | @apachepinot

, { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0” }, , ip userid remote_user time range index bytes time _time userid inverted index time range index request status bytes range index referrer agent referrer/agent/status star tree index . . . @gamussa | @devnexus | @apachepinot . . . Segment Structure

Segments @gamussa | @devnexus | @apachepinot

Segments @gamussa | @devnexus | @apachepinot

Segments 1 2 3 4 5 6 7 8 9 10 11 12 @gamussa | @devnexus | @apachepinot

Servers 1 2 3 4 5 6 7 8 9 10 11 12 SERVER @gamussa | @devnexus | @apachepinot

Servers 1 2 3 4 13 14 15 5 6 7 8 17 18 19 9 10 11 12 SERVER 1 SERVER 2 @gamussa | @devnexus | @apachepinot 16

Brokers SELECT * FROM SOMETHING BROKER (query needs segments 2 and 8) 1 3 4 6 1 2 3 5 2 4 5 7 8 9 10 7 9 11 12 8 10 12 6 11 SERVER 1 SERVER 2 DEEP STORAGE @gamussa | @devnexus | @apachepinot SERVER 3

All of Pinot BROKER ZOOKEEPE R BROKER CLIENTS CONTROLL ER scatter/gather [ { { { { }, }, }, } constant ingest 1 3 4 6 1 2 3 5 2 4 5 7 8 9 10 7 9 11 12 8 10 12 11 SERVER 1 SERVER 2 ] DEEP STORAGE . . . . . . . . . . . @gamussa | @devnexus | @apachepinot . 6 SERVER 3

, { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0” }, , 1 2 3 4 5 6 7 8 OFFLINE TABLE . . . @gamussa | @devnexus | @apachepinot . . . Batch Ingestion

Realtime Ingestion CONSUMING SEGMENT 1 REALTIME TABLE @gamussa | @devnexus | @apachepinot

Realtime Ingestion CONSUMING SEGMENT 1 2 REALTIME TABLE @gamussa | @devnexus | @apachepinot

Realtime Ingestion CONSUMING SEGMENT 1 2 3 REALTIME TABLE @gamussa | @devnexus | @apachepinot

Realtime Ingestion CONSUMING SEGMENT 1 2 3 4 REALTIME TABLE @gamussa | @devnexus | @apachepinot

Realtime Ingestion CONSUMING SEGMENT 1 2 3 4 REALTIME TABLE @gamussa | @devnexus | @apachepinot

Slack http://stree.ai/slack https://stree.ai/pinot-slack @gamussa | @devnexus | @apachepinot