DEVOPS D-DAY #5 Monitoring OVH: 350k servers, 30 DCs… and one Metrics platform Horacio Gonzalez @LostInBrittany DEVOPS D-DAY #5 @LostInBrittany
A presentation at Devops D-Day 2019 in November 2019 in Marseille, France by Horacio Gonzalez
DEVOPS D-DAY #5 Monitoring OVH: 350k servers, 30 DCs… and one Metrics platform Horacio Gonzalez @LostInBrittany DEVOPS D-DAY #5 @LostInBrittany
Who are we? Introducing myself and introducing OVH OVHcloud DEVOPS D-DAY #5 @LostInBrittany
Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter DEVOPS D-DAY #5 @LostInBrittany
OVH: A Global Leader on Cloud 250k Private cloud VMs running 1 Dedicated IaaS Europe 30 Datacenters Own 20Tbps Hosting capacity : 1.3M Physical Servers 360k Servers already deployed DEVOPS D-DAY #5 Netwok with 35 PoPs
1.3M Customers in 138 Countries @LostInBrittany
OVH: Our solutions Cloud Web Hosting Mobile Hosting Telecom VPS Containers ▪ Dedicated Server Domain names VoIP Public Cloud Compute ▪ Data Storage Email SMS/Fax Private Cloud ▪ Network and Database CDN Virtual desktop Serveur dédié Security Object Storage Web hosting Cloud HubiC Over theBox ▪ Licences Cloud Desktop Securities MS Office Hybrid Cloud Messaging MS solutions DEVOPS D-DAY #5 @LostInBrittany
And don’t forget, next week… OVHcloud Summit https://summit.ovhcloud.com/ DEVOPS D-DAY #5 @LostInBrittany
Once upon a time… Because I love telling tales DEVOPS D-DAY #5 @LostInBrittany
This talk is about a tale… A true one nevertheless DEVOPS D-DAY #5 @LostInBrittany
And as in most tales It begins with a mission DEVOPS D-DAY #5 @LostInBrittany
And a band of heroes Engulfed into the adventure DEVOPS D-DAY #5 @LostInBrittany
They fight against mishaps And all kind of foes DEVOPS D-DAY #5 @LostInBrittany
They build mighty fortresses Pushing the limits of possible DEVOPS D-DAY #5 @LostInBrittany
And defend them day after day Against all odds DEVOPS D-DAY #5 @LostInBrittany
But we don’t know yet the end Because this tale isn’t finished yet DEVOPS D-DAY #5 @LostInBrittany
It begins with a mission Build a metrics platform for OVH DEVOPS D-DAY #5 @LostInBrittany
A long time ago… DEVOPS D-DAY #5 @LostInBrittany
A long time ago… Monitoring: Does the system works? DEVOPS D-DAY #5 @LostInBrittany
Moving from monolith to μservices App DEVOPS D-DAY #5 @LostInBrittany
Moving from monolith to μservices App App App DEVOPS D-DAY #5 @LostInBrittany
Moving from monolith to μservices App App App DB App Slaves DEVOPS D-DAY #5 @LostInBrittany
Moving from monolith to μservices App App App Bus DB App Slaves DEVOPS D-DAY #5 @LostInBrittany
Moving from monolith to μservices RPXY LB Cache App App App Bus DB App Slaves DEVOPS D-DAY #5 @LostInBrittany
What could go wrong? RPXY LB Cache App App App Bus DB App Slaves DEVOPS D-DAY #5 @LostInBrittany
Microservices are a distributed system GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill DEVOPS D-DAY #5 @LostInBrittany
We need to have insights Observability: How the system works? DEVOPS D-DAY #5 @LostInBrittany
OVH decided go metrics-oriented DEVOPS D-DAY #5 @LostInBrittany
A metrics platform for OVH For all OVH DEVOPS D-DAY #5 @LostInBrittany
Building OVH Metrics One Platform to unify them all, One Platform to find them, One Platform to bring them all and in the Metrics monitor them DEVOPS D-DAY #5 @LostInBrittany
What is OVH Metrics? Managed Cloud Platform for Time Series DEVOPS D-DAY #5 @LostInBrittany
OVH monitoring story We had lots of partial solutions… DEVOPS D-DAY #5 @LostInBrittany
OVH monitoring story One Platform to unify them all What should we build it on? DEVOPS D-DAY #5 @LostInBrittany
OVH monitoring story Including a really big DEVOPS D-DAY #5 @LostInBrittany
OpenTSDB drawbacks OpenTSDB RowKey Design ! DEVOPS D-DAY #5 @LostInBrittany
OpenTSDB Rowkey design flaws ● .regex. => full table scans ● High cardinality issues (Query latencies) We needed something able to manage hundreds of millions time series OpenTSBD didn’t scale for us DEVOPS D-DAY #5 @LostInBrittany
OpenTSDB other flaws ● ● ● ● ● Compaction (or append writes) /api/query : 1 endpoint per function? Asynchronous Unauthenticated … DEVOPS D-DAY #5 @LostInBrittany
Scaling OpenTSDB DEVOPS D-DAY #5 @LostInBrittany
Metrics needs First need: To be massively scalable DEVOPS D-DAY #5 @LostInBrittany
Analytics is the key to success Fetching data is only the tip of the iceberg DEVOPS D-DAY #5 @LostInBrittany
Analysing metrics data To be scalable, analysis must be done in the database, not in user’s computer DEVOPS D-DAY #5 @LostInBrittany
Metrics needs Second need: To have rich query capabilities DEVOPS D-DAY #5 @LostInBrittany
Enter Warp 10… Open-source Time series Database DEVOPS D-DAY #5 @LostInBrittany
More than a Time Series DB Warp 10 is a software platform that ● Ingests and stores time series ● Manipulates and analyzes time series DEVOPS D-DAY #5 @LostInBrittany
Manipulating Time Series with Warp 10 A true Time Series analysis toolbox ○ Hundreds of functions ○ Manipulation frameworks ○ Analysis workflow DEVOPS D-DAY #5 @LostInBrittany
Manipulating Time Series with Warp 10 A Time Series manipulation language WarpScript DEVOPS D-DAY #5 @LostInBrittany
Did you say scalability? From the smallest to the largest… DEVOPS D-DAY #5 @LostInBrittany
More Warp 10 goodness ● Secured & multi tenant ● Synchronous (transactions) ● In memory Index ● Better Performance ● No cardinality issues ● Better Scalability ● Lockfree ingestion ● Versatile ● WarpScript Query Language (standalone, distributed) ● Support more data types DEVOPS D-DAY #5 @LostInBrittany
OVH Observability Metrics Platform DEVOPS D-DAY #5 @LostInBrittany
Building an ecosystem From Warp 10 to OVH Metrics DEVOPS D-DAY #5 @LostInBrittany
What protocols should we support? Who must do the effort? DEVOPS D-DAY #5 @LostInBrittany
Open source monitoring tools DEVOPS D-DAY #5 @LostInBrittany
Open source monitoring tools DEVOPS D-DAY #5 @LostInBrittany
Open source monitoring tools DEVOPS D-DAY #5 @LostInBrittany
Open source monitoring tools DEVOPS D-DAY #5 @LostInBrittany
Open source monitoring tools DEVOPS D-DAY #5 @LostInBrittany
Open source monitoring tools DEVOPS D-DAY #5 @LostInBrittany
Open source monitoring tools Why choose? Let’s support all of them! DEVOPS D-DAY #5 @LostInBrittany
Metrics Platform DEVOPS D-DAY #5 @LostInBrittany
Metrics Platform graphite influx https:// opentsdb .<region>.metrics.ovh.net prometheus Warp10 tsl … DEVOPS D-DAY #5 @LostInBrittany
Metrics Platform graphite influx https:// opentsdb .<region>.metrics.ovh.net prometheus Warp10 tsl … DEVOPS D-DAY #5 @LostInBrittany
TSL select(“cpu.usage_system”) .where(“cpu~cpu[0-7]*”) .last(12h) .sampleBy(5m,max) .groupBy(mean) .rate() github.com/ovh/tsl DEVOPS D-DAY #5 @LostInBrittany
Metrics Live In-memory, high-performance Metrics instances DEVOPS D-DAY #5 @LostInBrittany
In-memory: Metrics live millions of writes/s DEVOPS D-DAY #5 @LostInBrittany
In-memory: Metrics live DEVOPS D-DAY #5 @LostInBrittany
In-memory: Metrics live DEVOPS D-DAY #5 @LostInBrittany
Monitoring is only the beginning OVH Metrics answer to many other use cases DEVOPS D-DAY #5 @LostInBrittany
Graveline rack’s temperature DEVOPS D-DAY #5 @LostInBrittany
Even medical research… Metrics’ Pattern Detection feature helped Gynaecology Research to prove patterns on perinatal mortality DEVOPS D-DAY #5 @LostInBrittany
Use cases families • • • • Billing Monitoring IoT (e.g. bill on monthly max consumption) ……………………………………………..……. (APM, infrastructure,appliances,…) …..…………………………… (Manage devices, operator integration, …) …………………………………………….…………………. Geo Location DEVOPS D-DAY #5 (Manage localized fleets) ……..………………… @LostInBrittany
Use cases • • • • • • DC Temperature/Elec/Cooling map Pay as you go billing (PCI/IPLB) GSCAN Monitoring ML Model scoring (Anti-Fraude) Pattern Detection for medical applications DEVOPS D-DAY #5 @LostInBrittany
SREing Metrics With a great power comes a great responsibility DEVOPS D-DAY #5 @LostInBrittany
Metrics’s metrics 432.000.000.000 datapoints / day DEVOPS D-DAY #5 @LostInBrittany 71
Our stack overview More than 666 machines operated by 5 people >95% dedicated servers No Docker, only SystemD Running many Apache projects: ○ Hadoop ○ HBase ○ Zookeeper ○ Flink ● And Warp 10 ● ● ● ● DEVOPS D-DAY #5 @LostInBrittany
Our biggest Hadoop cluster 200 datanodes ~60k regions of 10Gb 2.3 PB of capacity 8.5Gb/s of bandwidth 1.5M of writes/s 3M of reads/s DEVOPS D-DAY #5 @LostInBrittany
Hadoop need a lot of DEVOPS D-DAY #5 @LostInBrittany
Warp10: distributed overview DEVOPS D-DAY #5 @LostInBrittany
Warp10: distributed overview DEVOPS D-DAY #5 @LostInBrittany
Warp10: distributed overview DEVOPS D-DAY #5 @LostInBrittany
Warp10: distributed overview DEVOPS D-DAY #5 @LostInBrittany
Warp10: distributed overview DEVOPS D-DAY #5 @LostInBrittany
Hadoop nodes Most of the nodes are the following: ● ● ● 16 to 32 cores 64 to 128 GB of RAM 12 to 16 TB DEVOPS D-DAY #5 But, we also have some huge nodes: ● ● ● 2x 20 cores (xeon gold) 320 GB of RAM 12x 4TB of Disk @LostInBrittany
Warp10 nodes Ingress (cpu-bound): ● ● 32 cores 128 GB of RAM Egress (cpu-bound): ● ● 32 cores 128 GB of RAM DEVOPS D-DAY #5 Directory (ram-bound): ● ● 48 cores 512 GB of RAM Store (cpu-bound): ● ● @LostInBrittany 32 cores 128 GB of RAM
Why you should care? DEVOPS D-DAY #5 @LostInBrittany
Why you should care? (>30s) DEVOPS D-DAY #5 @LostInBrittany
The only way to optimize: measure What is my application doing? App What is my runtime doing? How many GC triggered? Run tim Is there a hardware failure? Logs DEVOPS D-DAY #5 How many HTTP calls? e Hos t @LostInBrittany How many disk I have left? Metrics
Monitoring JVM with metrics DEVOPS D-DAY #5 @LostInBrittany
Monitoring JVM with metrics DEVOPS D-DAY #5 @LostInBrittany
Monitoring JVM with metrics DEVOPS D-DAY #5 @LostInBrittany
Monitoring JVM with metrics DEVOPS D-DAY #5 @LostInBrittany
Monitoring JVM with metrics DEVOPS D-DAY #5 @LostInBrittany
Tuning G1 is hard DEVOPS D-DAY #5 @LostInBrittany
Tuning G1 is hard DEVOPS D-DAY #5 @LostInBrittany
Our programming stack ● We mostly use garbage collected languages as ○ Go ○ Java ○ JavaScript DEVOPS D-DAY #5 @LostInBrittany
Our programming stack However, we are using non-garbage collected languages as Rust when needed DEVOPS D-DAY #5 @LostInBrittany
Our friends for µservices DEVOPS D-DAY #5 @LostInBrittany
We open-source Code contribution: ● ● ● ● ● ● https://github.com/ovh/beamium https://github.com/ovh/noderig https://github.com/ovh/tsl https://github.com/ovh/ovh-warp10-datasource https://github.com/ovh/ovh-tsl-datasource … DEVOPS D-DAY #5 @LostInBrittany Involved in: ● ● ● ● Warp10 community Apache Hbase/Flink development Prometheus/InfluxData discussions TS Query Language Working group
Conclusion That’s all folks! DEVOPS D-DAY #5 @LostInBrittany
