Data lake: how Red Hat maintains data quality across multiple Drupal sites

Data lake: how Red Hat maintains data quality across multiple Drupal sites Florida Drupalcamp 2023 1 Melissa Bent April Sides Senior Software Engineer Senior Software Engineer

Data lake: how Red Hat maintains data quality across multiple Drupal sites 2 Problem Discovery Solution Integration ● ● ● ●

Problem 3

Problem Redhat.com is built on Drupal and single-page applications. 4

Problem Organization-level data should be consistent and accurate across redhat.com. 5

Problem Organization-level data is duplicated and managed by different teams. 6

Problem Managing data quality across redhat.com is a manual process. 7

Problem There has to be a better way to share data across the redhat.com ecosystem. 8

Discovery 9

Discovery Requirements ▸ ▸ ▸ ▸ Share data in a scalable and maintainable way Connect with different tech stacks Serve as a single source of truth Provide flexible data model/schema 10 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Discovery Data Repository Types ▸ Relational database ▸ Data warehouse ･ Data mart ･ Operational data store ▸ Data lake 11 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Discovery Relational database - Structured transactional data - Main features: - Data normalization - Compliancy Example: Drupal database 12

Discovery Data warehouse - Business intelligence focus - Structured data - Schema-on-write 13

Discovery Data Lake - Flexible structure - Scalable - Schema-on-read - Low maintenance 14

Discovery Data Lake Challenges ▸ ▸ ▸ ▸ ▸ Data rot Data governance Data compliance Data security Data availability 15 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Discovery Data Lake Advantages ▸ Multiple data sources—Consistent access ▸ Protection against downtime ▸ Provides an additional, query-level, caching layer for production 16 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Solution 17

Solution Tech Stack ▸ Database Backend: MongoDB ▸ Indexing: Search API + Custom module ▸ Retrieval ･ GraphQL ･ Direct query 18 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Solution Indexing ▸ ▸ ▸ ▸ Custom Search API Backend Search API’s index management Flexible schema for each data source Drupal provides access control, the data model, and the editorial experience 19 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Solution Retrieval ▸ Single-page applications: GraphQL ▸ Drupal: PHP MongoDB Driver 20 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration 21

Integration: Products 22

Integration: Products Product Experience access.redhat.com/products 23 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration: Products Customer Portal Products ▸ Customer Portal helps our customers get the most out of their subscriptions ▸ Product information is core to our data organization ▸ Multiple teams and sites combine to make what is the Customer Portal 24 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration 25 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration 26 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration 27 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration: Products Strategy Product data managed in Drupal Indexed to the Data Lake Queried via GraphQL Page built via GitLab pipeline (statically generated) ▸ Refreshed every 30 minutes ▸ ▸ ▸ ▸ 28 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration: Learning Paths 29

Integration: Learning Paths Learning Path A curated collection of content, directing users to learn more about a particular topic or product. 30

Integration: Learning Paths developers.redhat.com 31 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration: Learning Paths developers.redhat.com Article 32 Data lake: how Red Hat maintains data quality across multiple Drupal sites Cheat Sheet

Integration: Learning Paths developers.redhat.com Article in Learning Path 33 Data lake: how Red Hat maintains data quality across multiple Drupal sites Cheat Sheet in Learning Path

Integration: Learning Paths Learning Path content type 34 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration: Learning Paths 35 Data lake: how Red Hat maintains data quality across multiple Drupal sites Resource content type

Integration: Learning Paths developers.redhat.com Article in Learning Path 36 Data lake: how Red Hat maintains data quality across multiple Drupal sites Cheat Sheet in Learning Path

Integration: Learning Paths developers.redhat.com Article in Learning Path 37 Data lake: how Red Hat maintains data quality across multiple Drupal sites Cheat Sheet in Learning Path

Integration: Learning Paths Resource displays hook_preprocess_node() 38 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration: Learning Paths Shared module Learning Paths shared module 39 Data lake: how Red Hat maintains data quality across multiple Drupal sites

• Data Lake Learning Paths schema - Data Lake query service - Reusable code: - Services - Controller - EventSubscriber - Blocks - Constraints/Validators

Integration: Learning Paths 40 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration: Future Plans 41

Integration Content Syndicated Patterns ▸ “Shared patterns” with embedded content for banners, footers, marketing content, etc. 42 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Integration Customer Portal ▸ Autobuilding product pages from the Data Lake (currently are Drupal nodes) ▸ Standardizing Product taxonomy across Customer Portal microsites ▸ Integrating with external systems (product life cycle, case management, developers.redhat.com, etc.) 43 Data lake: how Red Hat maintains data quality across multiple Drupal sites

Thank you Questions? Melissa Bent April Sides Red Hat linkedin.com/in/melissabent linkedin.com/in/aprilsides linkedin.com/company/red-hat twitter.com/merauluka twitter.com/weekbeforenext youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat 44