A presentation at Florida Drupal Camp in in Orlando, FL, USA by April Sides
Data lake: how Red Hat maintains data quality across multiple Drupal sites Florida Drupalcamp 2023 1 Melissa Bent April Sides Senior Software Engineer Senior Software Engineer
Data lake: how Red Hat maintains data quality across multiple Drupal sites 2 Problem Discovery Solution Integration ● ● ● ●
Problem 3
Problem Redhat.com is built on Drupal and single-page applications. 4
Problem Organization-level data should be consistent and accurate across redhat.com. 5
Problem Organization-level data is duplicated and managed by different teams. 6
Problem Managing data quality across redhat.com is a manual process. 7
Problem There has to be a better way to share data across the redhat.com ecosystem. 8
Discovery 9
Discovery Requirements ▸ ▸ ▸ ▸ Share data in a scalable and maintainable way Connect with different tech stacks Serve as a single source of truth Provide flexible data model/schema 10 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Discovery Data Repository Types ▸ Relational database ▸ Data warehouse ･ Data mart ･ Operational data store ▸ Data lake 11 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Discovery Relational database - Structured transactional data - Main features: - Data normalization - Compliancy Example: Drupal database 12
Discovery Data warehouse - Business intelligence focus - Structured data - Schema-on-write 13
Discovery Data Lake - Flexible structure - Scalable - Schema-on-read - Low maintenance 14
Discovery Data Lake Challenges ▸ ▸ ▸ ▸ ▸ Data rot Data governance Data compliance Data security Data availability 15 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Discovery Data Lake Advantages ▸ Multiple data sources—Consistent access ▸ Protection against downtime ▸ Provides an additional, query-level, caching layer for production 16 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Solution 17
Solution Tech Stack ▸ Database Backend: MongoDB ▸ Indexing: Search API + Custom module ▸ Retrieval ･ GraphQL ･ Direct query 18 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Solution Indexing ▸ ▸ ▸ ▸ Custom Search API Backend Search API’s index management Flexible schema for each data source Drupal provides access control, the data model, and the editorial experience 19 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Solution Retrieval ▸ Single-page applications: GraphQL ▸ Drupal: PHP MongoDB Driver 20 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration 21
Integration: Products 22
Integration: Products Product Experience access.redhat.com/products 23 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration: Products Customer Portal Products ▸ Customer Portal helps our customers get the most out of their subscriptions ▸ Product information is core to our data organization ▸ Multiple teams and sites combine to make what is the Customer Portal 24 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration 25 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration 26 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration 27 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration: Products Strategy Product data managed in Drupal Indexed to the Data Lake Queried via GraphQL Page built via GitLab pipeline (statically generated) ▸ Refreshed every 30 minutes ▸ ▸ ▸ ▸ 28 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration: Learning Paths 29
Integration: Learning Paths Learning Path A curated collection of content, directing users to learn more about a particular topic or product. 30
Integration: Learning Paths developers.redhat.com 31 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration: Learning Paths developers.redhat.com Article 32 Data lake: how Red Hat maintains data quality across multiple Drupal sites Cheat Sheet
Integration: Learning Paths developers.redhat.com Article in Learning Path 33 Data lake: how Red Hat maintains data quality across multiple Drupal sites Cheat Sheet in Learning Path
Integration: Learning Paths Learning Path content type 34 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration: Learning Paths 35 Data lake: how Red Hat maintains data quality across multiple Drupal sites Resource content type
Integration: Learning Paths developers.redhat.com Article in Learning Path 36 Data lake: how Red Hat maintains data quality across multiple Drupal sites Cheat Sheet in Learning Path
Integration: Learning Paths developers.redhat.com Article in Learning Path 37 Data lake: how Red Hat maintains data quality across multiple Drupal sites Cheat Sheet in Learning Path
Integration: Learning Paths Resource displays hook_preprocess_node() 38 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration: Learning Paths Shared module Learning Paths shared module 39 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration: Learning Paths 40 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration: Future Plans 41
Integration Content Syndicated Patterns ▸ “Shared patterns” with embedded content for banners, footers, marketing content, etc. 42 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Integration Customer Portal ▸ Autobuilding product pages from the Data Lake (currently are Drupal nodes) ▸ Standardizing Product taxonomy across Customer Portal microsites ▸ Integrating with external systems (product life cycle, case management, developers.redhat.com, etc.) 43 Data lake: how Red Hat maintains data quality across multiple Drupal sites
Thank you Questions? Melissa Bent April Sides Red Hat linkedin.com/in/melissabent linkedin.com/in/aprilsides linkedin.com/company/red-hat twitter.com/merauluka twitter.com/weekbeforenext youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat 44
This was co-presented with Melissa Bent.
Data accuracy and consistency is an important goal for any organization.
Maintaining data quality across multiple websites and applications (Drupal or otherwise), with different teams managing the same data in multiple systems, becomes complex and difficult to manage. Having a pool of data becomes an attractive solution to resolve some of these issues and allow for greater transparency and consistency across an organization. But, creating a scalable, reliable, and useful system brings its own challenges.
Join us, as we explore several ways that Red Hat is using a data lake architecture to share data between different Drupal sites.
We’ll cover:
The following resources were mentioned during the presentation or are useful additional information.
for free. You
can too.