Custom Drupal Data Migration: A Georgia GovHUB Story

A presentation at MidCamp 2020 in March 2020 in Chicago, IL, USA by April Sides

Slide 1

Slide 1

Slide 2

Slide 2

weekbeforenext weekbeforenext aprilsides

Slide 3

Slide 3

Slide 4

Slide 4

Slide 5

Slide 5

Slide 6

Slide 6

Slide 7

Slide 7

⭑ Discovery and Planning ⭑ Strategies and Workflow ⭑ Magical Nerdery

Slide 8

Slide 8

Photo by Elodie Oudot on Unsplash

Slide 9

Slide 9

⭑ Drupal 7 Multisite (85+ sites) ⭑ Drupal 8 Multisite (~6 at a time) ⭑ Hosted on Acquia ⭑ Hosted on Acquia ⭑ 27 Content Types (15 migrated) ⭑ 20 Content Types (14 populated) ⭑ 14 Taxonomy Vocabularies (9 migrated) ⭑ 17 Taxonomy Vocabularies (10 populated) ⭑ Paragraphs, Field Collections and Entity Embeds ⭑ “Micro-content” Types and Entity Embeds

Slide 10

Slide 10

Slide 11

Slide 11

Slide 12

Slide 12

Slide 13

Slide 13

Slide 14

Slide 14

Slide 15

Slide 15

https://www.drupal.org/project/migration_planner

Slide 16

Slide 16

Photo by mali maeder from Pexels

Slide 17

Slide 17

⭑ WYSIWYG string queries: ✩ Object tags ✩ Image tags ✩ Embedded entities ✩ Absolute links ✩ Span tags ✩ iFrames ✩ Script tags ✩ Tables ✩ ✩ ✩ ✩ ✩ ✩ ✩ ✩ Custom classes Custom forms Email links Style tags and attributes Social network links Video iFrames Links to PDFs File Lists

Slide 18

Slide 18

⭑ ⭑ ⭑ ⭑ ⭑ ⭑ ⭑ ⭑ Content by date thresholds Parent Content Unpublished Content Content with documents Paragraphs Field Collections Specific field values Redirects ⭑ Metatag string queries: ✩ Description ✩ Abstract ✩ Token ✩ Creator ✩ Canonical ✩ Title ✩ Image ✩ Keyword ✩ Video ✩ Twitter ✩ Open Graph

Slide 19

Slide 19

$ drush @[SITENAME].[ENVIRONMENT] squeal /sites/[SITENAME].georgia.gov/files/ga_squealer_reports/ [SITENAME].georgia.gov_YYYY-MM-DD.xlsx

Slide 20

Slide 20

Slide 21

Slide 21

https://www.drupal.org/project/squealer

Slide 22

Slide 22

Photo by chuttersnap on Unsplash

Slide 23

Slide 23

❌ ⭑ ⭑ ⭑ ✅ Structural elements: ⭑ Data elements: ✩ Content types and field definitions ✩ Select nodes and field data ✩ Vocabularies ✩ ✩ Paragraph and field collection bundles Select taxonomy terms Views Webform submissions ⭑ Menus ⭑ Webforms ⭑ Files

Slide 24

Slide 24

⭑ ⭑ Core: ⭑ Contributed: ✩ Migrate migrate ✩ Migrate Files (extended) migrate_file ✩ Migrate Drupal migrate_drupal ✩ Migrate Plus migrate_plus ✩ Migrate Source CSV migrate_source_csv ✩ Migrate Tools migrate_tools ✩ Drupal Upgrade migrate_upgrade ✩ Migrate Source UI migrate_source_ui Custom: ✩ ga_migrate ✩ ga_migrate_site ✩ ga_migrate_source_ui

Slide 25

Slide 25

⭑ Local development environment: ✩ ⭑ QA environment: ✩ ⭑ Lando Tugboat.qa DevOps magic: ✩ CircleCI ✩ Quay.io

Slide 26

Slide 26

https://tugboat.qa

Slide 27

Slide 27

⭑ Focus on field mapping by complexity/content type: ✩ Basic fields ✩ Rich fields ✩ File/image fields ✩ Paragraphs/Field Collections ✩ WYSIWYG cleanup

Slide 28

Slide 28

⭑ Create/edit migration configuration files directly in config sync directory ⭑ Preserve nids for standalone nodes ⭑ Migrate unpublished content ⭑ Prioritize ability to rollback and re-import ⭑ Use ga_migrate_site for site specific overrides ⭑ Log skips and exceptions using custom logging solution ⭑ Solution order: ✩ Configuration, core and contrib ✩ Custom source/process plugins and services in ga_migrate module ✩ hook_migrate_MIGRATION_ID_prepare_row()

Slide 29

Slide 29

Photo by Dorelys Smits on Unsplash

Slide 30

Slide 30

ga_migrate_log( $message, // String: detailed message with ids. $migration_id, // String: current migration id. $audience, // String: ‘DSGa’ or ‘dev’. $needs_fix, // Boolean: Does this need to be fixed. $severity, // String: ‘warning’, ‘notice’, or ‘error’. $category, // String: Short descriptor. $row_id // Integer: The current row id. );

Slide 31

Slide 31

Slide 32

Slide 32

Photo by Farzad Mohsenvand on Unsplash

Slide 33

Slide 33

⭑ Add/remove sites from Tugboat ⭑ Run migrations on Tugboat ⭑ Client QA’s migrations on Tugboat ⭑ Development team fixes issues ⭑ Run migration on Production ⭑ Client prepares sites for launch ⭑ Sites are launched

Slide 34

Slide 34

Slide 35

Slide 35

Photo by James Wheeler from Pexels

Slide 36

Slide 36

/modules/custom/ga_migrate_site /sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site

Slide 37

Slide 37

… ga_migrate: 0 ga_migrate_site: 0 … core.extension.yml

Slide 38

Slide 38

interface GaMigrateSiteInterface { // Press_release nodes with release date greater // than threshold are trashed. const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD = 3; // The date unit that accompanies the threshold value. const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD_UNIT = ‘y’; // The site node skip list. const GA_MIGRATE_SITE_SKIP_LIST = []; } /modules/custom/ga_migrate_site/src/Plugin/GaMigrateSiteInterface.php

Slide 39

Slide 39

// Implements hook_migrate_prepare_row(). // Implements hook_migrate_MIGRATION_ID_prepare_row(). /sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site/ga_migrate_site.module

Slide 40

Slide 40

Photo by Victor Larracuente on Unsplash

Slide 41

Slide 41

Slide 42

Slide 42

Slide 43

Slide 43

node field_content two_column gta_paragraph_related_links gta_paragraph_image one_column gta_paragraph_text_area

Slide 44

Slide 44

⭑ Convert Paragraphs into to Drupal 8 markup ✩ ⭑ Includes entity embeds Stack new content in Body field

Slide 45

Slide 45

… prepared_field_content: plugin: ga_micro_content_to_text source: - field_content micro_content: field_content: paragraphs_item …

Slide 46

Slide 46

⭑ What should the Drupal 8 markup be for this paragraph when placed in the WYSIWYG Body field? ✩ Text markup ✩ Entity embed code ✩ File download link ✩ Alignment adjustments

Slide 47

Slide 47

node field_content two_column 6 3 gta_paragraph_related_links gta_paragraph_image one_column 2 5 gta_paragraph_text_area 4 1

Slide 48

Slide 48

… body/0/value: plugin: concat source: - ‘@prepared_field_content’ - body/0/value …

Slide 49

Slide 49

Photo by Anita Austvika on Unsplash

Slide 50

Slide 50

site_page node link_collection node field_related_links field_rich_links Link Link Node reference Node reference Node reference Node reference Link Link Link Link ga_d7_site_page_field_related_links

Slide 51

Slide 51

site_page node link_collection node nid topic_page node body entity embed code ga_d7_node_site_page

Slide 52

Slide 52

… prepared_field_related_links: plugin: migration_lookup migration: ga_d7_site_page_field_related_links source: nid no_stub: true plugin: ga_entity_embed_code entity_type: node … ga_d7_node_site_page

Slide 53

Slide 53

… body/0/value: plugin: concat source: - body/0/value - ‘@prepared_field_related_links’ … ga_d7_node_site_page

Slide 54

Slide 54

⭑ ga_d7_node_site_page ✩ �� ga_d7_site_page_field_related_links ��

Slide 55

Slide 55

site_page node link_collection node field_related_links field_rich_links Link Link blog_post reference blog_post reference site_page reference NULL Link Link Link Link ga_d7_site_page_field_related_links

Slide 56

Slide 56

… site_page_stub_entity_references: plugin: sub_process source: field_related_links process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page …

Slide 57

Slide 57

site_page node link_collection node field_related_links field_rich_links Link Link blog_post reference blog_post reference site_page reference topic_page reference Link Link Link Link ga_d7_site_page_field_related_links

Slide 58

Slide 58

… site_page_stub_entity_references: plugin: sub_process source: field_related_links process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page …

Slide 59

Slide 59

⭑ ga_d7_node_site_page ✩ ⭑ �� �� �� ga_d7_site_page_field_related_links ga_d7_site_page_field_related_links ✩ ga_d7_node_blog_post ✩ “ga_d7_node_site_page” �� ��

Slide 60

Slide 60

⭑ The non-specified dependency migration id is alphabetically before the current migration: ✩ ga_d7_node_site_page ✩ ga_d7_site_page_field_related_links

Slide 61

Slide 61

⭑ ⭑ ⭑ The non-specified dependency migration id is alphabetically after the current migration: ✩ ga_d7_index_list_field_related_links ✩ ga_d7_node_index_list The fix hack: ✩ ga_d7_1_node_index_list ✩ ga_d7_index_list_field_related_links Migration Dependency When Stubbing Content: https://www.drupal.org/project/drupal/issues/3024634

Slide 62

Slide 62

👏

Slide 63

Slide 63

⭑ Georgia GovHub: a case study by Darren Peterson at Drupal GovCon 2019 ✩ ⭑ This is also coming to DrupalCon Minneapolis 2020! A forest of designs without subthemes: Implementing Georgia.gov’s front-end in Drupal 8 by Marc Drummond at Drupal GovCon 2019

Slide 64

Slide 64

⭑ Real Life Data Migrations on the Lullabot Podcast ⭑ An Overview for Migrating Drupal Sites to 8 by Juampy NR ⭑ Running and Testing Drupal 8 Migrations in CircleCI by Juampy NR ⭑ 31 Days of Drupal Migrations with Mauricio Dinarte on the Lullabot Podcast ⭑ Managing Authentication During API Migrations by April Sides ⭑ and more to come!

Slide 65

Slide 65