LULLABOT Custom Drupal Data Migration: A Georgia GovHUB Story FLORIDA DC 2020 + + SIDES GEORGIA GOVHUB MIGRATION APRIL 1

weekbeforenext weekbeforenext aprilsides GEORGIA GOVHUB MIGRATION + S R. D E V E L O P E R + L U L L A B O T FLORIDA DC 2020 April Sides 2

3 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

4 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

GEORGIA GOVHUB MIGRATION Lullabot Migration Team 5 + FLORIDA DC 2020

GEORGIA GOVHUB MIGRATION Digital Services Georgia 6 + FLORIDA DC 2020

⭑ Discovery and Planning ⭑ Strategies and Workflow ⭑ Magical Nerdery FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + What we will cover 7

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Discovery and Planning 8 Photo by Elodie Oudot on Unsplash

Drupal 7 Multisite (85+ sites) ⭑ Drupal 8 Multisite (~6 at a time) ⭑ Hosted on Acquia ⭑ Hosted on Acquia ⭑ 27 Content Types (15 migrated) ⭑ 20 Content Types (14 populated) ⭑ 14 Taxonomy Vocabularies (9 migrated) ⭑ 17 Taxonomy Vocabularies (10 populated) ⭑ Paragraphs, Field Collections and Entity Embeds ⭑ “Micro-content” Types and Entity Embeds FLORIDA DC 2020 ⭑ + Destination GEORGIA GOVHUB MIGRATION Source 9

10 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

11 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

12 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

13 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

14 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION https://www.drupal.org/project/migration_planner 15

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION SQueaLer 16 Photo by mali maeder from Pexels

✩ ✩ ✩ ✩ ✩ ✩ ✩ ✩ Custom classes Custom forms Email links Style tags and attributes Social network links Video iFrames Links to PDFs File Lists FLORIDA DC 2020 WYSIWYG string queries: ✩ Object tags ✩ Image tags ✩ Embedded entities ✩ Absolute links ✩ Span tags ✩ iFrames ✩ Script tags ✩ Tables + ⭑ GEORGIA GOVHUB MIGRATION SQueaLer scans 17

⭑ ⭑ ⭑ ⭑ ⭑ ⭑ ⭑ Metatag string queries: ✩ Description ✩ Abstract ✩ Token ✩ Creator ✩ Canonical ✩ Title ✩ Image ✩ Keyword ✩ Video ✩ Twitter ✩ Open Graph FLORIDA DC 2020 ⭑ Content by date thresholds Parent Content Unpublished Content Content with documents Paragraphs Field Collections Specific field values Redirects + ⭑ GEORGIA GOVHUB MIGRATION SQueaLer reports (continued) 18

/sites/[SITENAME].georgia.gov/files/ga_squealer_reports/ [SITENAME].georgia.gov_YYYY-MM-DD.xlsx GEORGIA GOVHUB MIGRATION + $ drush @[SITENAME].[ENVIRONMENT] squeal FLORIDA DC 2020 SQueaLer command 19

20 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION https://www.drupal.org/project/squealer 21

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Strategies and Workflow 22 Photo by chuttersnap on Unsplash

Migrated ⭑ ⭑ ⭑ Structural elements: ⭑ Data elements: ✩ Content types and field definitions ✩ Select nodes and field data ✩ Vocabularies ✩ ✩ Paragraph and field collection bundles Select taxonomy terms Views Webform submissions ⭑ Menus ⭑ Webforms ⭑ Files FLORIDA DC 2020 Did not migrate + ✅ GEORGIA GOVHUB MIGRATION ❌ 23

⭑ ⭑ Contributed: ✩ Migrate migrate ✩ Migrate Files (extended) migrate_file ✩ Migrate Drupal migrate_drupal ✩ Migrate Plus migrate_plus ✩ Migrate Source CSV migrate_source_csv ✩ Migrate Tools migrate_tools ✩ Drupal Upgrade migrate_upgrade ✩ Migrate Source UI migrate_source_ui Custom: ✩ ga_migrate ✩ ga_migrate_site ✩ ga_migrate_source_ui + Core: GEORGIA GOVHUB MIGRATION ⭑ FLORIDA DC 2020 Migration modules 24

✩ ⭑ QA environment: ✩ ⭑ Lando Tugboat.qa DevOps magic: ✩ CircleCI ✩ Quay.io FLORIDA DC 2020 Local development environment: + ⭑ GEORGIA GOVHUB MIGRATION Development tools 25

GEORGIA GOVHUB MIGRATION https://tugboat.qa 26 + FLORIDA DC 2020

✩ Basic fields ✩ Rich fields ✩ File/image fields ✩ Paragraphs/Field Collections ✩ WYSIWYG cleanup FLORIDA DC 2020 Focus on field mapping by complexity/content type: + ⭑ GEORGIA GOVHUB MIGRATION Development workflow 27

Create/edit migration configuration files directly in config sync directory ⭑ Preserve nids for standalone nodes ⭑ Migrate unpublished content ⭑ Prioritize ability to rollback and re-import ⭑ Use ga_migrate_site for site specific overrides ⭑ Log skips and exceptions using custom logging solution ⭑ Solution order: ✩ Configuration, core and contrib ✩ Custom source/process plugins and services in ga_migrate module ✩ hook_migrate_MIGRATION_ID_prepare_row() GEORGIA GOVHUB MIGRATION + ⭑ FLORIDA DC 2020 Migration development strategy 28

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Custom Logging 29 Photo by Dorelys Smits on Unsplash

$message, // String: detailed message with ids. $migration_id, // String: current migration id. $needs_fix, // Boolean: Does this need to be fixed. $severity, // String: ‘warning’, ‘notice’, or ‘error’. $category, // String: Short descriptor. $row_id // Integer: The current row id. ); GEORGIA GOVHUB MIGRATION + $audience, // String: ‘DSGa’ or ‘dev’. FLORIDA DC 2020 ga_migrate_log( 30

31 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Migration Phases 32 Photo by Farzad Mohsenvand on Unsplash

Add/remove sites from Tugboat ⭑ Run migrations on Tugboat ⭑ Client QA’s migrations on Tugboat ⭑ Development team fixes issues ⭑ Run migration on Production ⭑ Client prepares sites for launch ⭑ Sites are launched GEORGIA GOVHUB MIGRATION + ⭑ FLORIDA DC 2020 Migration phases 33

GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Magical Nerdery 34

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Site Specific Overrides 35 Photo by James Wheeler from Pexels

/sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + /modules/custom/ga_migrate_site 36

ga_migrate_site: 0 … core.extension.yml GEORGIA GOVHUB MIGRATION + ga_migrate: 0 FLORIDA DC 2020 … 37

// The date unit that accompanies the threshold value. const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD_UNIT = ‘y’; // The site node skip list. const GA_MIGRATE_SITE_SKIP_LIST = []; } /modules/custom/ga_migrate_site/src/Plugin/GaMigrateSiteInterface.php FLORIDA DC 2020 const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD = 3; + // Press_release nodes with release date greater // than threshold are trashed. GEORGIA GOVHUB MIGRATION interface GaMigrateSiteInterface { 38

// Implements hook_migrate_MIGRATION_ID_prepare_row(). /sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site/ga_migrate_site.module FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + // Implements hook_migrate_prepare_row(). 39

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Nested Paragraphs 40 Photo by Victor Larracuente on Unsplash

GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Container Paragraphs 41

Content Paragraphs GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Container Paragraphs 42

field_content two_column gta_paragraph_image one_column gta_paragraph_text_area GEORGIA GOVHUB MIGRATION + gta_paragraph_related_links FLORIDA DC 2020 node 43

✩ ⭑ Includes entity embeds Stack new content in Body field FLORIDA DC 2020 Convert Paragraphs into to Drupal 8 markup + ⭑ GEORGIA GOVHUB MIGRATION Paragraphs strategy 44

source: - field_content micro_content: field_content: paragraphs_item … FLORIDA DC 2020 plugin: ga_micro_content_to_text + prepared_field_content: GEORGIA GOVHUB MIGRATION … 45

What should the Drupal 8 markup be for this paragraph when placed in the WYSIWYG Body field? ✩ Text markup ✩ Entity embed code ✩ File download link ✩ Alignment adjustments FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION What do you mean by “render”? 46

field_content two_column 6 3 1 gta_paragraph_image one_column 2 5 gta_paragraph_text_area 4 GEORGIA GOVHUB MIGRATION + gta_paragraph_related_links FLORIDA DC 2020 node 47

body/0/value: source: - ‘@prepared_field_content’ - body/0/value … GEORGIA GOVHUB MIGRATION + plugin: concat FLORIDA DC 2020 … 48

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Circular Dependencies 49 Photo by Anita Austvika on Unsplash

link_collection node field_related_links field_rich_links Link Node reference Node reference Node reference Node reference Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 50

link_collection node nid topic_page node body GEORGIA GOVHUB MIGRATION + entity embed code FLORIDA DC 2020 site_page node ga_d7_node_site_page 51

plugin: migration_lookup migration: ga_d7_site_page_field_related_links source: nid no_stub: true plugin: ga_entity_embed_code entity_type: node … +

GEORGIA GOVHUB MIGRATION prepared_field_related_links: FLORIDA DC 2020 … ga_d7_node_site_page 52

body/0/value: - source: - body/0/value - ‘@prepared_field_related_links’ … ga_d7_node_site_page GEORGIA GOVHUB MIGRATION + plugin: concat FLORIDA DC 2020 … 53

✩ �� ga_d7_site_page_field_related_links �� FLORIDA DC 2020 ga_d7_node_site_page + ⭑ GEORGIA GOVHUB MIGRATION Dependencies 54

link_collection node field_related_links field_rich_links Link blog_post reference blog_post reference site_page reference NULL Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 55

… process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page … + source: field_related_links GEORGIA GOVHUB MIGRATION plugin: sub_process FLORIDA DC 2020 site_page_stub_entity_references: 56

link_collection node field_related_links field_rich_links Link blog_post reference blog_post reference site_page reference topic_page reference Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 57

… process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page … + source: field_related_links GEORGIA GOVHUB MIGRATION plugin: sub_process FLORIDA DC 2020 site_page_stub_entity_references: 58

✩ ⭑ �� �� �� ga_d7_site_page_field_related_links ga_d7_site_page_field_related_links ✩ ga_d7_node_blog_post ✩ “ga_d7_node_site_page” �� FLORIDA DC 2020 ga_d7_node_site_page + ⭑ GEORGIA GOVHUB MIGRATION Dependencies �� 59

The non-specified dependency migration id is alphabetically before the current migration: ✩ ga_d7_node_site_page ✩ ga_d7_site_page_field_related_links GEORGIA GOVHUB MIGRATION ⭑ FLORIDA DC 2020 It works if… 60

⭑ ⭑ ✩ ga_d7_index_list_field_related_links ✩ ga_d7_node_index_list The fix hack: ✩ ga_d7_1_node_index_list ✩ ga_d7_index_list_field_related_links Migration Dependency When Stubbing Content: https://www.drupal.org/project/drupal/issues/3024634 FLORIDA DC 2020 The non-specified dependency migration id is alphabetically after the current migration: + ⭑ GEORGIA GOVHUB MIGRATION It doesn’t work if… 61

FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION WYSIWYG DOM Processing 62 Photo by Iker Urteaga on Unsplash

… plugin: dom method: import plugin: ga_wysiwyg_cleanup_dom text_format: expanded plugin: dom method: export … +

GEORGIA GOVHUB MIGRATION … FLORIDA DC 2020 body/0/value: 63

\DOMDocument GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Xdebug 64

⭑ Loop and collect the items to alter, replace, or remove in an array. ⭑ Loop through collection array and alter, replace or remove each element. FLORIDA DC 2020 Get elements from the \DOMDocument object. + ⭑ GEORGIA GOVHUB MIGRATION \DOMDocument processing 65

// Get elements by tag. foreach ($elements as $element) { // Collect elements based on conditions. $remove_elements[] = $element; } } // Remove or replace elements. foreach ($remove_elements as $element) { $this->removeElementAndContent($element); } + if ($elements->length > 0) { GEORGIA GOVHUB MIGRATION $remove_elements = []; FLORIDA DC 2020 $elements = $dom->getElementsByTagName(‘*’); 66

⭑ D7 <img> tag → D8 embedded media entity ⭑ D7 link to file → D8 file download link ⭑ D7 <iframe> tag → D8 embedded Code Block node or media entity ⭑ General text filtering based on D8 field text format FLORIDA DC 2020 D7 embedded entity → D8 embed syntax + ⭑ GEORGIA GOVHUB MIGRATION WYSIWYG clean up 67

GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 We made it! 👏 68

Georgia GovHub: a case study by Darren Peterson at Drupal GovCon 2019 ✩ ⭑ This is also coming to DrupalCon Minneapolis 2020! A forest of designs without subthemes: Implementing Georgia.gov’s front-end in Drupal 8 by Marc Drummond at Drupal GovCon 2019 FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION More about the Georgia.gov project 69

Real Life Data Migrations on the Lullabot Podcast ⭑ An Overview for Migrating Drupal Sites to 8 by Juampy NR ⭑ Running and Testing Drupal 8 Migrations in CircleCI by Juampy NR ⭑ 31 Days of Drupal Migrations with Mauricio Dinarte on the Lullabot Podcast ⭑ Managing Authentication During API Migrations by April Sides ⭑ and more to come! FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION More about migration from Lullabot 70

GEORGIA GOVHUB MIGRATION + Thank you. 71 FLORIDA DC 2020