LULLABOT Custom Drupal Data Migration: A Georgia GovHUB Story FLORIDA DC 2020 + + SIDES GEORGIA GOVHUB MIGRATION APRIL 1
A presentation at Florida Drupalcamp 2020 in February 2020 in Orlando, FL, USA by April Sides
LULLABOT Custom Drupal Data Migration: A Georgia GovHUB Story FLORIDA DC 2020 + + SIDES GEORGIA GOVHUB MIGRATION APRIL 1
weekbeforenext weekbeforenext aprilsides GEORGIA GOVHUB MIGRATION + S R. D E V E L O P E R + L U L L A B O T FLORIDA DC 2020 April Sides 2
3 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
4 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
GEORGIA GOVHUB MIGRATION Lullabot Migration Team 5 + FLORIDA DC 2020
GEORGIA GOVHUB MIGRATION Digital Services Georgia 6 + FLORIDA DC 2020
⭑ Discovery and Planning ⭑ Strategies and Workflow ⭑ Magical Nerdery FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + What we will cover 7
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Discovery and Planning 8 Photo by Elodie Oudot on Unsplash
Drupal 7 Multisite (85+ sites) ⭑ Drupal 8 Multisite (~6 at a time) ⭑ Hosted on Acquia ⭑ Hosted on Acquia ⭑ 27 Content Types (15 migrated) ⭑ 20 Content Types (14 populated) ⭑ 14 Taxonomy Vocabularies (9 migrated) ⭑ 17 Taxonomy Vocabularies (10 populated) ⭑ Paragraphs, Field Collections and Entity Embeds ⭑ “Micro-content” Types and Entity Embeds FLORIDA DC 2020 ⭑ + Destination GEORGIA GOVHUB MIGRATION Source 9
10 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
11 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
12 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
13 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
14 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION https://www.drupal.org/project/migration_planner 15
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION SQueaLer 16 Photo by mali maeder from Pexels
✩ ✩ ✩ ✩ ✩ ✩ ✩ ✩ Custom classes Custom forms Email links Style tags and attributes Social network links Video iFrames Links to PDFs File Lists FLORIDA DC 2020 WYSIWYG string queries: ✩ Object tags ✩ Image tags ✩ Embedded entities ✩ Absolute links ✩ Span tags ✩ iFrames ✩ Script tags ✩ Tables + ⭑ GEORGIA GOVHUB MIGRATION SQueaLer scans 17
⭑ ⭑ ⭑ ⭑ ⭑ ⭑ ⭑ Metatag string queries: ✩ Description ✩ Abstract ✩ Token ✩ Creator ✩ Canonical ✩ Title ✩ Image ✩ Keyword ✩ Video ✩ Twitter ✩ Open Graph FLORIDA DC 2020 ⭑ Content by date thresholds Parent Content Unpublished Content Content with documents Paragraphs Field Collections Specific field values Redirects + ⭑ GEORGIA GOVHUB MIGRATION SQueaLer reports (continued) 18
/sites/[SITENAME].georgia.gov/files/ga_squealer_reports/ [SITENAME].georgia.gov_YYYY-MM-DD.xlsx GEORGIA GOVHUB MIGRATION + $ drush @[SITENAME].[ENVIRONMENT] squeal FLORIDA DC 2020 SQueaLer command 19
20 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION https://www.drupal.org/project/squealer 21
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Strategies and Workflow 22 Photo by chuttersnap on Unsplash
Migrated ⭑ ⭑ ⭑ Structural elements: ⭑ Data elements: ✩ Content types and field definitions ✩ Select nodes and field data ✩ Vocabularies ✩ ✩ Paragraph and field collection bundles Select taxonomy terms Views Webform submissions ⭑ Menus ⭑ Webforms ⭑ Files FLORIDA DC 2020 Did not migrate + ✅ GEORGIA GOVHUB MIGRATION ❌ 23
⭑ ⭑ Contributed: ✩ Migrate migrate ✩ Migrate Files (extended) migrate_file ✩ Migrate Drupal migrate_drupal ✩ Migrate Plus migrate_plus ✩ Migrate Source CSV migrate_source_csv ✩ Migrate Tools migrate_tools ✩ Drupal Upgrade migrate_upgrade ✩ Migrate Source UI migrate_source_ui Custom: ✩ ga_migrate ✩ ga_migrate_site ✩ ga_migrate_source_ui + Core: GEORGIA GOVHUB MIGRATION ⭑ FLORIDA DC 2020 Migration modules 24
✩ ⭑ QA environment: ✩ ⭑ Lando Tugboat.qa DevOps magic: ✩ CircleCI ✩ Quay.io FLORIDA DC 2020 Local development environment: + ⭑ GEORGIA GOVHUB MIGRATION Development tools 25
GEORGIA GOVHUB MIGRATION https://tugboat.qa 26 + FLORIDA DC 2020
✩ Basic fields ✩ Rich fields ✩ File/image fields ✩ Paragraphs/Field Collections ✩ WYSIWYG cleanup FLORIDA DC 2020 Focus on field mapping by complexity/content type: + ⭑ GEORGIA GOVHUB MIGRATION Development workflow 27
Create/edit migration configuration files directly in config sync directory ⭑ Preserve nids for standalone nodes ⭑ Migrate unpublished content ⭑ Prioritize ability to rollback and re-import ⭑ Use ga_migrate_site for site specific overrides ⭑ Log skips and exceptions using custom logging solution ⭑ Solution order: ✩ Configuration, core and contrib ✩ Custom source/process plugins and services in ga_migrate module ✩ hook_migrate_MIGRATION_ID_prepare_row() GEORGIA GOVHUB MIGRATION + ⭑ FLORIDA DC 2020 Migration development strategy 28
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Custom Logging 29 Photo by Dorelys Smits on Unsplash
$message, // String: detailed message with ids. $migration_id, // String: current migration id. $needs_fix, // Boolean: Does this need to be fixed. $severity, // String: ‘warning’, ‘notice’, or ‘error’. $category, // String: Short descriptor. $row_id // Integer: The current row id. ); GEORGIA GOVHUB MIGRATION + $audience, // String: ‘DSGa’ or ‘dev’. FLORIDA DC 2020 ga_migrate_log( 30
31 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Migration Phases 32 Photo by Farzad Mohsenvand on Unsplash
Add/remove sites from Tugboat ⭑ Run migrations on Tugboat ⭑ Client QA’s migrations on Tugboat ⭑ Development team fixes issues ⭑ Run migration on Production ⭑ Client prepares sites for launch ⭑ Sites are launched GEORGIA GOVHUB MIGRATION + ⭑ FLORIDA DC 2020 Migration phases 33
GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Magical Nerdery 34
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Site Specific Overrides 35 Photo by James Wheeler from Pexels
/sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + /modules/custom/ga_migrate_site 36
ga_migrate_site: 0 … core.extension.yml GEORGIA GOVHUB MIGRATION + ga_migrate: 0 FLORIDA DC 2020 … 37
// The date unit that accompanies the threshold value. const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD_UNIT = ‘y’; // The site node skip list. const GA_MIGRATE_SITE_SKIP_LIST = []; } /modules/custom/ga_migrate_site/src/Plugin/GaMigrateSiteInterface.php FLORIDA DC 2020 const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD = 3; + // Press_release nodes with release date greater // than threshold are trashed. GEORGIA GOVHUB MIGRATION interface GaMigrateSiteInterface { 38
// Implements hook_migrate_MIGRATION_ID_prepare_row(). /sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site/ga_migrate_site.module FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + // Implements hook_migrate_prepare_row(). 39
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Nested Paragraphs 40 Photo by Victor Larracuente on Unsplash
GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Container Paragraphs 41
Content Paragraphs GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Container Paragraphs 42
field_content two_column gta_paragraph_image one_column gta_paragraph_text_area GEORGIA GOVHUB MIGRATION + gta_paragraph_related_links FLORIDA DC 2020 node 43
✩ ⭑ Includes entity embeds Stack new content in Body field FLORIDA DC 2020 Convert Paragraphs into to Drupal 8 markup + ⭑ GEORGIA GOVHUB MIGRATION Paragraphs strategy 44
source: - field_content micro_content: field_content: paragraphs_item … FLORIDA DC 2020 plugin: ga_micro_content_to_text + prepared_field_content: GEORGIA GOVHUB MIGRATION … 45
What should the Drupal 8 markup be for this paragraph when placed in the WYSIWYG Body field? ✩ Text markup ✩ Entity embed code ✩ File download link ✩ Alignment adjustments FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION What do you mean by “render”? 46
field_content two_column 6 3 1 gta_paragraph_image one_column 2 5 gta_paragraph_text_area 4 GEORGIA GOVHUB MIGRATION + gta_paragraph_related_links FLORIDA DC 2020 node 47
body/0/value: source: - ‘@prepared_field_content’ - body/0/value … GEORGIA GOVHUB MIGRATION + plugin: concat FLORIDA DC 2020 … 48
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Circular Dependencies 49 Photo by Anita Austvika on Unsplash
link_collection node field_related_links field_rich_links Link Node reference Node reference Node reference Node reference Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 50
link_collection node nid topic_page node body GEORGIA GOVHUB MIGRATION + entity embed code FLORIDA DC 2020 site_page node ga_d7_node_site_page 51
GEORGIA GOVHUB MIGRATION prepared_field_related_links: FLORIDA DC 2020 … ga_d7_node_site_page 52
body/0/value: - source: - body/0/value - ‘@prepared_field_related_links’ … ga_d7_node_site_page GEORGIA GOVHUB MIGRATION + plugin: concat FLORIDA DC 2020 … 53
✩ �� ga_d7_site_page_field_related_links �� FLORIDA DC 2020 ga_d7_node_site_page + ⭑ GEORGIA GOVHUB MIGRATION Dependencies 54
link_collection node field_related_links field_rich_links Link blog_post reference blog_post reference site_page reference NULL Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 55
… process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page … + source: field_related_links GEORGIA GOVHUB MIGRATION plugin: sub_process FLORIDA DC 2020 site_page_stub_entity_references: 56
link_collection node field_related_links field_rich_links Link blog_post reference blog_post reference site_page reference topic_page reference Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 57
… process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page … + source: field_related_links GEORGIA GOVHUB MIGRATION plugin: sub_process FLORIDA DC 2020 site_page_stub_entity_references: 58
✩ ⭑ �� �� �� ga_d7_site_page_field_related_links ga_d7_site_page_field_related_links ✩ ga_d7_node_blog_post ✩ “ga_d7_node_site_page” �� FLORIDA DC 2020 ga_d7_node_site_page + ⭑ GEORGIA GOVHUB MIGRATION Dependencies �� 59
The non-specified dependency migration id is alphabetically before the current migration: ✩ ga_d7_node_site_page ✩ ga_d7_site_page_field_related_links GEORGIA GOVHUB MIGRATION ⭑ FLORIDA DC 2020 It works if… 60
⭑ ⭑ ✩ ga_d7_index_list_field_related_links ✩ ga_d7_node_index_list The fix hack: ✩ ga_d7_1_node_index_list ✩ ga_d7_index_list_field_related_links Migration Dependency When Stubbing Content: https://www.drupal.org/project/drupal/issues/3024634 FLORIDA DC 2020 The non-specified dependency migration id is alphabetically after the current migration: + ⭑ GEORGIA GOVHUB MIGRATION It doesn’t work if… 61
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION WYSIWYG DOM Processing 62 Photo by Iker Urteaga on Unsplash
GEORGIA GOVHUB MIGRATION … FLORIDA DC 2020 body/0/value: 63
\DOMDocument GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Xdebug 64
⭑ Loop and collect the items to alter, replace, or remove in an array. ⭑ Loop through collection array and alter, replace or remove each element. FLORIDA DC 2020 Get elements from the \DOMDocument object. + ⭑ GEORGIA GOVHUB MIGRATION \DOMDocument processing 65
// Get elements by tag. foreach ($elements as $element) { // Collect elements based on conditions. $remove_elements[] = $element; } } // Remove or replace elements. foreach ($remove_elements as $element) { $this->removeElementAndContent($element); } + if ($elements->length > 0) { GEORGIA GOVHUB MIGRATION $remove_elements = []; FLORIDA DC 2020 $elements = $dom->getElementsByTagName(‘*’); 66
⭑ D7 <img> tag → D8 embedded media entity ⭑ D7 link to file → D8 file download link ⭑ D7 <iframe> tag → D8 embedded Code Block node or media entity ⭑ General text filtering based on D8 field text format FLORIDA DC 2020 D7 embedded entity → D8 embed syntax + ⭑ GEORGIA GOVHUB MIGRATION WYSIWYG clean up 67
GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 We made it! 👏 68
Georgia GovHub: a case study by Darren Peterson at Drupal GovCon 2019 ✩ ⭑ This is also coming to DrupalCon Minneapolis 2020! A forest of designs without subthemes: Implementing Georgia.gov’s front-end in Drupal 8 by Marc Drummond at Drupal GovCon 2019 FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION More about the Georgia.gov project 69
Real Life Data Migrations on the Lullabot Podcast ⭑ An Overview for Migrating Drupal Sites to 8 by Juampy NR ⭑ Running and Testing Drupal 8 Migrations in CircleCI by Juampy NR ⭑ 31 Days of Drupal Migrations with Mauricio Dinarte on the Lullabot Podcast ⭑ Managing Authentication During API Migrations by April Sides ⭑ and more to come! FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION More about migration from Lullabot 70
GEORGIA GOVHUB MIGRATION + Thank you. 71 FLORIDA DC 2020
