A presentation at Florida Drupalcamp 2020 in in Orlando, FL, USA by April Sides
LULLABOT Custom Drupal Data Migration: A Georgia GovHUB Story FLORIDA DC 2020 + + SIDES GEORGIA GOVHUB MIGRATION APRIL 1
weekbeforenext weekbeforenext aprilsides GEORGIA GOVHUB MIGRATION + S R. D E V E L O P E R + L U L L A B O T FLORIDA DC 2020 April Sides 2
3 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
4 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
GEORGIA GOVHUB MIGRATION Lullabot Migration Team 5 + FLORIDA DC 2020
GEORGIA GOVHUB MIGRATION Digital Services Georgia 6 + FLORIDA DC 2020
⭑ Discovery and Planning ⭑ Strategies and Workflow ⭑ Magical Nerdery FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + What we will cover 7
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Discovery and Planning 8 Photo by Elodie Oudot on Unsplash
Drupal 7 Multisite (85+ sites) ⭑ Drupal 8 Multisite (~6 at a time) ⭑ Hosted on Acquia ⭑ Hosted on Acquia ⭑ 27 Content Types (15 migrated) ⭑ 20 Content Types (14 populated) ⭑ 14 Taxonomy Vocabularies (9 migrated) ⭑ 17 Taxonomy Vocabularies (10 populated) ⭑ Paragraphs, Field Collections and Entity Embeds ⭑ “Micro-content” Types and Entity Embeds FLORIDA DC 2020 ⭑ + Destination GEORGIA GOVHUB MIGRATION Source 9
10 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
11 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
12 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
13 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
14 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION https://www.drupal.org/project/migration_planner 15
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION SQueaLer 16 Photo by mali maeder from Pexels
✩ ✩ ✩ ✩ ✩ ✩ ✩ ✩ Custom classes Custom forms Email links Style tags and attributes Social network links Video iFrames Links to PDFs File Lists FLORIDA DC 2020 WYSIWYG string queries: ✩ Object tags ✩ Image tags ✩ Embedded entities ✩ Absolute links ✩ Span tags ✩ iFrames ✩ Script tags ✩ Tables + ⭑ GEORGIA GOVHUB MIGRATION SQueaLer scans 17
⭑ ⭑ ⭑ ⭑ ⭑ ⭑ ⭑ Metatag string queries: ✩ Description ✩ Abstract ✩ Token ✩ Creator ✩ Canonical ✩ Title ✩ Image ✩ Keyword ✩ Video ✩ Twitter ✩ Open Graph FLORIDA DC 2020 ⭑ Content by date thresholds Parent Content Unpublished Content Content with documents Paragraphs Field Collections Specific field values Redirects + ⭑ GEORGIA GOVHUB MIGRATION SQueaLer reports (continued) 18
/sites/[SITENAME].georgia.gov/files/ga_squealer_reports/ [SITENAME].georgia.gov_YYYY-MM-DD.xlsx GEORGIA GOVHUB MIGRATION + $ drush @[SITENAME].[ENVIRONMENT] squeal FLORIDA DC 2020 SQueaLer command 19
20 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION https://www.drupal.org/project/squealer 21
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Strategies and Workflow 22 Photo by chuttersnap on Unsplash
Migrated ⭑ ⭑ ⭑ Structural elements: ⭑ Data elements: ✩ Content types and field definitions ✩ Select nodes and field data ✩ Vocabularies ✩ ✩ Paragraph and field collection bundles Select taxonomy terms Views Webform submissions ⭑ Menus ⭑ Webforms ⭑ Files FLORIDA DC 2020 Did not migrate + ✅ GEORGIA GOVHUB MIGRATION ❌ 23
⭑ ⭑ Contributed: ✩ Migrate migrate ✩ Migrate Files (extended) migrate_file ✩ Migrate Drupal migrate_drupal ✩ Migrate Plus migrate_plus ✩ Migrate Source CSV migrate_source_csv ✩ Migrate Tools migrate_tools ✩ Drupal Upgrade migrate_upgrade ✩ Migrate Source UI migrate_source_ui Custom: ✩ ga_migrate ✩ ga_migrate_site ✩ ga_migrate_source_ui + Core: GEORGIA GOVHUB MIGRATION ⭑ FLORIDA DC 2020 Migration modules 24
✩ ⭑ QA environment: ✩ ⭑ Lando Tugboat.qa DevOps magic: ✩ CircleCI ✩ Quay.io FLORIDA DC 2020 Local development environment: + ⭑ GEORGIA GOVHUB MIGRATION Development tools 25
GEORGIA GOVHUB MIGRATION https://tugboat.qa 26 + FLORIDA DC 2020
✩ Basic fields ✩ Rich fields ✩ File/image fields ✩ Paragraphs/Field Collections ✩ WYSIWYG cleanup FLORIDA DC 2020 Focus on field mapping by complexity/content type: + ⭑ GEORGIA GOVHUB MIGRATION Development workflow 27
Create/edit migration configuration files directly in config sync directory ⭑ Preserve nids for standalone nodes ⭑ Migrate unpublished content ⭑ Prioritize ability to rollback and re-import ⭑ Use ga_migrate_site for site specific overrides ⭑ Log skips and exceptions using custom logging solution ⭑ Solution order: ✩ Configuration, core and contrib ✩ Custom source/process plugins and services in ga_migrate module ✩ hook_migrate_MIGRATION_ID_prepare_row() GEORGIA GOVHUB MIGRATION + ⭑ FLORIDA DC 2020 Migration development strategy 28
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Custom Logging 29 Photo by Dorelys Smits on Unsplash
$message, // String: detailed message with ids. $migration_id, // String: current migration id. $needs_fix, // Boolean: Does this need to be fixed. $severity, // String: ‘warning’, ‘notice’, or ‘error’. $category, // String: Short descriptor. $row_id // Integer: The current row id. ); GEORGIA GOVHUB MIGRATION + $audience, // String: ‘DSGa’ or ‘dev’. FLORIDA DC 2020 ga_migrate_log( 30
31 GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Migration Phases 32 Photo by Farzad Mohsenvand on Unsplash
Add/remove sites from Tugboat ⭑ Run migrations on Tugboat ⭑ Client QA’s migrations on Tugboat ⭑ Development team fixes issues ⭑ Run migration on Production ⭑ Client prepares sites for launch ⭑ Sites are launched GEORGIA GOVHUB MIGRATION + ⭑ FLORIDA DC 2020 Migration phases 33
GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Magical Nerdery 34
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Site Specific Overrides 35 Photo by James Wheeler from Pexels
/sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + /modules/custom/ga_migrate_site 36
ga_migrate_site: 0 … core.extension.yml GEORGIA GOVHUB MIGRATION + ga_migrate: 0 FLORIDA DC 2020 … 37
// The date unit that accompanies the threshold value. const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD_UNIT = ‘y’; // The site node skip list. const GA_MIGRATE_SITE_SKIP_LIST = []; } /modules/custom/ga_migrate_site/src/Plugin/GaMigrateSiteInterface.php FLORIDA DC 2020 const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD = 3; + // Press_release nodes with release date greater // than threshold are trashed. GEORGIA GOVHUB MIGRATION interface GaMigrateSiteInterface { 38
// Implements hook_migrate_MIGRATION_ID_prepare_row(). /sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site/ga_migrate_site.module FLORIDA DC 2020 GEORGIA GOVHUB MIGRATION + // Implements hook_migrate_prepare_row(). 39
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Nested Paragraphs 40 Photo by Victor Larracuente on Unsplash
GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Container Paragraphs 41
Content Paragraphs GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Container Paragraphs 42
field_content two_column gta_paragraph_image one_column gta_paragraph_text_area GEORGIA GOVHUB MIGRATION + gta_paragraph_related_links FLORIDA DC 2020 node 43
✩ ⭑ Includes entity embeds Stack new content in Body field FLORIDA DC 2020 Convert Paragraphs into to Drupal 8 markup + ⭑ GEORGIA GOVHUB MIGRATION Paragraphs strategy 44
source: - field_content micro_content: field_content: paragraphs_item … FLORIDA DC 2020 plugin: ga_micro_content_to_text + prepared_field_content: GEORGIA GOVHUB MIGRATION … 45
What should the Drupal 8 markup be for this paragraph when placed in the WYSIWYG Body field? ✩ Text markup ✩ Entity embed code ✩ File download link ✩ Alignment adjustments FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION What do you mean by “render”? 46
field_content two_column 6 3 1 gta_paragraph_image one_column 2 5 gta_paragraph_text_area 4 GEORGIA GOVHUB MIGRATION + gta_paragraph_related_links FLORIDA DC 2020 node 47
body/0/value: source: - ‘@prepared_field_content’ - body/0/value … GEORGIA GOVHUB MIGRATION + plugin: concat FLORIDA DC 2020 … 48
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION Circular Dependencies 49 Photo by Anita Austvika on Unsplash
link_collection node field_related_links field_rich_links Link Node reference Node reference Node reference Node reference Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 50
link_collection node nid topic_page node body GEORGIA GOVHUB MIGRATION + entity embed code FLORIDA DC 2020 site_page node ga_d7_node_site_page 51
GEORGIA GOVHUB MIGRATION prepared_field_related_links: FLORIDA DC 2020 … ga_d7_node_site_page 52
body/0/value: - source: - body/0/value - ‘@prepared_field_related_links’ … ga_d7_node_site_page GEORGIA GOVHUB MIGRATION + plugin: concat FLORIDA DC 2020 … 53
✩ �� ga_d7_site_page_field_related_links �� FLORIDA DC 2020 ga_d7_node_site_page + ⭑ GEORGIA GOVHUB MIGRATION Dependencies 54
link_collection node field_related_links field_rich_links Link blog_post reference blog_post reference site_page reference NULL Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 55
… process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page … + source: field_related_links GEORGIA GOVHUB MIGRATION plugin: sub_process FLORIDA DC 2020 site_page_stub_entity_references: 56
link_collection node field_related_links field_rich_links Link blog_post reference blog_post reference site_page reference topic_page reference Link Link Link Link GEORGIA GOVHUB MIGRATION + Link FLORIDA DC 2020 site_page node ga_d7_site_page_field_related_links 57
… process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page … + source: field_related_links GEORGIA GOVHUB MIGRATION plugin: sub_process FLORIDA DC 2020 site_page_stub_entity_references: 58
✩ ⭑ �� �� �� ga_d7_site_page_field_related_links ga_d7_site_page_field_related_links ✩ ga_d7_node_blog_post ✩ “ga_d7_node_site_page” �� FLORIDA DC 2020 ga_d7_node_site_page + ⭑ GEORGIA GOVHUB MIGRATION Dependencies �� 59
The non-specified dependency migration id is alphabetically before the current migration: ✩ ga_d7_node_site_page ✩ ga_d7_site_page_field_related_links GEORGIA GOVHUB MIGRATION ⭑ FLORIDA DC 2020 It works if… 60
⭑ ⭑ ✩ ga_d7_index_list_field_related_links ✩ ga_d7_node_index_list The fix hack: ✩ ga_d7_1_node_index_list ✩ ga_d7_index_list_field_related_links Migration Dependency When Stubbing Content: https://www.drupal.org/project/drupal/issues/3024634 FLORIDA DC 2020 The non-specified dependency migration id is alphabetically after the current migration: + ⭑ GEORGIA GOVHUB MIGRATION It doesn’t work if… 61
FLORIDA DC 2020 + GEORGIA GOVHUB MIGRATION WYSIWYG DOM Processing 62 Photo by Iker Urteaga on Unsplash
GEORGIA GOVHUB MIGRATION … FLORIDA DC 2020 body/0/value: 63
\DOMDocument GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 Xdebug 64
⭑ Loop and collect the items to alter, replace, or remove in an array. ⭑ Loop through collection array and alter, replace or remove each element. FLORIDA DC 2020 Get elements from the \DOMDocument object. + ⭑ GEORGIA GOVHUB MIGRATION \DOMDocument processing 65
// Get elements by tag. foreach ($elements as $element) { // Collect elements based on conditions. $remove_elements[] = $element; } } // Remove or replace elements. foreach ($remove_elements as $element) { $this->removeElementAndContent($element); } + if ($elements->length > 0) { GEORGIA GOVHUB MIGRATION $remove_elements = []; FLORIDA DC 2020 $elements = $dom->getElementsByTagName(‘*’); 66
⭑ D7 <img> tag → D8 embedded media entity ⭑ D7 link to file → D8 file download link ⭑ D7 <iframe> tag → D8 embedded Code Block node or media entity ⭑ General text filtering based on D8 field text format FLORIDA DC 2020 D7 embedded entity → D8 embed syntax + ⭑ GEORGIA GOVHUB MIGRATION WYSIWYG clean up 67
GEORGIA GOVHUB MIGRATION + FLORIDA DC 2020 We made it! 👏 68
Georgia GovHub: a case study by Darren Peterson at Drupal GovCon 2019 ✩ ⭑ This is also coming to DrupalCon Minneapolis 2020! A forest of designs without subthemes: Implementing Georgia.gov’s front-end in Drupal 8 by Marc Drummond at Drupal GovCon 2019 FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION More about the Georgia.gov project 69
Real Life Data Migrations on the Lullabot Podcast ⭑ An Overview for Migrating Drupal Sites to 8 by Juampy NR ⭑ Running and Testing Drupal 8 Migrations in CircleCI by Juampy NR ⭑ 31 Days of Drupal Migrations with Mauricio Dinarte on the Lullabot Podcast ⭑ Managing Authentication During API Migrations by April Sides ⭑ and more to come! FLORIDA DC 2020 + ⭑ GEORGIA GOVHUB MIGRATION More about migration from Lullabot 70
GEORGIA GOVHUB MIGRATION + Thank you. 71 FLORIDA DC 2020
As Digital Services Georgia upgraded their Drupal 7 multisite platform to Drupal 8, they capitalized on the opportunity to make improvements to their content model. Data migrations were customized to move and shape data to fit into new content types and fields.
Let’s take a look at some of the strategies, tools, and techniques used to migrate site data from the GeorgiaGov Platform to Georgia GovHub.
Topics that will be covered:
for free. You
can too.