Custom Drupal Data Migration: A Georgia GovHUB Story

weekbeforenext weekbeforenext aprilsides

⭑ Discovery and Planning ⭑ Strategies and Workflow ⭑ Magical Nerdery

Photo by Elodie Oudot on Unsplash

⭑ Drupal 7 Multisite (85+ sites) ⭑ Drupal 8 Multisite (~6 at a time) ⭑ Hosted on Acquia ⭑ Hosted on Acquia ⭑ 27 Content Types (15 migrated) ⭑ 20 Content Types (14 populated) ⭑ 14 Taxonomy Vocabularies (9 migrated) ⭑ 17 Taxonomy Vocabularies (10 populated) ⭑ Paragraphs, Field Collections and Entity Embeds ⭑ “Micro-content” Types and Entity Embeds

https://www.drupal.org/project/migration_planner

Photo by mali maeder from Pexels

⭑ WYSIWYG string queries: ✩ Object tags ✩ Image tags ✩ Embedded entities ✩ Absolute links ✩ Span tags ✩ iFrames ✩ Script tags ✩ Tables ✩ ✩ ✩ ✩ ✩ ✩ ✩ ✩ Custom classes Custom forms Email links Style tags and attributes Social network links Video iFrames Links to PDFs File Lists

⭑ ⭑ ⭑ ⭑ ⭑ ⭑ ⭑ ⭑ Content by date thresholds Parent Content Unpublished Content Content with documents Paragraphs Field Collections Specific field values Redirects ⭑ Metatag string queries: ✩ Description ✩ Abstract ✩ Token ✩ Creator ✩ Canonical ✩ Title ✩ Image ✩ Keyword ✩ Video ✩ Twitter ✩ Open Graph

$ drush @[SITENAME].[ENVIRONMENT] squeal /sites/[SITENAME].georgia.gov/files/ga_squealer_reports/ [SITENAME].georgia.gov_YYYY-MM-DD.xlsx

https://www.drupal.org/project/squealer

Photo by chuttersnap on Unsplash

❌ ⭑ ⭑ ⭑ ✅ Structural elements: ⭑ Data elements: ✩ Content types and field definitions ✩ Select nodes and field data ✩ Vocabularies ✩ ✩ Paragraph and field collection bundles Select taxonomy terms Views Webform submissions ⭑ Menus ⭑ Webforms ⭑ Files

⭑ ⭑ Core: ⭑ Contributed: ✩ Migrate migrate ✩ Migrate Files (extended) migrate_file ✩ Migrate Drupal migrate_drupal ✩ Migrate Plus migrate_plus ✩ Migrate Source CSV migrate_source_csv ✩ Migrate Tools migrate_tools ✩ Drupal Upgrade migrate_upgrade ✩ Migrate Source UI migrate_source_ui Custom: ✩ ga_migrate ✩ ga_migrate_site ✩ ga_migrate_source_ui

⭑ Local development environment: ✩ ⭑ QA environment: ✩ ⭑ Lando Tugboat.qa DevOps magic: ✩ CircleCI ✩ Quay.io

https://tugboat.qa

⭑ Focus on field mapping by complexity/content type: ✩ Basic fields ✩ Rich fields ✩ File/image fields ✩ Paragraphs/Field Collections ✩ WYSIWYG cleanup

⭑ Create/edit migration configuration files directly in config sync directory ⭑ Preserve nids for standalone nodes ⭑ Migrate unpublished content ⭑ Prioritize ability to rollback and re-import ⭑ Use ga_migrate_site for site specific overrides ⭑ Log skips and exceptions using custom logging solution ⭑ Solution order: ✩ Configuration, core and contrib ✩ Custom source/process plugins and services in ga_migrate module ✩ hook_migrate_MIGRATION_ID_prepare_row()

Photo by Dorelys Smits on Unsplash

ga_migrate_log( $message, // String: detailed message with ids. $migration_id, // String: current migration id. $audience, // String: ‘DSGa’ or ‘dev’. $needs_fix, // Boolean: Does this need to be fixed. $severity, // String: ‘warning’, ‘notice’, or ‘error’. $category, // String: Short descriptor. $row_id // Integer: The current row id. );

Photo by Farzad Mohsenvand on Unsplash

⭑ Add/remove sites from Tugboat ⭑ Run migrations on Tugboat ⭑ Client QA’s migrations on Tugboat ⭑ Development team fixes issues ⭑ Run migration on Production ⭑ Client prepares sites for launch ⭑ Sites are launched

Photo by James Wheeler from Pexels

/modules/custom/ga_migrate_site /sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site

… ga_migrate: 0 ga_migrate_site: 0 … core.extension.yml

interface GaMigrateSiteInterface { // Press_release nodes with release date greater // than threshold are trashed. const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD = 3; // The date unit that accompanies the threshold value. const GA_MIGRATE_SITE_PRESS_RELEASE_THRESHOLD_UNIT = ‘y’; // The site node skip list. const GA_MIGRATE_SITE_SKIP_LIST = []; } /modules/custom/ga_migrate_site/src/Plugin/GaMigrateSiteInterface.php

// Implements hook_migrate_prepare_row(). // Implements hook_migrate_MIGRATION_ID_prepare_row(). /sites/[SITENAME].georgia.gov/modules/custom/ga_migrate_site/ga_migrate_site.module

Photo by Victor Larracuente on Unsplash

node field_content two_column gta_paragraph_related_links gta_paragraph_image one_column gta_paragraph_text_area

⭑ Convert Paragraphs into to Drupal 8 markup ✩ ⭑ Includes entity embeds Stack new content in Body field

… prepared_field_content: plugin: ga_micro_content_to_text source: - field_content micro_content: field_content: paragraphs_item …

⭑ What should the Drupal 8 markup be for this paragraph when placed in the WYSIWYG Body field? ✩ Text markup ✩ Entity embed code ✩ File download link ✩ Alignment adjustments

node field_content two_column 6 3 gta_paragraph_related_links gta_paragraph_image one_column 2 5 gta_paragraph_text_area 4 1

… body/0/value: plugin: concat source: - ‘@prepared_field_content’ - body/0/value …

Photo by Anita Austvika on Unsplash

site_page node link_collection node field_related_links field_rich_links Link Link Node reference Node reference Node reference Node reference Link Link Link Link ga_d7_site_page_field_related_links

site_page node link_collection node nid topic_page node body entity embed code ga_d7_node_site_page

… prepared_field_related_links: plugin: migration_lookup migration: ga_d7_site_page_field_related_links source: nid no_stub: true plugin: ga_entity_embed_code entity_type: node … ga_d7_node_site_page

… body/0/value: plugin: concat source: - body/0/value - ‘@prepared_field_related_links’ … ga_d7_node_site_page

⭑ ga_d7_node_site_page ✩ �� ga_d7_site_page_field_related_links ��

site_page node link_collection node field_related_links field_rich_links Link Link blog_post reference blog_post reference site_page reference NULL Link Link Link Link ga_d7_site_page_field_related_links

… site_page_stub_entity_references: plugin: sub_process source: field_related_links process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page …

site_page node link_collection node field_related_links field_rich_links Link Link blog_post reference blog_post reference site_page reference topic_page reference Link Link Link Link ga_d7_site_page_field_related_links

… site_page_stub_entity_references: plugin: sub_process source: field_related_links process: entity_references: … plugin: migration_lookup migration: ga_d7_node_site_page source: field_related_content/0/nid stub_id: ga_d7_node_site_page …

⭑ ga_d7_node_site_page ✩ ⭑ �� ga_d7_site_page_field_related_links ga_d7_site_page_field_related_links ✩ ga_d7_node_blog_post ✩ “ga_d7_node_site_page” ��

⭑ The non-specified dependency migration id is alphabetically before the current migration: ✩ ga_d7_node_site_page ✩ ga_d7_site_page_field_related_links

⭑ ⭑ ⭑ The non-specified dependency migration id is alphabetically after the current migration: ✩ ga_d7_index_list_field_related_links ✩ ga_d7_node_index_list The fix hack: ✩ ga_d7_1_node_index_list ✩ ga_d7_index_list_field_related_links Migration Dependency When Stubbing Content: https://www.drupal.org/project/drupal/issues/3024634

👏

⭑ Georgia GovHub: a case study by Darren Peterson at Drupal GovCon 2019 ✩ ⭑ This is also coming to DrupalCon Minneapolis 2020! A forest of designs without subthemes: Implementing Georgia.gov’s front-end in Drupal 8 by Marc Drummond at Drupal GovCon 2019

⭑ Real Life Data Migrations on the Lullabot Podcast ⭑ An Overview for Migrating Drupal Sites to 8 by Juampy NR ⭑ Running and Testing Drupal 8 Migrations in CircleCI by Juampy NR ⭑ 31 Days of Drupal Migrations with Mauricio Dinarte on the Lullabot Podcast ⭑ Managing Authentication During API Migrations by April Sides ⭑ and more to come!