Migration's Missing Instruction Manual

Drupal Camp Atlanta
November 10, 2018

Drupal Camp Chattanooga
June 8, 2019

https://startinggravity.github.io/migration-presentation

Jim Smith

Jim Smith

Freelance Drupal Developer
Oak Ridge

Derek Brownlee

Derek Brownlee

Drupal Developer
Oak Ridge National Lab

ORNL

Oak Ridge National Laboratory
https://www.ornl.gov

Covered in this Session

  1. Basics of D7 to D8 migration
  2. Mistakes we made
  3. Helpful tips and tricks

Not Covered in this Session

  1. Migrating from other CMSs
  2. Solutions for every migration problem

Goals of a Migration

  1. Make it complete
  2. Make it easy
  3. Make it reproducible

Goals of a Migration

  1. Make it complete
  2. Make it easy
  3. Make it reproducible

Plan. Plan. Plan.

Migration Planning Spreadsheet

source: drupal.org

Plan what?

  • Content to move; content to ignore
  • Content to restructure (e.g., move to paragraphs)
  • Content to clean up (e.g., improve consistency)
  • Improvements to content editor experience

Don't Migrate

  • Configuration, if you are making major architecture changes
  • Content that no one sees
  • Content from a small number of nodes when copy/paste is easier
Mockup of new page
Mockup with fields highlighted

Plan Migration Order

  1. Configuration (optional)
  2. Taxonomy Terms
  3. Users
  4. Files
  5. Media Entities
  6. Paragraphs
  7. Nodes

Migration Steps

Migration Terms

Migration Steps

Migration Terms

Migration Steps

Migration Terms

Migration Steps

Migration Terms

Core Modules

  • Migrate
  • Migrate Drupal
  • Migrate Drupal UI

Contrib Modules

  • Migrate Upgrade
  • Migrate Tools
  • Migrate Plus
  • Migrate Source CSV

Contrib Modules

  • Migrate Devel
  • Migrate Manifest
  • Migrate Run

Migration YML File

Migration File

Migration YML File:
Metadata

Migration File

Source

  • Source plugins fetch data from the source
  • The source can be SQL database, CSV, JSON, XML or SOAP
  • The plugin can come from core, a contrib module, or custom module

Migration YML File:
Source

Migration File

Process

  • Process plugins can
    • Copy data
    • Add default data
    • Combine or extract data
    • Skip data
    • Pipe data to different fields
    • Manipulate data
  • These plugins can be chained together

Migration YML File:
Process

Migration File

Destination

  • Destination plugins define the kind of Drupal entities to be created
  • Content entities: Nodes, users, taxonomy terms, files
  • Configuration entities: Content types and field definitions

Migration YML File:
Destination

Migration File

Connect to Database

settings.php

Connect to CSV

Advantages:

  • See all of the data at once
  • Edit, clean up the data
  • Use a custom query to combine data from different content types

Config Single Import

Single Import

Create a Module

  • When you expect changes in your source before you go live
  • If you need a custom plugin
  • If you use a source other than a database

Using a Module

  • Don't forget to disable, then enable it each time you make a change

Use Drush

  • Status: drush ms
  • Import: drush mim
  • Rollback: drush mr
  • Reset: drush mrs

Use Drush

Single entity:
drush mim my_nodes --idlist=1234

Small batch:
drush mim my_nodes --limit=10

Look at the Database

Sequel Pro

Look at the Database

File table

Message Tables

Sequel Pro
idlist

Migrate Devel Module

Migrate Devel - source

Migrate Devel Module

Migrate Devel - source

See Problems in the Array

Array

More Tips

Let Drupal Help You

CKEditor can clean up markup

CKEditor

Dependencies

Dependencies
  • Required: 100% of content must be migrated.
  • Optional: if there are no items from the dependent migration, the migration will still run.

Comments

Leave yourself notes, test iteratively.

Comments

Pseudo-fields

Parse your data.

Pseudo-field

Pseudo-fields

Parse your data.

Pseudo-field

Dummy Variables

Overcome limitations of skip_on_empty.

Dummy variable

Dummy Variables

Overcome limitations of skip_on_empty.

Dummy variable

Pantheon Timeout

Use --feedback flag

Feedback

Lando Slowness

Use "Airplane mode"

Airplane mode

Lando Slowness

Switch locations

Switching locations

TripMode

Keep everything else online

TripMode app

www.tripmode.ch

Custom Plugins

Solve problems core and contrib plugins can't

Custom plugin

Custom Plugins

Solve problems core and contrib plugins can't

Custom plugin

Custom Plugins

Add as you would any plugin

Custom plugin

Examples of Complex Processing

Migrating the Primary Org field
Constructing a full name pseudofield
Building a pseudofield
The make_unique custom plugin
Setting an image caption field
The truncate_to_255 custom plugin

Thank you!
Good luck with your migrations.