An Overview for Migrating Drupal Sites to 8

Flock migrating

Over the past few months working on migrations to Drupal 8, researching best practices, and contributing to core and contributed modules, I discovered that there are several tools available in core and contributed modules, plus a myriad of how-to articles. To save you the trouble of pulling it all together yourself, I offer a comprehensive overview of the Migrate module plus a few other contributed modules that complement it in order to migrate a Drupal site to 8.

Let's begin with the most basic element: migration files.

Migration files

In Drupal 8, the migrate module splits a migration into a set of YAML files, each of them is composed by a source (like the node table in a Drupal 7 database), a process (the field mapping and processing), and a destination (like a node entity in 8). Here is a subset of the migration files present in core:

$ find core -name *.yml | grep migrations
core/modules/statistics/migrations/statistics_settings.yml
core/modules/statistics/migrations/statistics_node_counter.yml
core/modules/shortcut/migrations/d7_shortcut_set.yml
core/modules/shortcut/migrations/d7_shortcut.yml
core/modules/shortcut/migrations/d7_shortcut_set_users.yml
core/modules/tracker/migrations/d7_tracker_node.yml
core/modules/tracker/migrations/d7_tracker_settings.yml
core/modules/tracker/migrations/d7_tracker_user.yml
core/modules/path/migrations/d7_url_alias.yml
...

And below is the node migration, located at core/modules/node/migrations/d7_node.yml:

id: d7_node
label: Nodes
audit: true
migration_tags:
  - Drupal 7
  - Content
deriver: Drupal\node\Plugin\migrate\D7NodeDeriver
source:
  plugin: d7_node
process:
  nid: tnid
  vid: vid
  langcode:
    plugin: default_value
    source: language
    default_value: "und"
  title: title
  uid: node_uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
destination:
  plugin: entity:node
migration_dependencies:
  required:
    - d7_user
    - d7_node_type
  optional:
    - d7_field_instance
    - d7_comment_field_instance

Migration files can be generated dynamically via a deriver like the node migration defines above, which uses D7NodeDeriver to generate a migration for each content type’s data and revision tables. On top of that, migrations can be classified via the migration_tags section (the migration above has the Drupal 7 and Content tags).

Configuration vs content migrations

Migration files may have one or more tags to classify them. These tags are used for running them in groups. Some migration files may have the Configuration tag—like the node type or field migrations—while others might have the Content tag—like the node or user migrations. Usually, you would run the Configuration migrations first in order to configure the new site, and then the Content ones so the content gets fetched, transformed, and inserted on top of such configuration.

Notice though that depending on how much you are planning to change the content model, you may decide to configure the new site manually and write the Content migrations by hand. In the next section, we will examine the differences between generating and writing migrations.

Generating vs Writing migrations

Migration files living within the core, contributed, and custom modules are static files that need to be read and imported into the database as migration plugins so they can be executed. This process, depending on the project needs, can be implemented in two different ways:

Using Migrate Upgrade

Migrate Upgrade module implements a Drush command to automatically generate migrations for all the configuration and content in an existing Drupal 6 or 7 site. The best thing about this approach is that you don’t have to manually create the new content model in the new site since Migrate Upgrade will inspect the source database and do it for you by generating the migrations.

If the existing content model won’t need to go through major changes during the migration, then Migrate Upgrade is a great choice to generate migrations. There is an API that developers can interact with in order to alter migrations and the data being processed. We will see a few examples further down in this article.

Writing migrations by hand

If the content model will go through a deep reorganization such as merging content from different sources into one, reorganizing fields, and changing machine names, then configuring the new site manually and writing content migrations may be the best option. In this scenario, you would write the migration files directly to the config/sync directory so then they can be imported via drush config:import and executed via drush migrate:import.

Notice that if the content model has many entity types, bundles, and fields, this can be a tough job so even if the team decides to go this route, generating content migrations with Migrate Upgrade can be useful since the resulting migrations can serve as templates for the ones to be written.

Setting up the new site for running migrations

Assuming that we have a new site created using the Composer Drupal Project and we have run the installer, we need to require and install the following modules:

$ composer require drupal/migrate_tools drupal/migrate_upgrade drupal/migrate_plus
$ drush pm:enable --yes migrate_tools,migrate_upgrade,migrate_plus

Next, we need to add a database connection to the old site, which we would do at web/sites/default/settings.local.php:

// The default database connection details.
$databases['default']['default'] = [
  'database' => 'drupal8',
  'username' => 'root',
  'password' => 'root',
  'prefix' => '',
  'host' => '127.0.0.1',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
];

// The Drupal 7 database connection details.
$databases['drupal7']['default'] = [
  'database' => 'drupal7',
  'username' => 'root',
  'password' => 'root',
  'prefix' => '',
  'host' => '127.0.0.1',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
];

With the above setup, we can move on to the next step, where we will generate migrations.

Generating migrations with Migrate Upgrade

The following command will read all migration files, create migrate entities out of them and insert them into the database so they are ready to be executed:

$ drush --yes migrate:upgrade --legacy-db-key drupal7 --legacy-root sites/default/files --configure-only

It is a good practice to export the resulting migrate entities as configuration so we can track their changes. Therefore, we will export configuration after running the above command, which will create a list of files like the following:

$ drush --yes config:export
 [notice] Differences of the active config to the export directory:
+------------------------------------------------------------------+-----------+
| Config                                                           | Operation |
+------------------------------------------------------------------+-----------+
| migrate_plus.migration.upgrade_d7_date_formats                   | Create    |
| migrate_plus.migration.upgrade_d7_field                          | Create    |
| migrate_plus.migration.upgrade_d7_field_formatter_settings       | Create    |
| migrate_plus.migration.upgrade_d7_field_instance                 | Create    |
| migrate_plus.migration.upgrade_d7_field_instance_widget_settings | Create    |
| migrate_plus.migration.upgrade_d7_file                           | Create    |
| migrate_plus.migration.upgrade_d7_filter_format                  | Create    |
| migrate_plus.migration.upgrade_d7_filter_settings                | Create    |
| migrate_plus.migration.upgrade_d7_image_styles                   | Create    |
| migrate_plus.migration.upgrade_d7_menu                           | Create    |
| migrate_plus.migration.upgrade_d7_menu_links                     | Create    |
| migrate_plus.migration.upgrade_d7_node_article                   | Create    |
| migrate_plus.migration.upgrade_d7_node_revision_article          | Create    |
...

In the above list, we see a mix of Configuration and Content migrations being created. Now we can check the status of each migration via the migrate:status Drush command:

$ drush migrate:status
-----------------------------------------------------------------------------------------------------
 Migration ID                                 Status   Total   Imported   Unprocessed   Last Imported
-----------------------------------------------------------------------------------------------------
 upgrade_d7_date_formats                      Idle     7       0          0
 upgrade_d7_filter_settings                   Idle     1       0          0
 upgrade_d7_image_styles                      Idle     193     0          0
 upgrade_d7_node_settings                     Idle     1       0          0
 upgrade_d7_system_date                       Idle     1       0          0
 upgrade_d7_url_alias                         Idle     25981   0          0
 upgrade_system_site                          Idle     1       0          0
 upgrade_taxonomy_settings                    Idle     0       0          0
 upgrade_d7_path_redirect                     Idle     518     0          0
 upgrade_d7_field                             Idle     253     0          0
 upgrade_d7_field_collection_type             Idle     9       0          0
 upgrade_d7_node_type                         Idle     16      0          0
 upgrade_d7_taxonomy_vocabulary               Idle     34      0          0
 upgrade_d7_field_instance                    Idle     738     0          0
 upgrade_d7_view_modes                        Idle     24      0          0
 upgrade_d7_field_formatter_settings          Idle     1280    0          0
 upgrade_d7_field_instance_widget_settings    Idle     738     0          0
 upgrade_d7_file                              Idle     5731    0          0
 upgrade_d7_filter_format                     Idle     6       0          0
 upgrade_d7_menu                              Idle     5       0          0
 upgrade_d7_user_role                         Idle     6       0          0
 upgrade_d7_user                              Idle     82      0          0
 upgrade_d7_node_article                      Idle     322400  0          0
 upgrade_d7_node_page                         Idle     342     0          0
 upgrade_d7_menu_links                        Idle     623     0          0
 upgrade_d7_node_revision_article             Idle     742577  0          0
 upgrade_d7_node_revision_page                Idle     2122    0          0
 upgrade_d7_taxonomy_term_tags                Idle     1729    0          0
-------------------------------------------- -------- ------- ---------- ------------- ---------------

You should inspect any contributed modules in use on the old site and install them in the new one as they may contain migrations. For example, if the old site is using Redirect module, by installing itm and generating migrations like we did above, you should see a new migration provided by this module ready to go.

Running migrations

Assuming that we decided to run migrations generated by Migrate Upgrade (otherwise skip the following sub-section), we would first run configuration migrations and then content ones.

Configuration migrations

Here is the command to run all migrations with the tag Configuration along with their dependencies.

$ drush migrate:import --tag=Configuration --execute-dependencies
 [notice] Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_date_formats'
 [notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_filter_settings'
 [notice] Processed 20 items (20 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_image_styles'
 [notice] Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_settings'
 [notice] Processed 1 items (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_date'
 [notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_site'
 [notice] Processed 100 items (100 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field'
 [notice] Processed 200 items (200 created, 0 updated, 0 failed, 379 ignored) - done with 'upgrade_d7_field_instance'
...

The above command will migrate site configuration, content types, taxonomy vocabularies, view modes, and such. Once this is done, it is recommended to export the resulting configuration via drush config:export and commit the changes. From then on, if we make changes in the old site’s configuration or we alter the Configuration migrations (we will see how further down), we will need to roll back the affected migrations and run them again.

For example, the Media Migration module creates media fields and alters field mappings in content migrations so after installing it we should run the following commands to re-run them:

$ drush --yes migrate:rollback upgrade_d7_view_modes,upgrade_d7_field_instance_widget_settings,upgrade_d7_field_formatter_settings,upgrade_d7_field_instance,upgrade_d7_field
$ drush --yes migrate:import --execute-dependencies upgrade_d7_field,upgrade_d7_field_instance,upgrade_d7_field_formatter_settings,upgrade_d7_field_instance_widget_settings,upgrade_d7_view_modes
$ drush --yes config:export

Once we have executed all Configuration migrations, we can run content ones.

Content migrations

Content migrations are straightforward. We can run them with the following command:

$ drush migrate:import --tag=Content --execute-dependencies

Logging and tracking

Migrate keeps track of all the migrated content via the migrate_* tables. If you check out the new database after running migrations, you will see something like this:

  • A set of migrate_map* tables, storing the old and new identifiers of each migrated entity. These tables are used by the migrate:rollback Drush command to roll back migrated data.
  • A set of migrate_messages* tables, which hold errors and warnings that occurred while running migrations. These can be seen via the migrate:messages Drush command.

Rolling back migrations

In the previous section, we rolled back the field migrations before running them again. This process is great for reverting imported Configuration or Content, which you will do often while developing a migration.

Here is an example. Let’s suppose that you have executed content migrations but articles did not get migrated as you expected. The process you would follow to fix them would be:

  1. Find the migration name via drush migrate:status | grep article.
  2. Roll back migrations with drush migrate:rollback upgrade_d7_node_revision_article,upgrade_d7_node_article.
  3. Perform the changes that you need either directly at the exported migration at config/sync or by altering them and then recreating them with migrate:upgrade like we did at Generating migrations with Migrate Upgrade. We will see how to alter migrations in the next section.
  4. Run the migrations again with drush migrate:import upgrade_d7_node_article,upgrade_d7_node_revision_article.
  5. Verify the changes and, if needed, repeat steps 2 to 4 until you are done.

Migrate events and hooks

Before diving into the APIs to alter migrations, let’s clarify that there are two processes that we can hook into:

  1. The migrate:upgrade Drush command, which reads all migration files in core, contributed, and custom modules and imports them into the database.
  2. The migrate:import Drush command, which runs migrations.

In the next sub-sections, we will see how can we interact with these two commands.

Altering migrations (migrate:upgrade)

Drupal core offers hook_migration_plugins_alter(), which receives the array of available migrations that migrate:upgrade creates. Here is a sample implementation at mymodule.module where we delegate the logic to a service:

/**
 * Implements hook_migration_plugins_alter().
 */
function mymodule_migration_plugins_alter(array &$migrations) {
  $migration_alterer = \Drupal::service('mymodule.migrate.alterer');
  $migration_alterer->process($migrations);
}

And here is a subset of the contents of the service:

class MigrationAlterer {

  /**
   * Processes migration plugins.
   *
   * @param array $migrations
   *   The array of migration plugins.
   */
  public function process(array &$migrations) {
    $this->skipMigrations($migrations);
    $this->disablePathautoAliasCreation($migrations);
    $this->setContentLangcode($migrations);
    $this->setModerationState($migrations);
    $this->persistUuid($migrations);
    $this->skipFileCopy($migrations);
    $this->alterRedirect($migrations);
  }

  /**
   * Skips unneeded migrations.
   *
   * @param array $migrations
   *   The array of migration plugins.
   */
  private function skipMigrations(array &$migrations) {
    // Skip unwanted migrations.
    $migrations_to_skip = [
      'd7_block',
      'd7_comment',
      'd7_comment_entity_display',
      'd7_comment_entity_form_display_subject',
      'd7_comment_field',
      'd7_comment_entity_form_display',
      'd7_comment_type',
      'd7_comment_field_instance',
      'd7_contact_settings',
    ];
    $migrations = array_filter($migrations, function ($migration) use ($migrations_to_skip) {
      return !in_array($migration['id'], $migrations_to_skip);
    });
  }

  // The remaining methods would go here.

}

In the next section, we will see how to alter the data being migrated while running migrations.

Altering data while running migrations (migrate:import)

Drupal core offers hook_migrate_prepare_row() and hook_migrate_MIGRATION_ID_prepare_row, which are triggered before each row of data is processed by the migrate:import Drush command. Additionally, there is a set of events that we can subscribe to such as before and after the migration starts or before and after a row is saved.

On top of the above, Migrate Plus module exposes an event that wraps hook_migrate_prepare_row(). Here is a sample subscriber for this event:

class MyModuleMigrationSubscriber implements EventSubscriberInterface {

  /**
   * Prepare row event handler.
   *
   * @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event
   *   The migrate row event.
   *
   * @throws \Drupal\migrate\MigrateSkipRowException
   *   If the row needs to be skipped.
   */
  public function onPrepareRow(MigratePrepareRowEvent $event) {
    $this->alterFieldMigrations($event);
    $this->skipMenuLinks($event);
    $this->setContentModeration($event);
  }

  /**
   * Alters field migrations.
   *
   * @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event
   *   The migrate row event.
   *
   * @throws \Drupal\migrate\MigrateSkipRowException
   *   If a row needs to be skipped.
   * @throws \Exception
   *   If the source cannot be changed.
   */
  private function alterFieldMigrations(MigratePrepareRowEvent $event) {
    $field_migrations = [
      'upgrade_d7_field',
      'upgrade_d7_field_instance',
      'upgrade_d7_view_modes',
      'upgrade_d7_field_formatter_settings',
      'upgrade_d7_field_instance_widget_settings',
    ];

    if (in_array($event->getMigration()->getPluginId(), $field_migrations)) {
      // Here are calls to private methods that alter these migrations.
    }
  }

  /**
   * Skips menu links that are either implemented or not needed.
   *
   * @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event
   *   The migrate row event.
   *
   * @throws \Drupal\migrate\MigrateSkipRowException
   *   If a row needs to be skipped.
   */
  private function skipMenuLinks(MigratePrepareRowEvent $event) {
    if ($event->getMigration()->getPluginId() != 'upgrade_d7_menu_links') {
      return;
    }

    $paths_to_skip = [
      'some/path',
      'other/path',
    ];

    $menu_link = $event->getRow()->getSourceProperty('link_path');
    if (in_array($menu_link, $paths_to_skip)) {
      throw new MigrateSkipRowException('Skipping menu link ' . $menu_link);
    }
  }

  /**
   * Sets the content moderation field on node migrations.
   *
   * @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event
   *   The migrate row event.
   *
   * @throws \Exception
   *   If the source cannot be changed.
   */
  private function setContentModeration(MigratePrepareRowEvent $event) {
    $row = $event->getRow();
    $source = $event->getSource();

    if (('d7_node' == $source->getPluginId()) && isset($event->getMigration()->getProcess()['moderation_state'])) {
      $state = $row->getSourceProperty('status') ? 'published' : 'draft';
      $row->setSourceProperty('moderation_state', $state);
    }
    elseif (('d7_node_revision' == $source->getPluginId()) && isset($event->getMigration()->getProcess()['moderation_state'])) {
      $row->setSourceProperty('moderation_state', 'draft');
    }
  }

}

Conclusion

When you are migrating a Drupal site to 8, Migrate Upgrade module does a lot of the work for free by generating Configuration and Content migrations. Even if you decide to write any of them by hand, it is convenient to run the command and use the resulting migrations as a template for your own. In the next article, we will see how to work seamlessly on a migration while the rest of the team is building the new site.

Thanks to Andrew Berry, April Sides, Karen Stevenson, Marcos Cano, and Salvador Moreno for their feedback and help. Photo by Barth Bailey on Unsplash.

Juampy NR

Juampy Headshot
Loves optimizing development workflows. Publishes articles, books, and code.