--- title: "An Overview for Migrating Drupal Sites to 8" url: "/articles/overview-migrating-drupal-sites-8" type: article date: 2019-06-19 updated: 2019-06-19 --- # An Overview for Migrating Drupal Sites to 8 # An Overview for Migrating Drupal Sites to 8 An introduction to how Drupal to Drupal migrations work. A follow-up article will cover how to develop a migration seamlessly while the rest of the team is working on the new site. By [ Juampy NR ](/about/juampy-nr) June 19, 2019 Over the past few months working on migrations to Drupal 8, researching best practices, and contributing to core and contributed modules, I discovered that there are several tools available in core and contributed modules, plus a myriad of how-to articles. To save you the trouble of pulling it all together yourself, I offer a comprehensive overview of the Migrate module plus a few other contributed modules that complement it in order to migrate a Drupal site to 8. Let's begin with the most basic element: migration files. ## Migration files In Drupal 8, the [migrate module](https://www.drupal.org/docs/8/api/migrate-api) splits a migration into a set of YAML files, each of them is composed by a *source* (like the node table in a Drupal 7 database), a *process* (the field mapping and processing), and a *destination* (like a node entity in 8). Here is a subset of the migration files present in core: ``` $ find core -name *.yml | grep migrations core/modules/statistics/migrations/statistics_settings.yml core/modules/statistics/migrations/statistics_node_counter.yml core/modules/shortcut/migrations/d7_shortcut_set.yml core/modules/shortcut/migrations/d7_shortcut.yml core/modules/shortcut/migrations/d7_shortcut_set_users.yml core/modules/tracker/migrations/d7_tracker_node.yml core/modules/tracker/migrations/d7_tracker_settings.yml core/modules/tracker/migrations/d7_tracker_user.yml core/modules/path/migrations/d7_url_alias.yml ... ``` And below is the node migration, located at `core/modules/node/migrations/d7_node.yml`: ``` id: d7_node label: Nodes audit: true migration_tags: - Drupal 7 - Content deriver: Drupal\node\Plugin\migrate\D7NodeDeriver source: plugin: d7_node process: nid: tnid vid: vid langcode: plugin: default_value source: language default_value: "und" title: title uid: node_uid status: status created: created changed: changed promote: promote sticky: sticky revision_uid: revision_uid revision_log: log revision_timestamp: timestamp destination: plugin: entity:node migration_dependencies: required: - d7_user - d7_node_type optional: - d7_field_instance - d7_comment_field_instance ``` Migration files can be generated dynamically via a `deriver` like the node migration defines above, which uses [`D7NodeDeriver`](https://git.drupalcode.org/project/drupal/blob/8.7.x/core/modules/node/src/Plugin/migrate/D7NodeDeriver.php) to generate a migration for each content type’s data and revision tables. On top of that, migrations can be classified via the `migration_tags` section (the migration above has the `Drupal 7` and `Content` tags). ## Configuration vs content migrations Migration files may have one or more tags to classify them. These tags are used for running them in groups. Some migration files may have the *Configuration* tag—like the node type or field migrations—while others might have the *Content* tag—like the node or user migrations. Usually, you would run the Configuration migrations first in order to configure the new site, and then the Content ones so the content gets fetched, transformed, and inserted on top of such configuration. Notice though that depending on how much you are planning to change the content model, you may decide to configure the new site manually and write the Content migrations by hand. In the next section, we will examine the differences between generating and writing migrations. ## Generating vs Writing migrations Migration files living within the core, contributed, and custom modules are static files that need to be read and imported into the database as [migration plugins](https://git.drupalcode.org/project/drupal/blob/8.7.x/core/modules/migrate/src/Plugin/Migration.php) so they can be executed. This process, depending on the project needs, can be implemented in two different ways: ### Using Migrate Upgrade [Migrate Upgrade](https://www.drupal.org/project/migrate_upgrade) module implements a Drush command to automatically generate migrations for all the configuration and content in an existing Drupal 6 or 7 site. The best thing about this approach is that you don’t have to manually create the new content model in the new site since Migrate Upgrade will inspect the source database and do it for you by generating the migrations. If the existing content model won’t need to go through major changes during the migration, then [Migrate Upgrade](https://www.drupal.org/project/migrate_upgrade) is a great choice to generate migrations. There is an API that developers can interact with in order to alter migrations and the data being processed. We will see a few examples further down in this article. ### Writing migrations by hand If the content model will go through a deep reorganization such as merging content from different sources into one, reorganizing fields, and changing machine names, then configuring the new site manually and writing content migrations may be the best option. In this scenario, you would write the migration files directly to the `config/sync` directory so then they can be imported via `drush config:import` and executed via `drush migrate:import`. Notice that if the content model has many entity types, bundles, and fields, this can be a tough job so even if the team decides to go this route, generating content migrations with Migrate Upgrade can be useful since the resulting migrations can serve as templates for the ones to be written. ## Setting up the new site for running migrations Assuming that we have a new site created using the [Composer Drupal Project](https://github.com/drupal-composer/drupal-project) and we have run the installer, we need to require and install the following modules: ``` $ composer require drupal/migrate_tools drupal/migrate_upgrade drupal/migrate_plus $ drush pm:enable --yes migrate_tools,migrate_upgrade,migrate_plus ``` Next, we need to add a database connection to the old site, which we would do at `web/sites/default/settings.local.php`: ``` // The default database connection details. $databases['default']['default'] = [ 'database' => 'drupal8', 'username' => 'root', 'password' => 'root', 'prefix' => '', 'host' => '127.0.0.1', 'port' => '3306', 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ]; // The Drupal 7 database connection details. $databases['drupal7']['default'] = [ 'database' => 'drupal7', 'username' => 'root', 'password' => 'root', 'prefix' => '', 'host' => '127.0.0.1', 'port' => '3306', 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ]; ``` With the above setup, we can move on to the next step, where we will generate migrations. ### Generating migrations with Migrate Upgrade The following command will read all migration files, create migrate entities out of them and insert them into the database so they are ready to be executed: ``` $ drush --yes migrate:upgrade --legacy-db-key drupal7 --legacy-root sites/default/files --configure-only ``` It is a good practice to export the resulting migrate entities as configuration so we can track their changes. Therefore, we will export configuration after running the above command, which will create a list of files like the following: ``` $ drush --yes config:export [notice] Differences of the active config to the export directory: +------------------------------------------------------------------+-----------+ | Config | Operation | +------------------------------------------------------------------+-----------+ | migrate_plus.migration.upgrade_d7_date_formats | Create | | migrate_plus.migration.upgrade_d7_field | Create | | migrate_plus.migration.upgrade_d7_field_formatter_settings | Create | | migrate_plus.migration.upgrade_d7_field_instance | Create | | migrate_plus.migration.upgrade_d7_field_instance_widget_settings | Create | | migrate_plus.migration.upgrade_d7_file | Create | | migrate_plus.migration.upgrade_d7_filter_format | Create | | migrate_plus.migration.upgrade_d7_filter_settings | Create | | migrate_plus.migration.upgrade_d7_image_styles | Create | | migrate_plus.migration.upgrade_d7_menu | Create | | migrate_plus.migration.upgrade_d7_menu_links | Create | | migrate_plus.migration.upgrade_d7_node_article | Create | | migrate_plus.migration.upgrade_d7_node_revision_article | Create | ... ``` In the above list, we see a mix of Configuration and Content migrations being created. Now we can check the status of each migration via the `migrate:status` Drush command: ``` $ drush migrate:status ----------------------------------------------------------------------------------------------------- Migration ID Status Total Imported Unprocessed Last Imported ----------------------------------------------------------------------------------------------------- upgrade_d7_date_formats Idle 7 0 0 upgrade_d7_filter_settings Idle 1 0 0 upgrade_d7_image_styles Idle 193 0 0 upgrade_d7_node_settings Idle 1 0 0 upgrade_d7_system_date Idle 1 0 0 upgrade_d7_url_alias Idle 25981 0 0 upgrade_system_site Idle 1 0 0 upgrade_taxonomy_settings Idle 0 0 0 upgrade_d7_path_redirect Idle 518 0 0 upgrade_d7_field Idle 253 0 0 upgrade_d7_field_collection_type Idle 9 0 0 upgrade_d7_node_type Idle 16 0 0 upgrade_d7_taxonomy_vocabulary Idle 34 0 0 upgrade_d7_field_instance Idle 738 0 0 upgrade_d7_view_modes Idle 24 0 0 upgrade_d7_field_formatter_settings Idle 1280 0 0 upgrade_d7_field_instance_widget_settings Idle 738 0 0 upgrade_d7_file Idle 5731 0 0 upgrade_d7_filter_format Idle 6 0 0 upgrade_d7_menu Idle 5 0 0 upgrade_d7_user_role Idle 6 0 0 upgrade_d7_user Idle 82 0 0 upgrade_d7_node_article Idle 322400 0 0 upgrade_d7_node_page Idle 342 0 0 upgrade_d7_menu_links Idle 623 0 0 upgrade_d7_node_revision_article Idle 742577 0 0 upgrade_d7_node_revision_page Idle 2122 0 0 upgrade_d7_taxonomy_term_tags Idle 1729 0 0 -------------------------------------------- -------- ------- ---------- ------------- --------------- ``` You should inspect any contributed modules in use on the old site and install them in the new one as they may contain migrations. For example, if the old site is using [Redirect](https://git.drupalcode.org/project/redirect/blob/8.x-1.x/migrations/d7_path_redirect.yml) module, by installing itm and generating migrations like we did above, you should see a new migration provided by this module ready to go. ## Running migrations Assuming that we decided to run migrations generated by Migrate Upgrade (otherwise skip the following sub-section), we would first run configuration migrations and then content ones. ### Configuration migrations Here is the command to run all migrations with the tag `Configuration` along with their dependencies. ``` $ drush migrate:import --tag=Configuration --execute-dependencies [notice] Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_date_formats' [notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_filter_settings' [notice] Processed 20 items (20 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_image_styles' [notice] Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_settings' [notice] Processed 1 items (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_date' [notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_site' [notice] Processed 100 items (100 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field' [notice] Processed 200 items (200 created, 0 updated, 0 failed, 379 ignored) - done with 'upgrade_d7_field_instance' ... ``` The above command will migrate site configuration, content types, taxonomy vocabularies, view modes, and such. Once this is done, it is recommended to export the resulting configuration via `drush config:export` and commit the changes. From then on, if we make changes in the old site’s configuration or we alter the Configuration migrations (we will see how further down), we will need to roll back the affected migrations and run them again. For example, the [Media Migration](https://www.drupal.org/project/media_migration) module creates media fields and alters field mappings in content migrations so after installing it we should run the following commands to re-run them: ``` $ drush --yes migrate:rollback upgrade_d7_view_modes,upgrade_d7_field_instance_widget_settings,upgrade_d7_field_formatter_settings,upgrade_d7_field_instance,upgrade_d7_field $ drush --yes migrate:import --execute-dependencies upgrade_d7_field,upgrade_d7_field_instance,upgrade_d7_field_formatter_settings,upgrade_d7_field_instance_widget_settings,upgrade_d7_view_modes $ drush --yes config:export ``` Once we have executed all Configuration migrations, we can run content ones. ### Content migrations Content migrations are straightforward. We can run them with the following command: ``` $ drush migrate:import --tag=Content --execute-dependencies ``` ## Logging and tracking Migrate keeps track of all the migrated content via the migrate\_\* tables. If you check out the new database after running migrations, you will see something like this: - A set of *migrate\_map\** tables, storing the old and new identifiers of each migrated entity. These tables are used by the `migrate:rollback` Drush command to roll back migrated data. - A set of *migrate\_messages\** tables, which hold errors and warnings that occurred while running migrations. These can be seen via the `migrate:messages` Drush command. ## Rolling back migrations In the previous section, we rolled back the field migrations before running them again. This process is great for reverting imported Configuration or Content, which you will do often while developing a migration. Here is an example. Let’s suppose that you have executed content migrations but articles did not get migrated as you expected. The process you would follow to fix them would be: 1. Find the migration name via `drush migrate:status | grep article`. 2. Roll back migrations with `drush migrate:rollback upgrade_d7_node_revision_article,upgrade_d7_node_article`. 3. Perform the changes that you need either directly at the exported migration at `config/sync` or by altering them and then recreating them with `migrate:upgrade` like we did at *Generating migrations with Migrate Upgrade*. We will see how to alter migrations in the next section. 4. Run the migrations again with `drush migrate:import upgrade_d7_node_article,upgrade_d7_node_revision_article`. 5. Verify the changes and, if needed, repeat steps 2 to 4 until you are done. ## Migrate events and hooks Before diving into the APIs to alter migrations, let’s clarify that there are two processes that we can hook into: 1. The `migrate:upgrade` Drush command, which reads all migration files in core, contributed, and custom modules and imports them into the database. 2. The `migrate:import` Drush command, which runs migrations. In the next sub-sections, we will see how can we interact with these two commands. ### Altering migrations (migrate:upgrade) Drupal core offers [hook\_migration\_plugins\_alter()](https://api.drupal.org/api/drupal/core%21modules%21migrate%21migrate.api.php/function/hook_migration_plugins_alter), which receives the array of available migrations that `migrate:upgrade` creates. Here is a sample implementation at `mymodule.module` where we delegate the logic to a service: ``` /** * Implements hook_migration_plugins_alter(). */ function mymodule_migration_plugins_alter(array &$migrations) { $migration_alterer = \Drupal::service('mymodule.migrate.alterer'); $migration_alterer->process($migrations); } ``` And here is a subset of the contents of the service: ``` class MigrationAlterer { /** * Processes migration plugins. * * @param array $migrations * The array of migration plugins. */ public function process(array &$migrations) { $this->skipMigrations($migrations); $this->disablePathautoAliasCreation($migrations); $this->setContentLangcode($migrations); $this->setModerationState($migrations); $this->persistUuid($migrations); $this->skipFileCopy($migrations); $this->alterRedirect($migrations); } /** * Skips unneeded migrations. * * @param array $migrations * The array of migration plugins. */ private function skipMigrations(array &$migrations) { // Skip unwanted migrations. $migrations_to_skip = [ 'd7_block', 'd7_comment', 'd7_comment_entity_display', 'd7_comment_entity_form_display_subject', 'd7_comment_field', 'd7_comment_entity_form_display', 'd7_comment_type', 'd7_comment_field_instance', 'd7_contact_settings', ]; $migrations = array_filter($migrations, function ($migration) use ($migrations_to_skip) { return !in_array($migration['id'], $migrations_to_skip); }); } // The remaining methods would go here. } ``` In the next section, we will see how to alter the data being migrated while running migrations. ### Altering data while running migrations (migrate:import) Drupal core offers [hook\_migrate\_prepare\_row()](https://api.drupal.org/api/drupal/core%21modules%21migrate%21migrate.api.php/function/hook_migrate_prepare_row/8.7.x) and [hook\_migrate\_MIGRATION\_ID\_prepare\_row](https://api.drupal.org/api/drupal/core%21modules%21migrate%21migrate.api.php/function/hook_migrate_MIGRATION_ID_prepare_row/8.7.x), which are triggered before each row of data is processed by the `migrate:import` Drush command. Additionally, there is a [set of events](https://api.drupal.org/api/drupal/core%21modules%21migrate%21src%21Event%21MigrateEvents.php/class/MigrateEvents/8.7.x) that we can subscribe to such as before and after the migration starts or before and after a row is saved. On top of the above, Migrate Plus module exposes an [event that wraps hook\_migrate\_prepare\_row()](https://git.drupalcode.org/project/migrate_plus/blob/8.x-5.x/src/Event/MigrateEvents.php#L12). Here is a sample subscriber for this event: ``` class MyModuleMigrationSubscriber implements EventSubscriberInterface { /** * Prepare row event handler. * * @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event * The migrate row event. * * @throws \Drupal\migrate\MigrateSkipRowException * If the row needs to be skipped. */ public function onPrepareRow(MigratePrepareRowEvent $event) { $this->alterFieldMigrations($event); $this->skipMenuLinks($event); $this->setContentModeration($event); } /** * Alters field migrations. * * @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event * The migrate row event. * * @throws \Drupal\migrate\MigrateSkipRowException * If a row needs to be skipped. * @throws \Exception * If the source cannot be changed. */ private function alterFieldMigrations(MigratePrepareRowEvent $event) { $field_migrations = [ 'upgrade_d7_field', 'upgrade_d7_field_instance', 'upgrade_d7_view_modes', 'upgrade_d7_field_formatter_settings', 'upgrade_d7_field_instance_widget_settings', ]; if (in_array($event->getMigration()->getPluginId(), $field_migrations)) { // Here are calls to private methods that alter these migrations. } } /** * Skips menu links that are either implemented or not needed. * * @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event * The migrate row event. * * @throws \Drupal\migrate\MigrateSkipRowException * If a row needs to be skipped. */ private function skipMenuLinks(MigratePrepareRowEvent $event) { if ($event->getMigration()->getPluginId() != 'upgrade_d7_menu_links') { return; } $paths_to_skip = [ 'some/path', 'other/path', ]; $menu_link = $event->getRow()->getSourceProperty('link_path'); if (in_array($menu_link, $paths_to_skip)) { throw new MigrateSkipRowException('Skipping menu link ' . $menu_link); } } /** * Sets the content moderation field on node migrations. * * @param \Drupal\migrate_plus\Event\MigratePrepareRowEvent $event * The migrate row event. * * @throws \Exception * If the source cannot be changed. */ private function setContentModeration(MigratePrepareRowEvent $event) { $row = $event->getRow(); $source = $event->getSource(); if (('d7_node' == $source->getPluginId()) && isset($event->getMigration()->getProcess()['moderation_state'])) { $state = $row->getSourceProperty('status') ? 'published' : 'draft'; $row->setSourceProperty('moderation_state', $state); } elseif (('d7_node_revision' == $source->getPluginId()) && isset($event->getMigration()->getProcess()['moderation_state'])) { $row->setSourceProperty('moderation_state', 'draft'); } } } ``` ## Conclusion When you are migrating a Drupal site to 8, Migrate Upgrade module does a lot of the work for free by generating Configuration and Content migrations. Even if you decide to write any of them by hand, it is convenient to run the command and use the resulting migrations as a template for your own. In the next article, we will see how to work seamlessly on a migration while the rest of the team is building the new site. *Thanks to Andrew Berry, April Sides, Karen Stevenson, Marcos Cano, and Salvador Moreno for their feedback and help. Photo by [Barth Bailey](https://unsplash.com/@7bbbailey?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText).* Published in: - [ Data Migration ](/topics/data-migration) - [ Drupal Development ](/topics/drupal-development) You must have JavaScript enabled to use this form. ## Get in touch with us ### Tell us about your project. We'd love to hear from you! First Name Last Name Your email address Your organization How you heard about us Tell us more Leave this field blank