Managing Authentication During API Migrations

In the article, Hide Your Keys, Hide Your Access, I discussed using environment variables as a way to keep access credentials and sensitive data out of your code repository. Now, let's take a look at how environment variables can be used during API migrations.

For the purposes of the following examples, let's assume the .env file defines the following variables:

MY_API_USERNAME=myapiusername
MY_API_PASSWORD=myapipassword
MY_API_KEY=myapikey
ANOTHER_API_KEY=anotherapikey
ANOTHER_API_SECRET=anotherapikey

What about migration configuration files?

Because migration configuration files are Drupal configuration, you can use a configuration override in the settings.php file. A configuration override will set or override the values during run time, but will not store them in the database or export them to the configuration YAML files.

To determine what the configuration overrides will look like, you need to review the migration configuration files and determine where the secure values should be placed.

Migration Configuration

There are various authentication methods used to protect data access for an API endpoint. Let’s look at examples for basic authentication and URL Parameters.

Basic Authentication example

An endpoint may require a basic authentication username and password, meaning you have to log in to access the data. If so, a migration YAML configuration file named, migrate_plus.migration.my_basic_auth_migration_node_articles.yml, might include something like this:

...
id: my_basic_auth_migration_node_articles
...
migration_tags:
  - my_basic_auth_api_auth_from_env
...
source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  headers:
    Accept: 'application/json; charset=utf-8'
    Content-Type: application/json
  authentication:
    plugin: basic
    username: username_from_env
    password: password_from_env
  urls:
    - 'https://api.example.com/'
  item_selector: data
  fields:
    -
      name: id
      label: Id
      selector: id
    -
      name: attributes
      label: Attributes
      selector: attributes
  ids:
    id:
      type: integer
...

For this example, the following can be added to the settings.php file to insert the secure username and password at run time:

$config['migrate_plus.migration.my_basic_auth_migration_node_articles']['source']['authentication']['username'] = getenv('MY_API_USERNAME'); 
$config['migrate_plus.migration.my_basic_auth_migration_node_articles']['source']['authentication']['password'] = getenv('MY_API_PASSWORD');

Note that the top-level config value matches the file name, without the .yml extension. Adding this code will transform the migration configuration at run time from:

source:
  authentication:
    plugin: basic
    username: username_from_env
    password: password_from_env

to:

source:
  authentication:
    plugin: basic
    username: myapiusername
    password: myapipassword

URL Parameter Example

An endpoint may require that a key or secret is passed as a URL parameter like:

https://api.example.com/query?output=JSON&apikey=anotherapikey

In this case, a migration YAML configuration file named migrate_plus.migration.my_url_param_migration_node_articles.yml might include something like this:

...
id: my_url_param_migration_node_articles
...
migration_tags:
  - my_url_param_api_key_from_env
...
source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  headers:
    Accept: 'application/json; charset=utf-8'
    Content-Type: application/json
  urls:
    - 'https://api.example.com/query?output=JSON'
  item_selector: data
  fields:
    -
      name: id
      label: Id
      selector: id
    -
      name: attributes
      label: Attributes
      selector: attributes
  ids:
    id:
      type: integer
...

For this example, the following is added to the settings.php file to add the apikey value to each URL at run time:

foreach ($config['migrate_plus.migration.my_url_param_migration_node_articles']['source']['urls'] as $key => $url) {
  $config['migrate_plus.migration.my_url_param_migration_node_articles']['source']['urls'][$key] = $url . '&apikey=' . getenv('MY_API_KEY');
}

This will transform the migration configuration from:

source:
  urls:
    - 'https://api.example.com/query?output=JSON'

to:

source:
  urls:
    - 'https://api.example.com/query?output=JSON&apikey=myapikey'

But I have multiple migrations accessing this API endpoint. Is there a better way?

With this technique, each migration that accesses the secure API endpoint will need to be overridden in the settings.php file. But, there is another way...a custom source plugin!

Custom Source Plugin

The custom source plugin will live in a custom module, typically the one used to customize migrations. For this example, a new file is created in the src/migrate/source/ directory of a my_custom_migration_module module called MyCustomUrlPlugin.php. The file contents will start something like this:

<?php

namespace Drupal\my_custom_migration_module\Plugin\migrate\source;

use Drupal\migrate\Plugin\MigrationInterface;
use Drupal\migrate_plus\Plugin\migrate\source\Url;

/**
 * Source plugin for retrieving data via URLs, securely.
 *
 * @MigrateSource(
 *   id = "my_custom_url_plugin"
 * )
 */
class MyCustomUrlPlugin extends Url {

  /**
   * {@inheritdoc}
   */
  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration) {
    
    // Run the parent constructor.
    parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);
  }

}

You can see that this new class extends the migrate_plus Url source plugin. An entirely new source plugin doesn’t need to be created. You just need to retrieve your secure environment variables and insert them into the migration configuration before the migration process begins.

As it currently stands, this code will run the same as the Url source plugin because the parent constructor at the end of the __construct method is called. You will need to update your migration configuration files to use this new source plugin by changing this:

source:
  plugin: url

to:

source:
  plugin: my_custom_url_plugin

You also need to rebuild the cache so that Drupal will recognize your new source plugin.

Identifying Your Secure Migrations

You can use this custom source plugin to apply to multiple migrations and authentication methods using conditions, but you need a way to determine which method should be applied to the current migration.

One way to do this is to add custom migration tags to reference a particular API endpoint. To do this, you need to add the following to the beginning of the __construct method to retrieve the current migration’s tags:

// Get the current migration tags.
$migration_tags = $migration->getMigrationTags();

Basic Authentication Example

In the previous migration configuration examples, a custom migration tag called my_basic_auth_api_auth_from_env for our basic authentication migration was included. To add your username and password, you can include this snippet at the beginning of the __construct method, before the parent constructor is called:

// If this is my basic authentication API migration, inject the username and password values into the authentication configuration.
if (in_array('my_basic_auth_api_auth_from_env', $migration_tags)) {
  if (isset($configuration['authentication'])) {
    $configuration['authentication']['username'] = getenv('MY_API_USERNAME');
    $configuration['authentication']['password'] = getenv('MY_API_PASSWORD');
  }
}

This condition will recognize the migration tag and add the authentication credential values before each migration process begins.

URL Parameter Example

In the migration configuration file examples, a custom migration tag called my_url_param_api_key_from_env for the URL parameter migration was included. To add the apikey parameter to the migration URLs, you can include this snippet at the beginning of the __construct method, before the parent constructor is called:

// If this is my URL parameter API migration, add the parameter to the migration URLs.
if (in_array('my_url_param_api_key_from_env', $migration_tags)) {
  // Append the apikey parameter to each URL.
  foreach ($configuration['urls'] as $key => $url) {
    $configuration['urls'][$key] = $url . '&apikey=' . getenv('MY_API_KEY');
  }
}

Again, the condition will recognize the migration tag and add authentication key value before each migration process begins.

Things to Remember

In these examples, the migrations were named after the type of authentication used. In your real-world migrations, you will want to name, group, and tag migrations based on the data source, not on the authentication type.

Thanks to James Sansbury, Juampy NR, and Salvador Molina Moreno for their feedback.