Drupal 10 Migration from API Endpoint and Drupal 7

As everyone in the Drupal community is aware, Drupal 7 is officially reaching end of life on January 5, 2025 and migrations are in full swing. We recently migrated a client’s website from Drupal 7 to Drupal 10 and it proved to be an interesting opportunity to migrate content from an API endpoint along with the content on the Drupal 7 database.

The goal was to take advantage of the Migrate Module provided in Drupal core and update content from the API endpoint after the initial migration from Drupal 7. I’ve done a few Drupal migrations, but this one was unique with challenging obstacles along the way. 

Check out my GitHub repository for the code examples below.

Phase 1: Drupal 7 migration

To get started, I created a custom Drupal module for the migrations. Typically when I build migration modules, I create my migration files inside of the install directory where they would be a part of the configuration. In this case, I wanted the ability to modify my migration files. If you want to modify your migration files, you can place them inside of a migrations directory. This separates them from being a part of the configuration. 

• custom_module/config/install - migration files install to configuration.
• custom_module/migrations - migration files outside of configuration.

Below is an example of the fields that are processed in the migration file. The station_name is part of the API endpoint and migrated as title and passed as a value in source to a migration process plugin station_field created to query the Drupal 7 database.

process:
  # Migrate field from API endpoint.
  title: station_name
  # Migrate field from D7 database for initial migration.
  field_station: 
    - 
      plugin: station_field
      source: 
        stations_id: 'station_name'

In the code below overwrite_properties defines fields to be overridden in the migration file. This will allow for these fields to be updated on the entity ID if it already exists, while preserving the other field values that are part of the entity. With this property I could import new content and update any previously imported content from the migration. 

destination:
  plugin: 'entity:node'
  default_bundle: stations
  overwrite_properties:
    - title
    - field_station

The migrate plugin process class uses the station_name value as an identifier to query the Drupal 7 database. This query worked for my use case. You would want to develop your own solution on how to pass a unique identifier to query data for your database, but you can follow the same steps of passing the value to your transform function inside of the migrate plugin process class.

 public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
   $station_name = isset($value[0]) ? $value[0] : NULL;
   if($station_name == NULL) {
     return NULL;
   }
   // Set database connection to migrate Drupal 7.
   $db = \Drupal\Core\Database\Database::getConnection('migrate', 'migrate');
   // Query using station_name to get entity node.
   $query = $db->select('field_data_field_station', 'n');
   $query->fields('n', ['entity_id']);
   $query->condition('n.field_station_value', $station_name, '=');
   $query->condition('n.bundle', 'station_station', '=');
   $query->orderBy('n.entity_id', 'ASC');
   $query->range(0, 1);
   $result = $query->execute()->fetchAssoc();
   if($result == FALSE) {
     return NULL;
   }
   $nid = $result['entity_id'];
   // Query the entity by id and field name desitination property.
   $query = $db->select('field_data_' . $destination_property, 'f');
   $query->fields('f', [$destination_property .'_value']);
   $query->condition('f.entity_id', $nid, '=');
   $results = $query->execute()->fetchAssoc();
   if($results == FALSE) {
     return NULL;
   }
   // Return the migration field value for Drupal 10 migration.
   $field_value = $results[$destination_property . '_value'];
   return $field_value;
}

The line of code below uses the Database::getConnection | Drupal API to establish the connection to the Drupal 7 database.

$db = \Drupal\Core\Database\Database::getConnection('migrate', 'migrate');

A second database connection was configured in settings.php called migrate. 

$databases['migrate']['default'] = [...

If you are not familiar with database connections, I suggest looking at the Drupal Database configurations to get a better understanding. 

With the database connection, I wrote two queries to migrate the field value. The first query uses the unique identifier passed from the migration file to query the entity ID and the second query uses the entity ID to return the field value to the migration field.

Phase 2: API migration updates

With the migration file and migrate process plugin created, I could import and update the content from the API endpoint and Drupal 7 database with the Drush Migrate command example below.

drush migrate-import stations_import

This solved my first goal of migrating the Drupal 7 content. It was easy to remove the fields for the Drupal 7 migration and leave the fields for the API endpoint because all that I needed to do was delete the fields necessary in the migration files. The next step was to develop a way for an administrator to log in and run the migrations to update the content from the API endpoint. To do this I created a route and controller. 

I created a route similar to the one below where the migration_id is passed in the path for the controller. 

migrate_examples.import_migration:
path: '/admin/import/migrate/{migration_id}'
defaults:
  _controller: '\Drupal\migrate_examples\Controller\ImportController::importMigration'
  _title: 'Import stations'
options:
  parameters:
    migration_id:
      type: integer
requirements:
  _permission: 'access administration pages'

I provided the route on administration pages as a button to be clicked to run the migration updates. The function below creates the URL from the route and migration_id, and the link is created from the url and with the given button text. The link is styled with the button class.

public function createMigrationButton($migration_id, $route_name, $button_text) {
  $url = Url::fromRoute($route_name, ['migration_id' => $migration_id]);
  $link = Link::fromTextAndUrl($this->t($button_text), $url)->toRenderable();
  $link['#attributes'] = ['class' => ['button']];
  return $link;
}

One important note: I used Drupal\migrate_tools\MigrateExecutable class from the migrate_tools contrib module instead of the one provided by the migrate module in Drupal core. By using this class I was able to run the migration import from the importMigration function below in my Controller class.

public function importMigration($migration_id) {
  $migration_plugin_manager = \Drupal::service('plugin.manager.migration');
  $migration = $migration_plugin_manager->createInstance($migration_id);
  $request = \Drupal::request();
  $referer = $request->headers->get('referer');      
  if(!$migration) {
    $message = $this->t('Migration id is invalid.');
    \Drupal::messenger()->addMessage($message);
    \Drupal::logger('stations_import')->notice($message);
    return new RedirectResponse($referer);
  }
  $executable = new MigrateExecutable($migration, new MigrateMessage());
  $executable->import();
  $message = $this->t('Imported all rows.');
  \Drupal::messenger()->addMessage($message);
  \Drupal::logger('stations_import')->notice($message);
  return new RedirectResponse($referer);
}

I needed to build a page for administrators to view the API endpoint data before importing. I created an administration page to display the rows of data with a table and pager. In the code below, the build array is returned in a Controller for the administration page that sets the #type to declare using the table and pager theming templates provided by Drupal. It also uses the pager service to help calculate the rows and page to display.

// Build table and pager for migrate import.
$header_row = [
  'ID',
  'Title',
];
$total = count($rows);
$limit = 10;
$build[] = [
  '#theme' => 'stations_table__header',
];
$pager_manager = \Drupal::service('pager.manager');
$current_page = $pager_manager->createPager($total, $limit)->getCurrentPage();
$data = array_slice($rows, $current_page * $limit, $limit);
$pager_manager->createPager($total, $limit);
$build['table'] = [
  '#type' => 'table',
  '#header' =>  $header_row,
  '#rows' => $data,
  '#caption' => '',
];
$build['import_all'] = $this->createMigrationButton('stations_import',
'migrate_examples.import_migration', 'Import All');
$build['pager'] = [
  '#type' => 'pager',
];
return $build;

At this point the migrate module could be used to update the content from the API endpoint through the administration page through the controllers and routes created. I created a few more routes to run migrations for each row based on row ID and to rollback the migration. The end result looked like this:

This sums up how I was able to migrate data from an API endpoint and content from another Drupal 7 database and then used Drupal Migrate and Migrate Tools modules to migrate and update content from the API endpoint.