Drupal Migrate Plus and Custom Source Plugins: The Working Guide

Every Drupal migration reaches the same moment: the YAML you can write in configuration stops being enough. The source is a database nobody documented, or an API that pages through results, or a spreadsheet exported from a system that went out of support in 2014. Migrate Plus gets you further than core on its own, and then you write a source plugin. This is the working guide to both — what Migrate Plus actually adds, when a custom source plugin is the right answer, and the shape of one that survives contact with real data.

It assumes Drupal 10 or 11, and that you have read the core Migrate API documentation at least once. It is written from migrations we have run, not from the handbook.

What core gives you, and what Migrate Plus adds

Core’s Migrate API is the engine: a migration is a source plugin, a process pipeline and a destination plugin, tied together by a YAML definition, tracked in map and message tables so it can be rerun, rolled back and resumed. Core also ships the Drupal-to-Drupal migrations that Migrate Drupal uses, which is why a Drupal 7 upgrade can be surprisingly hands-off. Our Drupal 7 to 11 checklist covers that path.

Migrate Plus adds the things you reach for the moment the source is not Drupal. Migration groups, so a set of migrations shares a database key and can be run and rolled back together. A url source plugin with data fetchers and parsers, so JSON, XML and SOAP endpoints become sources without code. Process plugins that core lacks — entity_lookup and entity_generate for matching or creating referenced entities on the fly, skip_on_value, str_replace, dom for manipulating HTML in the pipeline. And migrations as configuration entities, which is what lets you keep them in config export alongside everything else. The Drush migrate:* commands live in Drush itself now; on older stacks they came from the companion Migrate Tools module.

When a custom source plugin is the right answer

Try not to write one. The url plugin with a JSON parser covers most APIs; the contributed CSV source covers most spreadsheets; core’s SqlBase subclasses cover a surprising amount of legacy SQL through configuration alone. Write a plugin when one of these is true:

The source query needs joins, subqueries or conditions that a declarative source cannot express — a product with its current price in a second table and its brand in a third.

Rows need cleaning before the pipeline sees them. Legacy HTML with inline styles, encoded entities, or a soft-delete flag that means “skip this” belongs in prepareRow(), not in twelve process plugins.

The source is incremental. A nightly sync from a system that keeps running needs a high-water mark, and a plugin gives you clean control over it.

The source is not a thing Migrate knows how to fetch — a proprietary export, a message queue, a filesystem walk.

The anatomy of a source plugin

A SQL-backed source extends SqlBase and implements four methods. This one pulls products from a legacy catalogue database declared in settings.php under the legacy key.

<?php

namespace Drupal\legacy_migrate\Plugin\migrate\source;

use Drupal\migrate\Plugin\migrate\source\SqlBase;
use Drupal\migrate\Row;

/**
 * Products from the legacy catalogue database.
 *
 * @MigrateSource(
 *   id = "legacy_product",
 *   source_module = "legacy_migrate"
 * )
 */
class LegacyProduct extends SqlBase {

  public function query() {
    $query = $this->select('catalogue_product', 'p')
      ->fields('p', ['product_id', 'sku', 'title', 'body', 'brand_id', 'updated']);
    $query->leftJoin('catalogue_price', 'pr', 'pr.product_id = p.product_id');
    $query->addField('pr', 'amount', 'price');
    return $query;
  }

  public function fields() {
    return [
      'product_id' => $this->t('Legacy product ID'),
      'sku'        => $this->t('SKU'),
      'title'      => $this->t('Title'),
      'body'       => $this->t('Description (HTML)'),
      'brand_id'   => $this->t('Legacy brand ID'),
      'price'      => $this->t('Current price'),
      'updated'    => $this->t('Last updated (unix)'),
    ];
  }

  public function getIds() {
    return ['product_id' => ['type' => 'integer', 'alias' => 'p']];
  }

  public function prepareRow(Row $row) {
    // The legacy system soft-deleted by blanking the SKU.
    if ($row->getSourceProperty('sku') === NULL) {
      return FALSE;
    }
    $row->setSourceProperty('body', $this->cleanLegacyHtml($row->getSourceProperty('body')));
    return parent::prepareRow($row);
  }

  private function cleanLegacyHtml(?string $html): string {
    $html = (string) $html;
    $html = preg_replace('/ style="[^"]*"/', '', $html);
    return trim($html);
  }

}

Four things to notice. query() returns a select query, not results, so Migrate can count, page and add the high-water condition itself. fields() is documentation that the UI and migrate:fields-source read; keep it honest. getIds() is what the map table keys on, and the alias matters when the ID column is ambiguous across a join. Returning FALSE from prepareRow() skips the row cleanly and records it as ignored, which is the right behaviour for soft-deleted records; throwing an exception is not.

The migration definition that uses it

id: legacy_product
label: Legacy catalogue products
migration_group: legacy
source:
  plugin: legacy_product
  key: legacy
  high_water_property:
    name: updated
    alias: p
process:
  title: title
  field_sku: sku
  'body/value': body
  'body/format':
    plugin: default_value
    default_value: basic_html
  field_price: price
  field_brand:
    plugin: migration_lookup
    migration: legacy_brand
    source: brand_id
destination:
  plugin: 'entity:node'
  default_bundle: product
migration_dependencies:
  required:
    - legacy_brand

The high-water mark on updated is what makes this rerunnable: the second run only fetches rows changed since the last one. migration_lookup resolves the legacy brand ID to the node the legacy_brand migration created, and the dependency guarantees it ran first. Put the group’s database key in the group entity, not in every migration, and every migration in the group can be run with one command.

Running it, and what goes wrong

drush migrate:status --group=legacy before anything, then drush migrate:import legacy_product --limit=500 --feedback=100 for the first pass — the limit is not caution, it is how you find the row that breaks the pipeline without waiting an hour to reach it. Check the messages table, fix, roll back with drush migrate:rollback legacy_product, run again. When a migration is stuck showing “Importing” after a crash, drush migrate:reset-status is the fix, not the database.

The failures are consistent across projects. Memory: a source that loads everything in query() is fine at a thousand rows and dead at a million; let Migrate page it, and never build arrays of the whole set in prepareRow(). IDs that are not unique: a composite key in getIds() when the legacy system reused numbers. Lookups to migrations that have not run, which look like empty references and are really an ordering problem. Encoding: a Latin-1 database read as UTF-8 produces garbage that only shows up in the third test. And the high-water mark that never advances because the source column is a string, not an integer or a date.

Where this sits in the larger job

A source plugin is a day of work when the source is understood and a week when it is not, and the difference is almost entirely in how well the legacy system was documented. Budget the discovery before the code. In a full migration the plugins are the smallest part; the content model mapping, the URL discipline and the testing are where the calendar goes, and our 90-day migration calendar shows how they fit together.

Gspikes builds Drupal migrations from sources that were never meant to be migrated from — legacy databases, retired CMSs, exports nobody remembers producing. If you have one of those, describe the source and we will tell you whether it is a configuration job or a plugin. More on our Drupal development page, on what a migration costs, and on what to look for when hiring for one.