Main navigation

Writing a Custom Migration Source Plugin in Drupal 8

18/12/2017 · par Jigar Mehta

Usually, Drupal migrations get run at the beginning of a project to get all your content into Drupal. But sometimes, you need to create a migration that runs on a regular basis. For example, you have an external database of courses and programs maintained by a university that needs to be displayed on a Drupal site. Another example: you have a database of book data that needs to be pulled into Drupal nightly.

When working with migrations where the source files are updated every day, it can get really tedious to download the updated source files manually each time the migration runs.

In this tutorial, we'll write a source plugin based on the CSV source plugin which will allow us to automatically download CSV files from a remote server via SFTP before running migrations. This article was co-authored by my colleague David Valdez - gracias David for your contribution.

The Problem

In a project we worked on recently, we had the following situation:

CSV files are updated by a PowerShell script every night on the client's server.

These CSV files are accessible via SFTP.

Our task is to download the CSV source files over SFTP and to use them as our migration source.

Before We Start

This articles assumes that you can write custom modules. If you have never written a custom module, you can try reading this article on creating custom modules.

It is assumed that you have working knowledge of migrations in Drupal 8. If you are new to Drupal 8 migrations, I recommend you to start by reading these articles first:

The Plan

The goal is to avoid downloading the file manually every time we run our migrations. So we need a way to doing this automatically everytime we execute a migration. To achieve this, we create a custom source plugin extending the CSV plugin provided by the Migrate Source CSV module, which will download CSV files from a remote server and pass it to the CSV plugin to process them.

The Source Migrate Plugin

To start, let's create a custom module and call it migrate_example_source and implement a custom migrate source plugin by creating a PHP class inside it at /src/Plugin/migrate/source/MigrateExampleSourceRemoteCSV.php

We start implementing the class by simply extending the CSV plugin provided by the migrate_source_csv module:

If you are building a source plugin from scratch, you will need to extend the SourcePluginBase class instead of the CSV class given in this example. Adding the annotation @MigrateSource is very important because that is what will make the migrate module detect our source plugin. In our plugin, we use the phpseclib/phpseclib libraries to make SFTP connections. Hence, we need to include the libraries in our project by running the following command in the Drupal root:

composer require phpseclib/phpseclib

Our plugin will download the source CSV file and will simply pass it to the CSV plugin to do the rest. We do the download when the plugin is being instantiated like this:

Note: The code block above has been simplified a bit. If you see the actual source plugin, there are some lines of code which make things more compatible with the migration_lookup plugin.

This method creates an SFTP connection, downloads the file to a temporary location and returns the path to the downloaded file. The temporary file path is then passed to the Migrate Source CSV and that's it! Finally, to use the plugin in our migration we just set our plugin as the source/plugin: