Entity Import: A User Interface for Drupal 8 Migrations

If you’ve ever wished there was a way to easily import CSV data into your Drupal website then this post is for you. With the new Entity Import module, website admins can easily add and configure new importers that map CSV source files to Drupal 8 entities. Entity Import adds a user interface for Drupal 8’s core migration functionality. It lets you add importers for any entity in the system, map source data to entity fields, configure process pipelines, and of course, run the import.

Why Another Drupal Import Module?

In Drupal 8 there are already several good options for importing and migrating content using the admin interface.

The Feeds module is one approach for importing content, with a great user interface and huge community of support. Feeds has been around for 7+ years, has been downloaded 1.8 million times, and, according to its Drupal project page, is being used on more than 100,000 websites.

The Migrate module, a part of core for Drupal 8, is an incredibly powerful framework for migrating content into Drupal. With Migrate, developers can create sophisticated migrations and leverage powerful tools like rollback, Drush commands, and countless process plugins for incredibly complex pipelines.

We use both Migrate and Feeds extensively. (Check out this recent post from Joel about getting started with Migrate.) Recently, though, we needed something slightly different: First, we wanted to provide Drupal admins with a user interface for configuring complex migrations with dependencies. Second, we needed Drupal admins to be able to easily run new imports using their configured migrations by simply uploading a CSV. Essentially, we wanted to give Drupal admins an easy-to-use control panel built on top of Drupal core’s migrate system.

My First Use Case: Complex Data Visualizations

Here at Aten, we do a lot of work helping clients with effective data visualizations. Websites like the Guttmacher Institute’s data center and the Commonwealth Fund’s 2018 interactive scorecard are great examples. When I started working on another data-heavy project a few months ago, I needed to build a system for importing complex datasets for dynamic visualizations. Drupal’s core migration system was more than up for the task, but it lacks a simple UI for admins. So I set about building one, and created Entity Import.

Getting Started with Entity Import

Download and Install

Entity Import is a Drupal module and can be downloaded at https://Drupal.org/project/entity_import. Alternatively, you can install Entity Import with composer:

composer require drupal/entity_import

Entity Import is built directly on top of Drupal’s Migrate module, and no other modules are required.

Adding New Importers

Once the Entity Import module is installed, go to Admin > System > Entity Import to add new importers. Click “Add Importer.”

For each importer, you will need to provide:

• Name - In my case I used “Dataset,” “Indicators,” and “Topics.” Generally speaking, I would name your importers after whatever types of data you are importing.
• Description - An optional description field to help your Drupal administrators.
• Page - Toggle this checkbox if you want to create an admin page for your importer. Your administrators will use the importer page to upload their CSV files.
• Source Plugin - At the time of this writing, Entity Import provides just one source plugin: CSV. The system is fully pluggable, and I hope to add others – like XML or even direct database connections – in the future.
• Plugin Options - Once you choose your source Plugin (i.e. CSV) you’ll have plugin-specific configuration options. For CSVs, you can choose whether or not to include a header row, as well as whether or not to support multiple file uploads.
• Entity Type - Specify which entity type your data should be imported into.
• Bundles - Once you pick a entity type, you can choose one or more bundles that your data can be imported into.

Configuring Importers

Each Importer you add will have its own configuration options, available under Admin > Configuration > System > Entity Importer. Configuration options include:

• Mappings - Add and configure mappings to map source data to the appropriate destination fields on your entity bundles. (Important side note: when creating mappings, the human readable name can be whatever you wish; the machine name, however, needs to match the column header from your source data.)
• Process Plugins - When you add new mappings you can specify one or more process plugins directly in the admin interface. This is where things get interesting – and really powerful. Drupal 8 provides a number of process plugins for running transformations on data to be migrated (read more about process plugins in Joel’s recent migrations post). With Entity Import, you can specify one or more process plugins and drag them into whatever order you wish, building process pipelines as complicated (or simple) as you need. Your admins can even manage dependencies on other imports; for example, mapping category IDs for an article importer to categories from a category importer. Again, no coding necessary. Entity Import provides a user interface for leveraging some of Migrate’s most powerful functionality.

Importing the Data

Once you’ve added and configured importers, go to Admin > Content > [Importer Name] to run a new import. You’ll see three tabs, as follows:

• Import - This is the main import screen with upload fields to upload your CSV(s). If your import has dependencies specified in any of its process plugins, the import(s) it depends on will show up on this screen as well. (TODO: Currently, the interface for managing multiple, interdependent imports is a little complicated. I’d like to make it easier for users to visualize the status of all interdependent migrations at-a-glance.)
• Actions - After you run an import, you can rollback using the options on the actions tab. (TODO: I’d like to add other actions as well; for example, the ability to change or reset import statuses.)
• Logs - Migration logs are listed for each import, allowing admins to quickly see if there are any errors. You can quickly filter logs by message type (i.e. “notice” or “error”).

My Second Use Case: Importing Courses for a University

Soon after wrapping up a working prototype for the data visualization project I mentioned above, I was tasked with another project. A prominent university client needed to quickly import an entire course catalog into Drupal. Beyond running a single import, this particular organization needed the ability to upload new CSVs and update the catalog at any point in the future. The use case was a perfect match for Entity Import. I installed the module, spent a few minutes adding and configuring the course importer, and presto!

Next Steps for Entity Import

Writing clean, reusable code packaged as modules is a huge benefit for Drupal development workflows. Even better, Drupal.org module pages provide really great tools for collaboration with features like issue tracking, release management, and version control built into the interface. I have a few TODOs that I’ll be posting as issues in the days ahead, and I am excited to see if Entity Import fills a need for others like it has for me.

If you run a data or content -intensive website and have trouble keeping everything up to date, Entity Import might be just the ticket. We’d love to give you a quick demo or talk through how this approach might help – just give us a shout and we’ll follow up!