Introducing Blend's Little Import Tool

Content migration is difficult — really difficult.

On the surface, the process seems simple enough. It typically breaks down into three steps:

• Extract — Migrating the content out of the current system.
• Transform — Manipulating the content into a shape the new system can understand.
• Load — Place the manipulated content into the destination system.

In real life — at least, for Optimizely projects — this frequently amounts to a one-off scheduled job that pulls HTML from one site, grabs and adjusts pieces of the markup, and uses the internal content APIs to create new content.

This works, but it can also have some issues:

• The scheduled job code must exist in the destination environment. Seems obvious — but it also means any mistakes or changes to the import process require a deployment to fix.
• If the job isn’t carefully written to be idempotent — which means it can produce the same outcome, even if the same file, event, or message is received more than once — Bad Things™ can happen. I’m looking at you, “dozens of duplicate pages appearing after several failed import attempts.”
• One-off scheduled jobs are just messy. What happens after the one-time import is complete? Do you remember to delete the now useless scheduled job? Or do you hope no one gets curious and clicks that “Start Job” button?
• Imported content that references other imported content is complicated. Say page “A” references page “B.” You’ll have to import page “B” first if you want to reference the new URL for page “A.” Not so bad, but what if both pages reference each other? Now you’ve got a chicken-or-the-egg paradox.

We’ve been working on an experimental library to address some of these concerns.

What is BLIT (Blend's Little Import Tool)?

Blend’s Little Import Tool (or BLIT as we’ve code-named it) is our utility for improving the import process. It focuses on the “Load” step; it leaves the “Extract” and “Transform” steps up to you.

BLIT requires content to be put into a specific structure that the tool understands. That structure is the secret sauce that enables a lot of features:

• The format has a query syntax that allows content to be found. This is key for a couple of reasons:
  • Content can “find itself” in the system, meaning if that content already exists, it can be updated, otherwise it can be created, making imports idempotent.
  • Content can find its parent node. This means you don’t have to hard-code IDs. Imported content can even be the child of other imported content, as long as the parent node is first in the import process.
• Properties are imported in two distinct stages:
  • The Initial Stage handles creating/publishing all content nodes and fills in any properties that (you have determined) have no dependencies on other content.
  • The Deferred Stage then fills in any properties that might refer to other imported content, such as content references, XHtmlString links, etc.
  • This solves the chicken-or-the-egg paradox by ensuring all imported content exists before attempting to resolve links between those pieces of content.
• The format has a fragmented text format that allows you to mix HTML, variables, and references to other content, either as queries for existing content, or new content directly embedded in the fragment stream. For example, you can embed image data directly into your XhtmlString property, and it will be imported into the “For this page” container and referenced correctly when imported.

Those files are then consumed by our import service, which runs on the site and uses the Optimizely CMS APIs to import the stored content data into the CMS. As mentioned, the service is idempotent and makes no changes to the published content if no changes are detected.

Usage Scenarios

While we primarily wrote this with content migration in mind, it can be valuable in any scenario where importing content is useful.

For example, do you have test content in your development environment that you don’t want to be overwritten when you bring the production database back? No worries: instead of creating pages through the CMS, you can create BLIT files and import them quite easily in any and all environments where you may need them.

Sometimes we don’t want to do a full content migration, but building out the site tree is helpful. BLIT is an excellent choice for automating that. It will create any missing pages without overwriting the properties that aren’t included in your files, allowing you to safely automate page tree creation with little risk of destroying existing content.

Optimizely's Built-in Import/Export

I hear you. “But Optimizely already has an export/import functionality! Why bother with this?” The primary difference here is that the Optimizely import is built around matching content GUIDs. In contrast, our import format is built around content being able to “query for itself,” meaning less likelihood of hard-coded GUIDs or content IDs. We’re targeting a very different use case.

The Future

We’ve been using BLIT in some form for about a year and have had great success with it, and we’d like to share it with the world soon. We’re currently in a small closed beta, gathering feedback and bug reports. We have some updates and bugs to squash, but we hope to introduce the world to this as an open-source add-on for Optimizely CMS.