Umbraco Migrations Made Easy by Benjamin Carleski

The Challenge

Recently I started a new job with ProWorks, an Umbraco Gold-Partner specializing in installing, upgrading, and maintaining Umbraco instances. I got put onto a project to upgrade a new client from Umbraco version 6.2.6 to version 7.7.7. One of the biggest challenges we faced was migrating all the data from the ID-based pickers of v6 to the GUID-based pickers of v7. With over 20,000 nodes, each containing 5-50 different pickers, re-selecting all the data was just not a feasible option.

One of my colleges, Mark Bowser, suggested taking a look at an Umbraco technology called Migrations. Looking into it, I immediately saw two things. First, this was exactly the technology I had been looking for. The ability to easily write scripts that were versioned and conditionally applied was exactly what was needed for this project and for any other project where I needed to move, merge, split, or otherwise alter data. Second, I realized the process to detect migrations, find what needed to be applied, and then kick-off applying it was going to be just a chunk of repetitive code. For every project I would want to use this in, I would need to copy in a bunch of code and reconfigure it all to the project.

A Better Way

Not really liking that second realization, I set out to write my own initialization code. I wanted something independent of any project, that I could package up on NuGet and just include, with a few configuration lines, into any project. Hence the Our.Umbraco.Migration package was born. As I used it in the project, I developed a few common base classes to help with the migrations. There are resolvers for extensibility, and common migration helpers to facilitate data alterations. The configuration is a few lines at the bottom of the web.config file. Let me show you how to use this.

NOTE: I'm assuming the reader is familiar with the Umbraco Migration framework. If you need a primer, or a refresher, I suggest stopping for a moment and reading this excellent article on Cultiv or this one here on Skrift. There is also the official Umbraco API Documentation, though these classes don't have much in the way of comments or examples.

ACME Toys Upgrade

In our example, Acme Toys is upgrading their main content site from Umbraco 6 to 7. They have two main problems with this upgrade. Their htmlContent document type has an old media picker property called "banner" that they want switched to the newer v7 media picker. The old media picker stored data as node IDs, which the new picker uses UDIs. Additionally, their homePage type has a set of 40 properties representing featured items to display on the home page. They are a repeating set of 4 properties ending in a number from 1-10. They want these combined into a single nested content property called featuredItems. You quickly make the document changes, but now the task of moving the data is at hand. With 15 home pages for different languages, and hundreds of content pages, migrating the data seems like a daunting task. Where to start?

To give you an idea of what this looks like, here's some screenshots of the back-office after the document type changes, but before any migrations are run. You can see that the home page is a mess, and the image on the content page isn't showing anything, even though there is a value picked in the database.

Configuration

The first thing we do is NuGet in the Our.Umbraco.Migration package. This adds a DLL reference to the project and some XML to the bottom of the web.config file. Taking a look at that migrationResolvers block and we can see that it is looking for a project name. This name is the first argument to the MigrationAttribute meta-data that has to be on any Umbraco Migration class. It can be any string, and you can have as many different project names in a single code base as you like. The web.config parameter is just a comma-separated list of project names. In our case, we are just going to use Acme.Toys.

<configuration>
  <migrationResolvers>
    <add name="ProductMigrations" type="Our.Umbraco.Migration.ProductMigrationResolver, Our.Umbraco.Migration">
      <add key="MonitoredProductNames" value="Acme.Toys" />
    </add>
  </migrationResolvers>
</configuration>

ID to UDI Migration

The next thing we want to create is two classes. One to do the ID-to-GUID migration, and a second to do the Nested Content migration. We could do these with a single class, but doing them separately lets us use the IdToUdiMigration base class, and not have to recode it ourselves. First, let's add the V1_0_0_HtmlContentIdMigration class to our project with the following code.

[Migration("1.0.0", 1, "Acme.Toys")]
public class V1_0_0_HtmlContentIdMigration : IdToUdiMigration
{
    private static readonly IDictionary<string, IDictionary<string, ContentBaseType>> Fields =
    new Dictionary<string, IDictionary<string, ContentBaseType>>
    {
        ["htmlContent"] = new Dictionary<string, ContentBaseType>
        {
            ["banner"] = ContentBaseType.Document
        }
    };

    public V1_0_0_HtmlContentIdMigration(ISqlSyntaxProvider sqlSyntax, ILogger logger)
        : base(Fields, sqlSyntax, logger)
    {}
}

Taking a look at this, you can see that we are using the constructor of the IdToUdiMigration class to pass in the document type (htmlContent) and the property name (banner) of the data we want to convert, as well as what type of content this data refers to. The passed in dictionary only contains a single reference in this example, but could touch as many properties as you like. In the production project I mentioned earlier, we passed in an dictionary with over 30 document types and over 100 properties! You can also see in the MigrationAttribute at the top where I reference the version number (1.0.0), the step sequence within the version (1), and the project name (Acme.Toys) which must match exactly what you put into the web.config file.

Content Consolidation

The next thing we need to do is migrate the featuredItems content. As this is a very custom migration, we are going to simply use the Umbraco base class, to give us more control over what exactly happens.

[Migration("1.0.1", 1, "Acme.Toys")]
public class V1_0_1_FeaturedItemsMigration : MigrationBase
{
    public V1_0_1_FeaturedItemsMigration(ISqlSyntaxProvider sqlSyntax, ILogger logger)
        : base(sqlSyntax, logger)
    {}

    public override void Up()
    {
        var cts = ApplicationContext.Current.Services.ContentTypeService;
        var cs = ApplicationContext.Current.Services.ContentService;
        var docType = cts.GetContentType("homePage");
        var contents = cs.GetContentOfContentType(docType.Id) ?? new IContent[0];

        foreach (var content in contents)
        {
            UpgradeContent(cs, content);
        }
    }

    private void UpgradeContent(IContentService cs, IContent content)
    {
        var nestedContent = new List<Dictionary<string, object>>(20);
        var published = content.Published;

        for (var i = 0; i < 20; i++)
        {
            nestedContent.Add(new Dictionary<string, object>(4)
            {
                ["title"] = content.GetPropertyValue<string>($"featuredItemTitle{i}"),
                ["summary"] = content.GetPropertyValue<string>($"featuredItemSummary{i}"),
                ["author"] = content.GetPropertyValue<string>($"featuredItemAuthor{i}"),
                ["date"] = content.GetPropertyValue<string>($"featuredItemDate{i}")
            });
        }

        var newValue = JsonConvert.SerializeObject(nestedContent);
        content.SetValue("featuredItems", newValue);

        if (published) cs.SaveAndPublishWithStatus(content)
        else cs.Save(content);
    }

    public override void Down()
    {
    }
}

You can see a few things going on here. First, for simplicity's sake we have not implemented a Down method, or any real error handling. In this case, a down method also might not make much sense anyway since the original properties will remain intact after the migration. You can also see that we are manually constructing the JSON-style content used by the Nested Content picker. The documentation, and a bit of Google and trial and error, were helpful in getting the syntax just right here.

Summary

With our web.config updated, and our two new classes added, we are ready to roll! The next time Umbraco starts with our new code in place, the Our.Umbraco.Migration code will search the code base for any IMigration classes that have the MigrationAttribute with Acme.Toys as the project name. It will find our two, and will then compare the highest version (1.0.1) with the version recorded in the database (none applied yet, so 0.0.0). Since there are versions to be applied at a higher version than what is applied, it will run through all of the unapplied migrations in order, applying each one. This ensures that we have a consistent process every time.

Here's some screenshots of the back-office after our migrations have completed. The featured items on the home page are much easier to work with and maintain, and the content page's image is showing correctly again.

A Few Parting Shots

For these examples fleshed out with comments in an actual solution, check out the Migration.Demo project. The project is a fully functional Umbraco instance that you can spin up and play with on your own to quickly see migrations, and the Our.Umbraco.Migration package, in action.

Also, in our production project, we use uSync in a manual mode to apply document and data type changes to the website. To ensure these changes had been applied correctly, we first commented out the web.config section, started the site, applied the uSync changes, and only then uncommented the web.config section and restarted the site. This ensured that the document type changes were in place to support the data migration. We have not yet tested this with Umbraco Deploy to see if this two-step process will still be needed going forward.

Finally, we noticed that Umbraco 7.7.7 seems to have a problem accessing the ContentService during the migration process. When using it, the migration version doesn't get correctly recorded and an error shows up in the log. We added code to the end of our Up method to manually insert the version into the umbracoMigrations table, but we have not tested this yet at other version to see if this bug is in something we are doing, or a bug in the Migration framework. If you try it and have any experiences with this, let us know.

Feedback and Follow-up

What kinds of data migrations do you do? Are there common base classes or migrations you would find helpful in a general-purpose data migration package? Add a comment and let me know what your experiences are!

Umbraco Migrations Made Easy