At ProWorks we believe Umbraco Cloud should be the first choice for most new Umbraco sites and we're big fans. We're also big fans of automation for team development. For nearly every client we work with we have a team workflow configured that allows our team to work with very little friction by using a (more or less) standard git flow approach for our branching strategy and build and deploy services to keep our development sites up to date with the latest commits to the development branch. This is a process that the team at ProWorks has refined over the years and that all team members are familiar with. Using this well-defined and well-structured process gives our team a productivity boost and removes the "human error" element from merging, building, and deploying.
As with many busy agencies, the proworks.com website has suffered from a lack of attention due to the team being focused primarily on client work, so we decided it was time to give it a makeover (now in progress). At the same time we are taking the opportunity to move the site from a legacy host to Umbraco Cloud. We also decided this was the right time to validate the use of our existing team development workflow with an Umbraco Cloud site. And there we begin the rest of this story.
For the updated proworks.com site we naturally choose Umbraco Cloud configured with a development and a live site for our project. I asked Ryan (our DevOps and IT engineer) to join me in setting up a team development workflow. One of the goals with our initial setup was to use the existing toolset that ProWorks has and that the team is familiar with; BitBucket git repositories, TeamCity build server, and OctopusDeploy deployment services. Taking inspiration from Morten's innovative deployment for Umbraco Cloud with a Visual Studio Team Services task work we setup a basic structure using our current tools. As you'll see, our workflow does not exactly match the VSTS workflow - but massive kudos to Morten for the inspiration.
Our goal is to work with Umbraco Cloud sites using a team development workflow that is as similar to what we currently use as possible. For us at ProWorks that means:
- We develop locally using a feature branch from a fork of the development branch of the main repository
- Our local development setup varies somewhat depending on client, role, and personal preference but generally is:
- VisualStudio, IIS Express, LocalDb or local SqlServer Express
- Once a feature is complete or ready for testing, the forked feature branch is merged into the main development branch via a pull request
- TeamCity watches the main repository's development branch and triggers a build when new commits are merged in
- Once TeamCity has built the project, it is packaged into a NuGet package and sent to OctopusDeploy
- OctopusDeploy registers a new package and deploys the build artifacts to the target - a development instance or, in this case, the development site of our Umbraco Cloud project
We'll omit some of the details of our complete Umbraco Cloud project configuration for the sake of brevity. However, note that we follow the prescribed best-practices for Umbraco Cloud in terms of configuration and the use of config transforms to correctly set configuration values for local, development, and live sites.
The Workflow
When first starting to work with the solution a developer will fork the master branch, then clone the .Core ProWorks repository from Bitbucket. In the root of the solution files we've placed a script that will clone the Umbraco Cloud site's git repository into the correct location. So the developer runs that:
cd Proworks.Web
git init
git remote add origin https://dev-great-site.scm.s1.umbraco.io/b2b363b5-f804-4528-b636-e63b53ba9fdb.git
git fetch
git reset origin/master --hard
git checkout -t origin/master
With this in place the developer can create a working (feature) branch, add her updates, and then create a Pull Request for the original .Core repository targeting the master branch. Once that is merged into the repository the TeamCity build process is notified.
TeamCity pulls the latest commit from the master branch and executes a build using the build definition. If this is successful TeamCity notifies OctopusDeploy that a new build is ready.
OctopusDeploy downloads the build output (the nuget package) and runs the deploy process which consists of, 1) extracting the nuget package, 2) cloning the Umbraco Cloud development site repository, 3) applying and committing all changes to the repository, and 4) pushing the changes to the Umbraco Cloud repository which initiates the Umbraco Cloud (Kudu) deployment.
Once the push from OctopusDeploy is complete, the Umbraco Cloud deployment starts which applies any schema updates, runs config transformations, and updates files - among other things.
Once this is complete, the local Umbraco Cloud repository needs to be updated for local development. To do that the developer runs a script from the root of the .Core repository:
cd Proworks.Web
git pull origin master
cd data
echo > deploy
The last line in that script sets a deploy marker file in the local development site. When that sites runs next time the development marker triggers the local instance of Umbraco Deploy to start a deployment and process any pending updates to the Umbraco schema. This isn't strictly necessary but we've found it a useful addition as any pending updates are always applied.
Visual Studio Structure
Our Visual Studio solution setup is very similar to the recommended structure accoring to the official Umbraco Cloud documentation with a few key differences. The solution is structured with two main projects (it's a simple site and, of course, the actual number of projects is arbitrary so you may have more projects in your solution):
- ProWorks.Core (all custom code including Views, JS, SASS, Models)
- ProWorks.Web (a clone of the Umbraco Cloud website with no custom code)
It's important to note that we never commit to the .Web repository. This allows us to take advantage of the autoupgrade feature from Umbraco Cloud. We consider the Proworks.Web project a black box and always git pull the latest from the Umbraco Cloud development site. On that same note, we never make updates directly to the development site running on Umbraco Cloud. This part of the workflow does require some discipline from our team, but it follows how we work with other projects so it is familiar.
Keeping localhost Up to Date
In our solution we decided to keep it simple and use post-build events in Visual Studio to copy our custom code from the .Core project to the .Web project when doing local development. This is not a requirement for the build server, but only used locally:
XCOPY /s /y /i "$(TargetDir)$(ProjectName).*" "$(ProjectDir)..\$(SolutionName).Web\bin"
XCOPY /s /y /i "$(ProjectDir)Views" "$(ProjectDir)..\$(SolutionName).Web\Views"
XCOPY /y /i "$(ProjectDir)scripts" "$(ProjectDir)..\$(SolutionName).Web\scripts"
XCOPY /y /i "$(ProjectDir)Css" "$(ProjectDir)..\$(SolutionName).Web\Css"
XCOPY /s /y /i "$(ProjectDir)App_Plugins" "$(ProjectDir)..\$(SolutionName).Web\App_Plugins"
XCOPY /s /y /i "$(ProjectDir)..\$(SolutionName).Web\data\revision" "$(ProjectDir)data\revision"
XCOPY /s /y /i "$(ProjectDir)..\$(SolutionName).Web\data\backoffice\users" "$(ProjectDir)data\backoffice\users"
You may have noticed references to the \data\ folder. This contains the .uda files generated by Umbraco Deploy that are used by Umbraco Cloud to deploy the site's schema (document types, data types, users, etc…). The files here are included in the git repository for the .Core project. It's a bit counter-intuitive but read-on and we'll explain.
Models
We believe the best way to work with ModelsBuilder models with this configuration is to use the ModelsBuilder API and the corresponding ModelsBuilder Visual Studio custom tool. However, doing so requires each developer to correctly configure this as the dependency cannot be defined in the Visual Studio solution - as far as we know. Our current config requires the building of Models from the local Umbraco Backoffice using theAppData models mode. We then include the generated models in the .Core project and they are deployed to the Umbraco Cloud development site included in the Proworks.Core.dll.
What to Ignore
Since we do not commit any of the local changes to the .Web project back to the remote Umbraco Cloud repository we need to include the .uda files from the .Web project's \data\ folder in the .Core project's repository. We do this by an addition to the .gitignore file for the .Core project. We've discovered there is a bit of an art to getting a .gitignore file to apply to exactly what you intend. Our version of this is:
# Exclude .Web which is synced with the Umbraco Cloud site
src/Proworks.Web/*
# add the .uda files for umbraco structure, data, and users
!src/Proworks.Web/data
src/Proworks.Web/data/*
!src/Proworks.Web/data/revision
src/Proworks.Web/data/backoffice/users
!src/Proworks.Web/data/backoffice/users/*
So all of the .Web project is ignored except for .uda files in the \data\ folder.
Build Definition
We include build assets just as with our other projects. This includes a.nuspec file and agulpfile.js used to minify assets. One important note is that thegulpfile.js is included in both projects while the.nuspec file is only added to the .Core project where we specify the target directory of the build output within the nuget package.
Our TeamCity instance uses a custom meta-runner for most of our projects. The meta-runner builds the project according to the Visual Studio solution (sln) file or according to a.nuspec file if present, runs Octopack to create a Nuget package file, uploads the resulting Nuget file to a defined target in our OctopusDeploy instance, then triggers a new release in OctopusDeploy which runs the deployment. For our Umbraco Cloud workflow we rely on the.nuspec definition to define which files are included in the nuget package output as we do not directly build the Umbraco web site project. Similar to the following nuspec file snippet:
<files>
<file src="data\revision\**" target="data\revision" />
<file src="data\backoffice\users\**" target="data\backoffice\users" />
<file src="App_Plugins\**" target="App_Plugins" />
<file src="Css\**" target="Css" />
<file src="Views\**" target="Views" />
…
The above snippet from the.nuspec file is needed in order to define, for the Octopack nuget builder, the output locations for the build artifacts. This is required since we do not build the .Web project, only the .Core project which does not itself contain a definition for build output that matches the target Umbraco Cloud site. This part of the build process is the least intuitive, at least it was for us initially.
Umbraco Cloud specifics
One hurdle we encountered was our use of TeamCity to package the build output as a nuget package, using the OctoPack add-in. While there are many advantages to having a nuget package as the build output, obvious deployment to an Umbraco Cloud site is not one of them. To work with this build output format we added a step to our OctopusDeploy process that unpacks the nuget package. Obviously, if your build does not result in a nuget package output then you won't need this step. We created a custom step template via a Powershell script to accomplish this task. You'll find the Powershell script and details on configuring the steps in our shared repository https://bitbucket.org/proworks/clouddeploy/
This step has an output parameter defined [ExtractedFileLocation] which passes the location of the extracted nuget package to the next custom step.
The next step in the deploy process git clones the remote Umbraco Cloud repository, then copies the extracted files into the location where the repository was cloned, commits these changes, then pushes the updates back to the remote Umbraco Cloud repository. All of that follows the manual deployment workflow for Umbraco Cloud, just using an automated OctopusDeploy step to accomplish the result.
Upgrades
As we take advantage of the auto upgrade feature from Umbraco Cloud for the .Web project this means that we need to manually upgrade the .Core project to match the Umbraco version of the .Web project. This is done easily enough by adding the UmbracoCms.Core nuget package reference to the .Core project and simply using the nuget Update-Package command as needed.
Summary
Overall we achieved our goal of using the existing ProWorks team development workflow and tools with Umbraco Cloud to create an automated build and deployment workflow. Even though our goal was to work "almost exactly" the same way with Umbraco Cloud as with a legacy hosted site, there are necessarily some differences in the workflow. Namely these differences are the hands-off approach to the .Web project and the additional build definition required to create the build artifacts correctly from the .Core project.
Things we may do differently
We've already made some adjustments to the original configuration and we're certain to make others as we continue to use the workflow daily. One change we'll make soon is to use the development branch as we do for non-Umbraco Cloud projects while still deploying to the master branch on the remote Umbraco Cloud development repository. We'd love to see support for deploying a branch other than master to Umbraco Cloud, but this is a relatively minor item in the overall workflow.
Another item we may factor out of the workflow is the use of the nuget package to contain the build artifacts. While it is convenient from a tools and versioning perspective, handling the nuget package format adds some amount of overhead to the deployment step in the workflow. On our wish list is support for nuget package deployment in Umbraco Cloud - we don't know if this is possible but it would make deployments very sweet.
Finally, as this was our first go at automating an Umbraco Cloud team development workflow using a, self-inflicted, requirement of the current ProWorks tools, we also plan to create the same workflow functionality using VSTS (building on Morten's excellent task) and, based on popular demand, AppVeyor. We plan to share the outcome of these exercises via future Skrift.io articles so please let us know your experiences.
Source Code
We have created a public repository with the code mentioned here. Please feel free to use and send us improvements via pull requests. https://bitbucket.org/proworks/clouddeploy