Introduction
In August 2015, Skrift gave me the honour of writing an article for Issue No.5. Since then, a huge amount has changed in the way we structure projects and the tools we use so I thought it was time that I rewrote the article. In the intervening 30 months we've had 5 major releases of Umbraco, io.js was merged back into Node.js (thank goodness), there were at least 7 major releases of Node.js, Bower went the way of the Dodo and Umbraco HQ has grown many times over.
Having re-read my issue 5 article I was happy to feel that things had moved forward in a positive way across the board; the tools have become fewer and they do more, which is a good thing. We try to not make changes to our process too quickly once we have something that works well but, at the same time, we don't want our skills to be left behind everyone else so finding a balance is key.
As I wrote 30 months ago, there is no right or wrong way to structure Umbraco projects - it's about finding what works for you, and so here we go: the low-down of the development tools we use and how we structure our solutions at Crumpled Dog in 2018 and also what we are intending to change in the next 12 months.
The Prerequisites that have to be installed
A great improvement in the last 30 months is that the number of prerequisites that we have to install on a developer's machine has gone down from 6 to 2! No longer do we have to have Git, Ruby, Sass or Bower installed on our developer machines or build servers.
NodeJs
NodeJs is now the core of all things relating to front end development.
Grab the download, I recommend you use the LTS (long term support) version, and run the installer. Once it's completed check it worked using the command line:
node –v
If successful you should see something like this:
GulpJs v4
We now use Gulp as our task runner we switched from using Grunt to Gulp about 18 months ago. While Grunt is still a good task running development on the project came to a halt in summer 2017.
The Gulp team have been working on version 4 for well over a year prior to it's full release in January 2018. Many developers have been working with the various pre-releases over that period so it was very well fed when it was eventually released. If you want to found out more information, be sure to check the release notes https://github.com/gulpjs/gulp/releases/tag/v4.0.0
Gulp has a requirement for the gulp-cli and this should be installed as global Node Module on a developer machine or build server. This is very straight-forward in command prompt (run as Administrator):
npm install --global gulp-cli
If you want to check the version of the Gulp CLI you have:
gulp -v
Visual Studio 2017
https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx
To get the most out of this project setup you will need Visual Studio. If you don’t have it already you can download the free & fully featured Community Edition.
Umbraco Models Builder Custom Tool - Visual Studio Plugin
All of our projects are using the Umbraco Models builder and we prefer to use the custom tool to generate the models classes into our ".Common" project rather than a DLL. This tool uses the Models Builder API to gather the required data to generate the code that your can then compile.
https://marketplace.visualstudio.com/items?itemName=ZpqrtBnk.UmbracoModelsBuilderCustomTool
SlowCheetah Visual Studio Plugin
We use SlowCheetah for transforming configuration files such as umbracoSettings.config when projects are published via WebDeploy. It's actually not required that the Visual Studio Plugin is installed as SlowCheetah is a primarily a NuGet package however, installing it brings you the ability to add transforms to config files and awesomely (well, I think so) preview the transforms in Visual Studio.
The Solution Structure
In Visual Studio we start with two projects in our solution, a ".Site" project and a ".Common" project. The ".Common" project is a Class Library project and the ".Site" project is a Web Application. For more complex projects we may add further projects but this is our starting point. We also have a collection of scripts in our build folder that are used for our CI/CD.
.Site project
The project is used to install Umbraco, Umbraco Packages, Views & all front end project source assets. We do not keep any C# code in this project.
.Common project
In this project we keep all of our C# code. We also use the Umbraco Models Builder tool to generate our Models classes into this project (more on that later). The .Site project has reference to this project so it's DLL is copied to the bin folder on build.
Sass Files
Generally, our front end gurus are using Sublime Text for authoring Sass (a few have dipped their toes into VS Code) so we keep our .scss files in the css folder with only "main.scss" in the root.
package.json
What we have lost in installable dependencies we have gained as NodeJs modules. Our package.json has grown a fair bit in the last 30 months. Below is a bare minimum version that can be used to get started.
A key change is that we now have devDependencies (the tools used to build) and dependencies (the vendor packages we included in our website) both defined within the package.json. We use WebPack (more on that below) to copy the dependencies into our .Site project.
{
"name": "buildingblocks2018",
"description": "All NodeJs stuff needed to be cool",
"repository": "https://bitbucket.org/crumpleddog/buildingblocks2018",
"license": "UNLICENSED",
"author": "Crumpled Dog",
"version": "0.0.1",
"devDependencies": {
"babel-core": "^6.26.0",
"babel-loader": "^7.1.2",
"babel-preset-env": "^1.6.1",
"copy-webpack-plugin": "^4.3.1",
"del": "^2.2.0",
"es6-promise": "^3.1.2",
"gulp": "^4.0.0",
"gulp-autoprefixer": "^4.1.0",
"gulp-banner": "^0.1.3",
"gulp-notify": "^3.0.0",
"gulp-plumber": "^1.1.0",
"gulp-sass": "^3.1.0",
"gulp-scss-lint": "^0.4.0",
"gulp-sourcemaps": "^2.6.3",
"jshint": "^2.9.4",
"merge-stream": "^1.0.1",
"webpack": "^3.10.0",
"webpack-stream": "^4.0.0"
},
"dependencies": {
"html5shiv": "^3.7.3",
"jquery": "^3.2.1",
"jquery-validation": "^1.17.0",
"jquery-validation-unobtrusive": "^3.2.6",
"responsive-bp": "^4.1.4",
"handlebars": "^4.0.11",
"jquery-unveil": "1.3.2",
"font-awesome": "^4.7.0",
"picturefill": "^3.0.2",
"lazysizes": "^4.0.1"
}
}
More info on package.json can be found on the following link https://docs.npmjs.com/files/package.json
webpack.config.js
We are currently only using a small fraction of what WebPack can do. We have used it to replace the now deprecated Bower package manager by using it to copy our third party vendor assets into the .Site project from their node_modules location, we use the copy-webpack-plugin for this. We also use WebPack to transcode ES6/7 JS using Babel.
Here's how a basic config starts out:
const path = require('path');
const CopyWebpackPlugin = require('copy-webpack-plugin');
const styleAssets = require('./scripts/assets/style-assets');
const scriptAssets = require('./scripts/assets/script-assets');
const fontAssets = require('./scripts/assets/font-assets');
module.exports = {
entry: {
app: "./scripts/src/index.js"
},
output: {
path: __dirname + "/scripts",
filename: "buildingblocks.bundle.js"
},
plugins: [
new CopyWebpackPlugin(
styleAssets.map(asset => {
return {
from: path.resolve(__dirname, `./node_modules/${asset}`),
to: path.resolve(__dirname, './css/vendor/')
};
})
),
new CopyWebpackPlugin(
scriptAssets.map(asset => {
return {
from: path.resolve(__dirname, `./node_modules/${asset}`),
to: path.resolve(__dirname, './scripts/vendor/')
};
})
),
new CopyWebpackPlugin(
fontAssets.map(asset => {
return {
from: path.resolve(__dirname, `./node_modules/${asset}`),
to: path.resolve(__dirname, './css/fonts/')
};
})
)
],
module: {
loaders: [
{
test: /\.js$/,
loader: 'babel-loader',
query: {
presets: ['env']
}
}
]
}
};
We then have separate files to define which style, script and font assets to copy.
Here is how script-assets.js looks:
const JS = [
'responsive-bp/build/responsive.min.js',
'jquery/dist/jquery.min.js/',
'jquery-validation/dist/jquery.validate.min.js',
'html5shiv/dist/html5shiv.min.js',
'jquery-validation-unobtrusive/jquery.validate.unobtrusive.js',
'handlebars/dist/handlebars.min.js',
'jquery-unveil/jquery.unveil.js',
'picturefill/dist/picturefill.min.js',
'lazysizes/lazysizes.min.js'
];
module.exports = [...JS];
More info on webpack.config.js can be found at https://webpack.js.org/configuration/
gulpfile.js
In the starter gulpfile.js below we have 5 tasks:
- js - executes WebPack to get returned a js file that can be referenced in the website
- sass - compiles Sass and returns css. We also add a handy banner into the top of the css file so we know what version of css is being used. The data for this banner comes from the values in package.json.
- sass-source - this task also compiles Sass to css but it also adds sourcemaps inline to the css file. This makes it easy to debug in the browser as the inspector will reference the .sass files.
- dev - this task executes the js task then the sass-source task. This is the task used when developing the website locally.
- release - this task executes the js task then the sass task. This is used by the build server.
I've tried to keep this example to the minimum that provides the core functionality, but there is of course a whole universe of extra processes and tasks that you can add.
var gulp = require('gulp');
var sass = require('gulp-sass'),
sourcemaps = require('gulp-sourcemaps'),
plumber = require('gulp-plumber'),
banner = require('gulp-banner'),
autoprefixer = require('gulp-autoprefixer'),
notify = require('gulp-notify'),
pkg = require('./package.json'),
webpack = require('webpack'),
webpackStream = require('webpack-stream'),
webpackConfig = require('./webpack.config.js');
var config = {
dest: './',
name: pkg.name
};
var comment =
'/*\n' +
' * <%= pkg.name %> <%= pkg.version %>\n' +
' * Updated: <%= new Date().toString() %>\n' +
' *\n' +
' * Copyright <%= new Date().getFullYear() %>, <%= pkg.author %>\n' +
'*/\n\n';
gulp.task('sass', function () {
return gulp.src(config.dest + 'css/main.scss')
.pipe(sass({ sourceComments: 'normal' }))
.pipe(autoprefixer({
browsers: ['last 2 versions'],
cascade: false
}))
.pipe(banner(comment, {
pkg: pkg
}))
.pipe(gulp.dest(config.dest + './css'))
.pipe(notify({ message: 'Sass task complete' }));
});
gulp.task('sass-source', function () {
return gulp.src(config.dest + 'css/main.scss')
.pipe(plumber())
.pipe(sourcemaps.init())
.pipe(sass.sync()) // Have to use sass.sync - See Issue (https://github.com/dlmanning/gulp-sass/issues/90)
.pipe(sourcemaps.write())
.pipe(banner(comment, {
pkg: pkg
}))
.pipe(gulp.dest(config.dest + './css'))
.pipe(notify({ message: 'Sass source task complete' }));
});
gulp.task('js', function () {
return gulp.src('./scripts/src/index.js')
.pipe(webpackStream(webpackConfig), webpack)
.pipe(gulp.dest('./scripts'));
});
gulp.task('dev', gulp.series(['js', 'sass-source']));
gulp.task('release', gulp.series(['js', 'sass']));
Visual Task Runner
To easily integrate the front end build process into our Visual Studio projects we us the Task Runner Explorer. 30 months ago this was a Visual Studio Plugin but it was so useful that Microsoft built it into Visual Studio 2017! With this tool you can execute Gulp tasks directly in the IDE or you can add bindings to certain events such as "After Build". This is what we do. So when the Web Application project is built a certain Gulp task will be executed, we generally have the "dev" Gulp task as the one that is executed.
.editorconfig
Another cool thing that was built into Visual Studio 2017 was the EditorConfig plugin. Now all you need to do is to add a .editorconfig file to the root of your project and all developers will be obeying the same rules for things such as white space.
These rules will also be obeyed by other IDE's such as Sublime as long as they also have the EditorConfig plugin installed.
Here's how ours looks (yes, we like tabs for indentation):
# top-most EditorConfig file
root = true
# Tab indentation
[*]
indent_style = tab
indent_size = 4
trim_trailing_whitespace = true
[*.cshtml]
charset = utf-8-bom
indent_style = tab
indent_size = 4
trim_trailing_whitespace = true
More info on EditorConfig can be found at http://editorconfig.org/
Compression, minification and bundling of assets
We still use the Optimus package for Umbraco which gives us a UI layer for the .Net Optimisation framework for our bundling https://our.umbraco.org/projects/developer-tools/optimus.
CI/CD (continuous integration/continuous deployment) automation
If you would like a good introduction to CI/CD I recommend this blog post as a good starting point.
We use AppVeyor for both CI and CD. As you can see from the Solution Structure image we have a collection of build scripts that do different things. If you would like to know more about these you can watch the video of my session at the Umbraco Deutschland 2017 festival where I go into detail about each of them https://www.youtube.com/watch?v=oiPmRJlkL1Q
What about LightInject, Test projects, custom file systems etc....?
Yes, we do all of those things too but I'm outlining the underlying foundation of how we structure all our projects and the fundamental tools used in our workflow. Besides, I can't write down everything or you would never get to the end of this article.
What Next?
Our plans for the next 6 to 12 months are:
- Switch from NPM to Yarn - if you want to know why, have a read of https://www.sitepoint.com/yarn-vs-npm/ There also a Visual Studio Plugin that allows VS to use Yarn instead of NPM https://marketplace.visualstudio.com/items?itemName=MadsKristensen.YarnInstaller
- Convert our gulpfile.js to gulpfile.babel.js - our gulpfile.js written in ES6, we very recently completed this but it's still too fresh for me to write up in this article.
- We may decide to use only WebPack and drop Gulp. WebPack can do almost everything we currently do in Gulp so we may eventually remove the dependency. There is already a WebPack task runner Plugin for Visual Studio and I wouldn't be surprised if it was included in a future release of VS. https://marketplace.visualstudio.com/items?itemName=MadsKristensen.WebPackTaskRunner
- We may use Gulp/WebPack to compress, minify and bundle our assets instead of leaving it to .Net. This practice is also changing with the wider adoption of HTTP/2 so we may be going back to multiple files.
- Some other as yet undefined stuff. We are in a brave, fast-moving world; we are always learning and thinking about usefulness.
Summing up
The technology we use to develop solutions in our daily developer lives now changes at a rate that can at times feel uncomfortable and uncontrollable for many developers. At the same time as being scary, the level and pace of innovation that's materialising can be inspiring and exciting. This is what we should focus on. I enjoy trying to keep up, even if I'm never fast enough, but by attempting to keep up I pick and choose the tools and processes I think will improve the quality of our output and our overall productivity.
I hope this article will be inspiring for some and become useful in their daily developer lives. Personally, I'm looking forward to seeing where we are in another 30 months!
Thanks
Many thanks to my colleagues William Phillips and Jack Stunell who between them manage to figure all of these wonderful crazy tools and techniques into something we can use.