Having worked on a number of agile as an Umbraco back office developer, using scrum I wanted to share a few thoughts I have had about agile as a development methodology based on my experiences, where it goes well and how to set up a project so that it is less likely to fail.
Adopting an agile approach impacts all areas of the project from procurement through to support, I will explore the impact in the following sections. I make no apologies for writing an article that seems a bit project managementesque in its approach, in an agile team every team member is responsible for ensuring successful delivery of a project and for collaborating with the client.
Key Benefits
Agile was developed as a methodology to help create faster feedback loops and handle changes within projects. It is based on the agile manifesto which was written in 2001 and can be found here. It was designed to tackle the issue of a time lag between specifying requirements and the delivery of the software. It allows business requirements to adapt as customer needs change and deliver working software quickly. Using agile software is developed iteratively and allows clients and users to provide feedback quickly.
It should be noted the agile manifesto does not mention anything about estimation, velocity, sprints user stories, documentation tools, development tools or build processes, yet these are tools teams will adopt to help determine if the project goals can be met and to aid communications.
Scrum builds on the concepts of agile; scrum outlines a number of key activities (called ceremonies) and roles that should be followed in order to achieve a successful project delivery, more information on scrum can be found here
Whilst the benefits of agile are tangible, teams should be mindful that agile projects still have budget constraints and the development must still take place within those constraints. At the heart of agile is a functioning team, which includes the customer, to develop functioning software.
The Agile Procurement Paradox
The Contract
One of the key tenets of agile is "Customer Collaboration over contract negotiation" that is the idea the client and the software house work together as a team to build working software; and this enables the agile process to “welcome changing requirements even late in development”.
For companies just setting out on their agile journey this a difficult concept to grasp, that they are paying for a piece of software but the exact shape of the software is yet to be defined. Companies moving from a traditional procurement processes into agile for the first time will specify exactly what they are receiving (and when) and will try to build this into the contract, specifying the requirements and locking in development to a specific piece of software.
When development is complete the procurement team will then go through the project and tick everything off that’s been developed, and request that contracted items are then built (even if those items have been descoped). In addition, clients new to agile can forget they are still working within a budget, wanting to iterate repeatedly over one element of the software without stepping back and considering the wider piece of development.
There are a number of ways to handle this:
Consider the approach and the contract
- The procurement team have clearly got a reason for why they are commissioning software and a broad idea of what the software should do. Contract negotiation should therefore define these as goals, but not specify how the goals are met (for example "there should be a product search," but not that there “should be a product search which lists 20 items a page, results should include partial matches, a ranking and allow preferred products to be listed at the top of the results”)
- If the contract is highly detailed, such that the client, development team and QA all agree that they can develop and test to it, and it meets the user’s needs, then a waterfall methodology could be considered (but with contingency).
- If the estimate of the work for the highly detailed piece is less than 4 weeks consider a waterfall development instead, as 4 weeks generally isn’t enough time to sprint more than twice.
Accept Change
In a highly detailed contract accept the changes but ensure that the procurement team are notified of the changes. This does require some high level diplomacy skills, but this will smooth things with the procurement team when they come to review the delivered software. One tool that can aid this is a project control document which everyone in the team maintains. This logs among other things key decisions, actions, assumptions and changes. Details logged include the date they were raised, who raised them, a description and any supporting comments. This then provides a historical record of decisions and acts as a reminder for any outstanding actions and risks. Also make sure that everyone understands the impact of the changes.
The Initial Estimate
In order to determine a budget for the project the development time required to build the piece of software has to be estimated. In agile the estimate for the project has to be done before the details of the requirements are known. The estimate will be based on the teams combined knowledge, but before the requirements are known. This can result in accurate estimates. It’s worth noting this happens with "traditional methodologies" tm too when responding to an RFP.
At this point because not enough detail is known the initial estimates are in tshirt sizes and are reviewed by all the agile team members to ensure that they are feasible.
As well as considering the traditional roles of UX/ Development / QA/ Project manager, additional roles will need to be considered for the scrum master and product owner roles (although a BA will traditionally be a product owner). Estimates will also need to consider that in a 2 week sprint 5 hours can be spent per team member on scrum ceremonies (this is aside from any other meetings that are required by team members). This can make agile seem like an expensive option, but the ceremonies:
- Enforce faster feedback loops
- Ensure that the client and development team become one team, fostering a positive collaborative relationship.
Roles
When working in scrum agile the roles of product owner and scrum master are well defined, as will be the roles of each member of the development team (UX, front end developer, back office developer, full stack developer etc). However it is important to recognise the client will also play a role in the development, as will end users of the software.
Client
Whilst working through the goals of the project it is important to gain an understanding of the internal stakeholders and their timelines. This is particularly true in a large development or when working with a larger organisation as the client contact may be working with a number of different stakeholders each with competing priorities.
By gaining understanding to some extent the project can be shaped to ensure that the project goals align to key stakeholder goals and availability. It also ensures that the right people are consulted at the right time. A RACI matrix works well for this. Without this information the scrum team may not have enough information to make an informed decision as to what shape the sprints should take.
Users
From the agile manifesto "Working software is the primary measure of progress." For me software works if the end users can use the piece of software. In order to do this we utilise a technique where we visualise a user’s journey through the site, for example on a shopping site it might be:
- View a list of products
- Add product to basket
- View basket contents
- Confirm basket contents and go to check out
- Enter delivery details
- Enter billing details
- Enter credit card details
- See order confirmation screen
Each of these items become a story, and from each story a set of tasks can be built.
It’s important that the working software delivers a story and not a prioritised task within that story. For example, in an ecommerce site there’s no point integrating with a courier service app if the user can’t actually add the product to the basket.
As the agile process is iterative with the next iteration of the software may be to improve the product listing, or allow multiple delivery addresses.
Prior to development
Discovery
As developers we are all eager to jump in to build a thing. As a developer who wants to build working software that end users will use, for me it’s important that prior to development there is a discovery phase where ideas are prototyped with the intended users.
Prior to showing a prototype to the intended users, it is useful if the multi-disciplinary development team review it. Making time for an internal review prior to showing the prototype to clients means that the whole team can gain an understanding of the prototype. This also gives the team a chance to ask questions and make tweaks. They can even advise that a certain feature may blow the whole development budget to build, meaning that the software won’t fulfill the client’s original goals.
The prototype may go through a number of iterations, ideally with end users. Once the final prototype is agreed upon, the journeys it explores can be broken into stories, and the stories into tasks. Usually these are plotted on post-it notes to enable the stories and tasks to be moved around until the journey is optimal. The tasks become part of a product backlog.
It is important throughout the build process that the team returns to the journeys to ensure that the piece of software being developed will perform the goals required.
Stories are usually written at this stage "as a [type of user] I want [the goal] so that [the reason]", known acceptance criteria may also be added.
Keeping the same team on the project from prototyping through to build complete ensures continuity of knowledge throughout the process. However for longer projects onboarding of other team members should be considered and planned to ensure that knowledge is shared throughout the team.
Prior Discovery
The client may already have been through a discovery piece before moving to work with a new development team. This does mean that whilst the client has an understanding of what the software should do, the new development team does not. Here the key to helping ensure that the project is a success is to not only read through the discovery the client has undertaken, but to actually sit with the client to talk through the background to the project, why they feel there’s a need for it, and who the project is for. This process often tells the team more about the project than any set of requirements. It acts as a team building exercise, and a fresh set of eyes may help shape the project further.
Build
After discovery the project will move into build. Here the development team will work with the client to prioritize key journeys. This prioritisation may require a trade off with riskier pieces of development which must be developed in order to allow a user journey to complete, for example integrating with an online bank. By prioritising the riskier pieces it is easier to understand how much time is left to complete the easier-to-develop functionality, which can start to have complexity reduced if time starts to slip.
Once a broad outline of the development order is agreed, the tickets for the first sprint are then reviewed to ensure there is enough detail on them to progress development and QA. In order to give the product owner the space to flesh the initial tickets out in more detail, the development team tend to send up the environments and the CI/CD process as far as possible. They will also work closely as a multi-disciplinary team with the product owner to ensure that the tickets contain the detail required.
Sprint Planning and Refinement Paradox
The agile manifesto says "Working software over comprehensive documentation." It does not say, and I cannot emphasize this enough, “no documentation”. Agile works well because there is a recognition that not all requirements are not known up front. By the time a development team member comes to work on a task the requirement for that task should be known. The task should be defined enough that the developer, tester and customer can all agree when the task is completed.
It's also important to say the manifesto emphasises working software, so as part of each sprint, rather than working on a prioritised list of tasks, the tasks should align to delivering software which works.
The sprint planning and sprint refinement sessions enable everyone on the team to understand the goals of the upcoming sprint, and allows everyone to focus on reaching that goal.
Sprint refinement sessions (where the next sprint is planned) allows the product owner to flesh out the tasks for the next sprint and ensure they have time to work with the build team to fill in the detail required. Once they get into the swing of how the development is taking shape, product owners can also start to work out the tasks in the next but one sprint.
At the start of sprint planning the team availability should be understood so that it takes into consideration holidays, conference attendance, training and team makeup due to budgetary considerations.
Supporting Live Software
New development can be ongoing at the same time as the software is live. In this case it is easier to run a unified backlog where support tickets are assessed for urgency and are either handled as a high priority above the existing sprint, or form part of the backlog to be addressed as part of sprint work.
Stretch goals
In some cases a stretch goal may be defined in a sprint. This tends to be set when out of sprint planning there is a recognition that a particular goal will be met, but not the key goal. For example the goal may be "complete the add to basket functionality" with a stretch goal of “complete the checkout process”.
I find this demotivating and frustrating. Tt is asking the team to an agreed "basic" goal, whilst saying to the team it is recognised that the “actual” goal may not be achieved. However the stretch goal also implies “if you work this little bit harder you might just” achieve it.
At the same time as the product owner you are reporting back to your stakeholders the goal of this sprint is to complete the stretch goal.
This results in frustration from the stakeholders when the stretch goal is not met, and from the build team, as they advised this goal would never be met. Far better to set a realistic goal and allow team members who finish early to support other team members. If no support is required then they can pull in tickets from the next sprint should they finish early.
Changing Sprint Goals
Where a product owner insists on changing goals in a sprint it is important that they understand they are asking the development team to context switch and this may have a detrimental impact on the sprint. **BUT **it is also important that the team recognise that one of the aims of agile is to allow last minute changes to sprints.
Clients In Sprint Planning and Refinement
One of the key tenets of agile is that everyone involved in the development, including the client works as a team to produce working software.
This means that the client should be included in sprint planning and refinement sessions. However, as previously discussed, in larger organisations or larger projects there are competing goals, so what the client agrees on the sprint planning/refinement session and then feeds back to the wider organisation may subsequently result in a change in direction of the sprint due to internal politics. This is why it’s important to understand the client’s stakeholder roles, and to align them to the sprints so the stakeholders can be managed in each sprint, this will alleviate much of the politics.
Estimating
Tasks can only be played in sprint and estimated when the developer and the QA have all the information they need to be able to develop and test that story.
Some teams will estimate in hours, others in story points. My preference is for story points ensuring they are relative to a baseline so it may be that we know it takes 1 story point to build a footer, but the header is twice as complex so it is two story points.
The team should agree an estimate size limit, where if the tasks are over this limit they require dividing into sub tasks. Some people advise this should be the equivalent of no more than 4 hours others 3 days. An upper limit as it could indicate a lack of knowledge about a task, and that it requires further definition. There are always exceptions to the upper limit rule, for example when doing a large upgrade of Umbraco.
There is much debate as to whether a client should be involved in estimating a sprint. In all honesty it depends on the client. Some just want the goal to be met. Others take a deep interest in the "how", and want to understand the complexities.
In my experience when clients are involved in the estimation part of sprint planning more time is taken by the team to explore, explain and clarify the tasks and stories. This results in a greater understanding of the effort required. When the sprint planning and sprint refinement sessions work well they can help establish trust between everyone in the team.
Whenever a client joins in the estimation listen to how they estimate:
- When the client’s development team are involved they may explore the estimation in more detail, especially when their estimate differs widely from the delivery team.
- There may be a person on the client side who uses estimation to indicate their priorities. If the client really wants a piece of work the estimate will be low, if they don’t want it will be high.
Sprinting
In my view (and it is just my view) two week sprints tend to work best, with one week being too short, and three weeks too long. However the length of a sprint is arbitrary and can be set by the build team. It is important that the build team are not developing new features for the full duration of the sprint, that they can quickly and easily release to test, and that the last couple of days on sprint are spent bug fixing the current sprint. If there is time they can pick up new tickets towards the end of the sprint, but if there is no chance of turning them around in test, they are not to be released to the current sprint, and must not form part of the sprint review/ showcase.
Velocity
Often times a team will assign values to the tickets in the back log to determine if the backlog can be met in the sprint values can be story points, t shirt sized or hours. What can happen is that product managers then track the values and try to guess a velocity. If this is happening it is important to not work over time to meet a sprint goal, as this will screw up velocity and could mean committing the team to extra hours each sprint. Project managers will also try and allocate time to a velocity.
It’s important to note that changing the team means changing the team and the velocity. Also having the same team working on the project for half the time does not necessarily mean the velocity could be halved, it could reduce even further due to loss of context in the meantime.
Tool Paradox
Because there can be a lot of tasks and stories, and post-its are easily lost, tasks and stories usually get put into a tool such as Jira, Trello or Azure DevOps. Using such tools does mean that the context of the task and story can be lost, so prior to moving them take photos of the journey map.
To help with analysing the work load it is useful prefix the tasks with the role of the build team member who will be fulfilling them (so UX, BED, FED etc) and also attach an estimate of how long the tasks will take.
By moving to a tool work hard to prevent all the ceremonies becoming about the tasks and stories located in the tool, where they sit in the flow. Remember to focus on the individuals in the team and the end users of the software we are making.
Sprint planning and refinement becomes about moving tickets so they are in a priority order, and becomes boring watching someone type up Jira tickets. We forget that the original purpose of the stories was to gain a common understanding of the feature being developed.
We have looked at ways of stopping this from happening, including:
- Post-its: break the stories in the sprint into post-its, explain the posts-it, place notes on the post-its (or if there’s a lot, number the post-its and make notes against the numbers).
- Printing out the Jira backlog and move that around as applicable (not the best for the trees).
- For remote teams using https://mural.co/.
Collaboration tools
When picking a ticket tracking tool, consider the users of the tool and who will have add, edit and delete rights. If everyone has edit rights then pick a tool which contains a full audit history of all changes on the ticket. This is important because if anyone working within the agile process can edit tickets anyone can change the requirements on a ticket. An audit trail allows everyone to see that the agreed requirements have been changed, and allows there to be no confusion as to what the original requirements were.
For day-to-day communications moving from Slack to Microsoft Teams has allowed team space to be created for each client, with that team space being subdivided into channels as required. Allowing quick sharing of ideas and an easy method for discussing issues and queries directly with the client; turning them around quickly.
However tools such as Microsoft Teams and Slack allow side conversations to take place where visibility of the work being discussed is lost to the wider team. As a side note, teams work well when all our clients are in the same workspace, and you don’t have to jump into their workspaces.
The daily communications should include notifying the scrum team if a member is off ill. On longer lived projects sharing a holiday calendar will also help, so that sprints can be planned around team members availability (both the build team and the client team). If the build team also includes client side team members, having visibility of both their onboarding and offboarding from the team also helps, particularly where knowledge has to be shared with the wider build team.
Development Process
Once the outline journeys have been described and the stories start to take shape, as developers we can start to see what the environment configuration should be like. These may include environments for test, UAT and live.
To ensure consistent, rapid deployments each project will need a build and deployment process, whether this utilises a build server like TeamCity or Jenkins and a deployment server like Octopus Deploy, or Azure DevOps, or Umbraco Cloud. Failing to configure a build and deployment process at the start of the project (even if it is just to the test environment whilst the other environments are configured) will come back to haunt the team the further into the development they move, resolving build issues just before a showcase is not the most relaxing of moves!
Every developer has their go-to tool set; mine includes using Umbraco with uSync. This is because Umbraco allows developers to focus on building a thing iteratively, especially since the introduction of models builder. Umbraco also allows content editors and content designers to enter content quickly as part of the testing process. Entering content early allows gives the QA team, editors and designers the chance to:
- Test that implementation is accessible.
- Verify that the design works with the various types of content that will be entered.
- Review the implementation against the original user journeys.
Early content entry can also lead suitable planing for load testing the site.
Feedback from the team can then be fed back quickly into the process.
Branching strategy
The development team must also decide on a branching strategy. This could be to utilise Gitflow, or to have a build process which ensures that only signed off work is deployed to each environment.
To ease the deployment process projects may utilise an ORM such as EntityFramework to deploy database changes and a tool such as uSync to deploy Umbraco changes. When these tools are used on projects with a lot of branches care must be taken to prevent unintentional data loss and deployment issues which may occur should a branch without the latest amends in them be deployed.
If the software is live the branching strategy should support hotfixes to the live site. In the case of Umbraco and other packages which modify the database on upgrade, the branching and release strategy should recognise that an upgrade may take place as part of sprinting.
Review
At the heart of the agile process is iterative review and development. Everyone within the agile team should be reviewing the software developed in the previous sprint to determine areas for improvement and these should be fed into the backlog for prioritisation for future sprints. However adding extra stories into the client backlog requires trust on both sides; with the client being advised that these are recommendations for improvement to the software and not an actual agreement of work to be carried out.
Changes
At the heart of agile is the ability to acknowledge and embrace change, and this could result in the backlog growing from that envisaged originally as new stories are found or it could result in stories in the original backlog being deprioritized. It is important that where a tight deadline has already been identified for delivering the original contracted goals the client is made aware that the changes will impact deliverables. An easy way to do this is to have a spreadsheet with the key stories prioritised and a line projecting where the deliverables will fall in that spreadsheet. This allows the client to see the impact of introducing a change will hit the existing agreed deliverables. In such cases a project control document will also help.
Deployment
The application will require deployment. Plan for deployment at the earliest opportunity¸ not only in building out the release processes, but confirm everything about the deployment such as URLs and people who need to be involved as soon as possible. This is particularly important if there’s a hard deadline (also find out about the hard deadline as soon as possible).
Summary
Agile done badly becomes all about the tools, process and box ticking. With change and collaboration at the heart if its values, when it agile works well it’s a great way to ensure that the client has a working piece of software that their users will use. Agile done well means building the right thing for both the client and their users; meaning that when the project reaches a major milestone everyone in the team is happy.