Agile in the UK Government - An Insider Reveals All
Described as “The Biggest IT failure ever seen”, the £12 billion wasted by the UK government on the failed NHS Patient Management System highlights that something is fundamentally broken in UK government IT. That needlessly-wasted £12 billion could have helped fix many problems in society - it could have been used to pay for 1.3 million university tuition fees, build 250 new schools, or purchase 50,000 new ambulances. Unfortunately, it is one of many examples of wasted taxpayer millions in government IT.
But there is hope, the UK government now has GDS - Government Digital Service. Not only do GDS plan to fix government IT, but they plan to completely transform the relationship between citizen and state, moving the UK towards becoming a world-leading digital-by-default government. GDS boldly claim that they are going to put users first and deliver services that address genuine needs of UK citizens. GDS also claim they are going to promote a culture of agile software development across government; to make efficient use of taxpayer money, and bring an end to the many Big IT, outsourced waterfall failures of the past.
So after 5 years, how much, if anything, have GDS achieved? Are £12 billion IT disasters still likely to happen? And is agile software development in the public sector saving or draining taxpayer money? I’m Nick, I have been working with government agencies over the past couple of years, and I am going to tell you exactly what I have found out and why you should care.
Users Must Come First - No Really!
My biggest surprise working on modern IT projects in the public sector is the uncompromising focus on delighting users - citizens of the UK. Many organisations wildly proclaim in public the importance of putting customers first and how an organisation must revolve around them. Until you work or visit those companies, and realise it’s all just PR. But that’s not true in the UK government now GDS are around - user needs, invariably, must always come first.
User-oriented Project Lifecycle
IT projects in the UK government must follow an approved GDS format. During each phase of the lifecycle you have to personally demonstrate to GDS that your team has understood who your users are, what their needs are, and how the system you are building addresses those needs.
Image source: GOV.UK
A project kicks off with a short discovery phase. A government agency must get a small team together who can conduct preliminary research to work out who the users of the service are likely to be and what their high-level needs are. They’ll need to work out how they will finance the project and the people they will need in their teams. When the agency has done all of this, they can then move on to the alpha phase.
It’s during the alpha phase where the pace starts to pick up. User researchers will start interviewing users and asking them to trial out a prototype. The developers and designers in the team will continue to rapidly iterate on that prototype as feedback is gained from user research. During alpha, developers will also conduct technical investigations. Among which, they will be spiking technologies and infrastructure to determine what they will use to build the service.
At some point, the team will feel content with their prototype. They’ll want to start building the real, production quality system. They’ll want to move on to the beta phase. And that is the first moment of reckoning. Before a team can move to the beta phase, they must book their alpha assessment with GDS.
User-oriented GDS Assessments
If you think that all of the “users needs must come first” hype that pours out of GDS is just self-congratulatory praise, you haven’t been to a GDS assessment. It was during the first GDS assessment I attended where I learned that it’s not just hype - that user needs really must come first in government IT. I always had the impression that GDS were just a government agency who did their friends in other government agencies a favour. But I couldn’t have been more wrong, and my team found out the hard way; our user research had focused on one particular type of user (for a particular business size). GDS pointed out how we had neglected other types of user and would need to interview more of them before we could proceed.
When assessment day rolls round, a small selection of your team are invited to the GDS offices in High Holborn. Eventually, you are escorted into a meeting room, where, sat opposite your team are the GDS assessors. Once everyone is settled and the small talk subsides, for the next few hours, the assessors will quiz you on their 18 point Digital Service Standard.
Of most prominence in the assessment is the focus on user needs. The Digital Service Standard, dives straight into user needs. For example, point 1 is a series of questions about understanding user needs, and point 2 starts the discussion on your future plans for user research. In the GDS era, user research is something that is ongoing over the lifetime of a service. If your agency hasn’t done enough user research you will fail. You will need to have conducted research on a respectable number of users and a representative sample of the different users, including those who identify as assisted digital (needing support to use computers).
As a developer, you don’t actually have to interview users yourself, but you are expected to watch your user researcher’s sessions with users in the lab or at the very least watch the recordings.
Working in government was actually a real eye-opener for me. Watching those user research sessions and seeing users struggling to use the application me and my teammates built. At first, it was actually quite shocking. We would spend all day building and testing the application, and we’d know every piece of text, every button click and every valid combination for every form input - it would all seem so easy and obvious to us. We would then watch users on the brink of giving up because it was so complicated for them. Watching citizens feel the pain of using your applications really does change your mindset and help you to better satisfy their needs
Agile and Technical Assessment Criteria
An alpha assessment involves technical questioning, too, from a technical GDS assessor. They’ll be asking which technologies you've chosen and why you chose them, and they’ll want to see a high level architecture of how your system will fit together. For example, point 6 on the standard is “Evaluate Tools and Systems”.
Some of my experience of technology in government was very positive. The agency I worked for had a good relationship with another government agency who let us use their feature-rich platform which provided all of the infrastructure and facilities we needed, including a continuous delivery pipeline, testing environments and rich monitoring capabilities. As part of the deal we had to use Scala and the Play Framework. It’s fair to say that GDS were quite happy with our cross-government sharing. It’s a good example of what should be happening in government IT.
Remember how GDS promised to introduce agile practices into government? Well during your alpha assessment they are also looking for evidence that your team is cross-functional and capable of work in short iterations. Can you take feedback from users and quickly & efficiently update your service to satisfy those needs? Are you using tools, technologies and working practices that provide you the flexibility to iterate quickly?
You probably won’t be surprised to hear that scrum is absolutely dominant in the public sector, at-least in my experience. But GDS don’t enforce it; they are looking for cross-functional teams of user researchers, designers, developers, testers, business analysts etc. They want to make sure you are capable of iterating on your service quickly as you gain feedback from users. Some teams do choose a more continuous flow/kanban approach and that’s absolutely fine, too.
The teams I worked with, and many others I worked closely with, all achieved impressive levels of delivery. For example, at times we were deploying four times a week, benefitting from having a small, co-located, cross-functional team who could quickly make decisions, quickly change our code and quickly ship value to UK citizens. Engineering practices such as TDD, pair programming and code review played a key role in that, and so did cultural philosophies like making quality a responsibility of the whole team and keeping our batch sizes small.
You may think that four times a week sounds very little compared to giants like Amazon who are deploying hundreds of times per-day, but it’s a very respectable level for a government agency who are just finding their feet with agile.
Beta & Live Phases
Passing your alpha and moving on to beta is an exciting phase of delivering a new service. You’ll start building the real service, connecting up all of the different technologies and having real user data.
At the start of beta, though, you’ll start off in a private beta phase. An MVP of your service is built and selected users are invited to use the service as you continue to iterate on user needs. During beta, you’ll hopefully be starting to get into the habit of continuous delivery, deploying to production multiple times per-week.
When you reach that point of wanting to open up your service to a wider numbers of users in a public beta phase, it needs to be deployed on gov.uk with a beta banner. But first you must book another assessment with GDS - your beta assessment.
Beta assessments are more of the same. Back to GDS offices. Back to the 18 point digital service standard. And back needing to prove you deserve to spend more taxpayer money. Only, the beta assessment is far more demanding, in my experience. GDS will be looking deeper into you user research and the design choices you’ve made based on user feedback. And the technical questioning is far more thorough, as they ask to see the architecture of the system, your development toolchain and examples of how you are working as a team. For example, during a beta assessment I was asked to explain how a piece of work flows all the way through our team, from the initial moment where someone suggests the idea until the moment users are interacting with it.
If you manage to pass beta, and you want to go fully live on gov.uk without the beta banner, you guessed it - you have to do another GDS assessment.
Gov.uk: An Example of GDS Brilliance
A lot of people get the impression that GDS assessments are tiring and a waste of effort. That is simply not true, though. Look at what they are trying to achieve in government and where they started from. They want to move to a digital-by-default government, transforming the relationship between citizen and state. But as you saw, their starting point is a government who is responsible for wasting £12 billion on the biggest IT failure ever seen. As tiring as they may sound, I believe assessments are key ingredient of the GDS strategy, and they work impressively well - so impressive that governments around the world are copying the GDS model.
It Almost Feels Like Government Is One Cohesive Organisation
Gov.uk is the home of the UK government on the web. Citizens start by coming here when they want to interact with government. Accordingly, it has to be easy and painless for users of all different abilities if GDS are going to get anywhere near their dream of digital-by-default. Fortunately, gov.uk really is that good.
Look at two screenshots below. On the left is the Rent and Lease Details service created by the Valuation Office Agency (VOA) and on the right is a vehicle checking service created by the Driver Vehicle Licensing Agency (DVLA). But thanks to GDS, these, and all services on gov.uk, must provide a consistent user experiences, so citizens don’t have to learn lots of confusing websites built by lots of different government agencies.
If you look at those two screenshots, it’s almost as if government is one cohesive organisation…. If only that were true.
Rent and Lease Details Service, by the Valuation Office Agency
Vehicle Enquiry Service, by the DVLA
If a service does not follow the consistent look and feel of gov.uk, it will fail its assessment. Consequently, without GDS assessments, it would be challenging to create this vital level of consistency across government.
Open Source Government Code
Another example of GDS’s brilliance is how they’ve managed to bring the open source philosophy into government. Not just using open source - but creating open source code. If you go to HMRC’s github for example, you can see hundreds of open source projects. And not just libraries either. You can find the code for HMRC’s web frontends and domain-driven microservices that are actually running on gov.uk.
I think it’s amazing that GDS have achieved this; promoting reuse across government, showing taxpayers how their money is being spent, and mitigating vendor lock-in. Personally, I loved coding in the open. It holds all developers to a high standard of work so that we are not creating a maintenance burden for the next generation that wastes taxpayer money and stops us providing value to UK citizens.
It’s Not All Rosy in UK Government IT
After genuinely complimenting GDS on their remarkable achievements in government, I also want to be honest and share with you what isn’t working so well in government IT. I want GDS to be successful. I want the UK to have a world-leading digital-by-default government that enhances the lives of citizens and saves taxpayer money. But we have to fix some fundamental problems first that I have observed in some government agencies.
After showing you the problems I’ve seen in government IT in the remainder of this section, I’ll then discuss solutions in the following section: “How We Can Fix The Problems”. Please free to leave a comment or suggestion if you have different opinions.
Digital and Internal IT Silos
A quote from Mike Bracken, the ex Head of GDS, sums up what is easily the biggest problem I have seen in government IT, and one that poses a serious risk to GDS’ laudable digital-by-default ambitions. He said this a year ago, just before he left GDS:
You can’t be half agile
Ironically or perhaps expectedly, that’s exactly what I’ve seen in some government agencies - half agile organisations. “Digital” is a buzzword at the moment with huge amounts of hype surrounding it. Some government agencies think that all they need to do is create “digital” teams who build the pretty frontend websites that run on gov.uk and magically government IT will be fixed! But these frontend websites need to talk to backend systems that contain business rules and databases which are built by internal IT teams who still have an outdated IT mentality.
In the public sector, I have seen digital teams starved of the autonomy and freedom to deliver their business outcomes - starved of the ability to satisfy user needs and to make efficient use of taxpayer money. Even though digital teams are practicing continuous delivery and have lead times in hours or days, the backend systems have lead times of months. The collaborative overheads, all the unnecessary meetings, and the pain of outsourcing core functionality to different suppliers result in end-to-end lead times that are anything but agile. That’s ignoring all of the expensive coordination costs of integrating the different parts of the system, integration testing them and trying to debug problems when there are multiple teams from multiple organisations involved.
Then there’s the politics. All of these different teams trying to wrestle as much control as they can into their parts of the system. I need counselling just thinking about the politics and the amount of resource wasted on pointless meetings and bloated enterprise software that claims to magically make you agile without you needing to address your IT culture.
In my experience, HMRC are one of the better government agents who are trying to do IT the right way by aligning teams with business outcomes and moving to an organisation-wide culture of continuous delivery. But even they admit to having a few culture clashes:
We frequently have to plumb our digital services into legacy back-ends, and so there has been some friction as the two different cultures meet. Our journey over the last three years has seen this friction reduce though, as we’ve learnt better ways of working together.
I struggle to comprehend how anyone who understands agile software development would think having a distinction between digital and IT makes sense considering that digital is frontend websites that are completely dependent on the backend systems and databases built by IT. Essentially, this is bimodal IT, which has been ripped to shreds by respected community experts like Martin Fowler and Simon Wardley. It has no place in a government that is trying to build and rapidly iterate on services whilst making efficient use of taxpayer money.
High-performing agile organisations now have end-to-end continuous delivery, enabled by cross-functional delivery teams who are empowered to achieve business outcomes. Until this is happening in every government agency, agile in the public sector is not good enough and we are letting down our citizens.
GDS Don’t Assess Full End-to-end Systems
What shocks me the most about the conflicting digital and IT silos is that GDS don’t seem to assess any of the backend systems in my experience. They expect digital teams building the websites to work in an agile way, focusing on users and open sourcing their code, but I never saw the same standards being applied to the internal IT teams who manage all of the backends.
I know of one government project where the digital team couldn’t even add one extra textbox to their address fields, something users were complaining about, because the backend IT teams were too busy to make the change. Despite all the positive things I said earlier, I have no idea how a clearly un-agile system that could not even satisfy a basic user need got through all of its GDS assessments.
In truth, I don’t know why GDS don’t assess the full end-to-end system. Maybe they don’t have the power to, or maybe my experiences are not representative of what happens to other government agencies. I do find it quite baffling, though, since GDS weren’t scared of making huge changes, like introducing spend controls to prevent Big IT contracts. I have tried to contact GDS in the hope that I can understand their reasoning and offer constructive feedback. They are yet to reply, though.
As long as GDS continues with this approach, they are indirectly encouraging agencies to have multiple IT silos that are biased towards local optimisations and incur expensive handover phases, which saddens me.
Long-term Focus Is Missing
After all the publicity on the GDS blog, and the enormity of their long-term digital-by-default vision, I expected to see lots of bold changes happening in government IT. Lots of teams all continuously trying to improve how they develop software, how they make technology work better in their government agency, how they can do their bit to drive government towards becoming digital-by-default.
Surprisingly, though, I didn’t see the passion and drive in some parts of government IT that I was expecting. I think there are a number of underlying reasons. One reason is that some people told me they thought agile was just a fad and they expected an eventual return to the old ways of outsourcing. Others had slightly stronger opinions, implying they didn’t even want to change.
In my experience, high-performing agile organisations have an engineering culture that encourages people to grow through continuous improvement, to develop broad skill sets and to feel motivated enough to drive the organisation forward through innovation. To me, it seems vital that all government agencies should be nurturing this type of vibrant engineering culture that encourages people to think long-term if we are going to take government from £12 billion “biggest IT failure ever seen” to digital-by-default.
How We Can Fix The Problems
All of the problems I’ve seen in government IT can be fixed. Some government agencies are already extremely competent, proving it’s possible. Somehow, we need a formula for ensuring consistent high-performance in government IT. It doesn’t need to be revolutionary, we can build on the exceptional foundations GDS have laid and supplement that with ideas from organisations that have succeeded with agile.
Make Government IT More Transparent
Just like source code is open sourced and hold developers to a high standard of work, so too should the way agencies develop and deliver services be more open. The GDS assessment criteria could be updated. Now to get past their assessments, government agencies should be asked to demonstrate blog posts, or white papers or conference talks where they have shown the public how they are spending taxpayer money - how they are working as teams and the technical architecture of their systems.
If government agencies have different silos with expensive handover phases and complex enterprise architecture that uses expensive commercial software, they will need to justify these decisions in public. There will be no hiding for those resistant to change who are holding back the digital-by-default progression. They will be held accountable, just like developers who proudly open source their code.
It’s not an unrealistic expectation to expect government agencies to be more transparent, either. GDS like to talk in public about their culture and the technical architecture of the systems they are building. HMRC, too, are very open about their journey to continuous delivery and DevOps and about their MDTP PaaS. It’s now time for the rest of government IT to open up and be more transparent.
In my opinion, GDS have to be assessing the full implementation of a service. From frontend web page, to database column, to internal case management system. All of this should be subject to a GDS assessment if we are going to have confidence that agile in the public sector really is making efficient use of taxpayer money.
Overall lead times should be part of the assessment. GDS should be assessing at the system level and not the silo level. For example: “How quickly can you add a text box to a web page?”, “Can you demonstrate how you changed a business rule in your system? How long it did it take? How many teams were involved? Can we see the changes in your source control?”. Equally, all of the code should be open source, too, holding everyone to the same standards of quality and collaboration.
If a team who can’t even add a textbox to their web page can get through a GDS assessment, perhaps the same level of scrutiny that is placed on user research should also be placed on organisational agility.
Empower Leaders Who Understand Agile Software Development
To keep doing the same thing and expecting different results is insanity. To keep the same people who did Big IT, outsourcing and waterfall for 20 years and expect them to lead an agile transformation is miracle-expecting insanity. Government agencies need people who understand agile software development - who have experienced continuous delivery - to be in positions where they can influence digital transformation. Where they avoid digital and IT silos, and where they can start to focus on a long-term culture.
If you’ve ever been part of an organisation who embraced continuous delivery and were deploying value to their customers multiple times a day, you’ll know that it takes business buy-in and a leadership that understands agile to get there. Such organisations also tend to be high-trust, empowering their people and helping them to grow through an engineering culture that encourages continuous improvement. But how can we expect government agencies to reach that level when they rely on the same people who haven’t experienced self-organisation and continuous delivery?
Wholesale changes aren’t necessary. But government agencies do need to bring in people who understand agile software development, and give them enough authority to make the big changes as HMRC did:
HMRC knew it needed to change, but didn’t know how. The experts brought in knew that successful organisational change is enabled by proving that the new approach delivers value. Thus, the founding team used agile, lean and DevOps practices to deliver small changes, early and often, into production.
Embed GDS in Agencies
There’s one solution to all of these problems. It’s something I thought about in government almost every day I worked there: “why aren’t GDS here with us?”.
Instead of GDS having to interrogate government agencies with 4 hour assessments and hope they catch everything, imagine if assessors from GDS spent time at those government agencies. Just like in education where we are graded on exams and coursework, GDS assessments could also be practical and evidence-based.
And whilst GDS are in government agencies, they can be the leaders who understand agile software development. They can spot the IT silos and help the agency to fix their dysfunctions - helping them to achieve end-to-end continuous delivery. With this hands-on approach, it’s entirely possible that one day, all government agencies might be able to add a textbox to a webpage in under a month.
Being respectful to GDS, they are extremely receptive to agencies who are looking for help, and will go to a lot of effort to support them. But the problem is that some agencies don’t know they need help, or don’t want to change.
It’s also possible for embedded GDS people to help promote a culture of openness and transparency. To encourage the blog posts and white papers. To give people the encouragement to proudly talk at conferences about the amazing things they are doing in government IT. To lead by example.
Is Digital-by-default Really Achievable?
I have no doubt the digital-by-default dream is entirely possible. GDS have laid solid foundations and have already achieved great things with gov.uk. They are the inspiration for governments worldwide. With the user-first mindset they have instilled across government, the agile principles they encourage in digital teams, and the openness and transparency they have demonstrated, GDS are agonisingly close to being on course to achieve their monumental ambition of transforming the relationship between citizen and state. But they need to find a way to penetrate all of government IT - not just the pretty frontend websites.
About the Author
Nick Tune is passionate about delighting users, creating business impacts, and crafting quality software, placing an equal focus on improving both the execution capabilities and alignment of an organisation. He specialises in transformation projects, having worked with a number of organisations in both the public and private sector to achieve continuous delivery. He is the co-author of Patterns, Principles and Practices of Domain-Driven Design, blogs at ntcoding.com, and can often be found speaking at conferences and user groups.
More thoughts along similar lines
Re: More thoughts along similar lines
I wouldn't argue with too many of the things you've said in your blog post. I do hugely admire the incredible achievements GDS have made at transforming large parts of government IT, and I find their blog posts highly encouraging - their attempts at making government IT more open.
You're right about the self-congratulatory nature of their blogs, though. I do also find it a little over the top at times. And I agree with you about wanting to know about failures - as I mentioned in this article, I want government agencies to be more open. That includes how they work as teams, how they build services, the technologies they are using, and as you are keen to see - how they've failed and how they plan to learn from their mistakes and make better use of taxpayer money in the future.
I worked in government and I know increased transparency is definitely achievable. As you touched on in your post, it's about cultural transformation; government has a history of outsourcing, big IT, waterfall and not being transparent and it's not easy to expect overnight transformation at government scale. But as you also said, it would be great if it was at least on a public roadmap and we knew they were working towards it.
Great observations that correlate with mine
I wouldn't be too critical of the tone of the GDS blogs, it's critical part of agile to identify and repeat what works as well as what doesn't, for me it simply highlights the good stuff worth shouting about.
In terms of lack of end-to-end coordination, this is down to legacy contractual obligations government has with the big consultancies who still have a strangle-hold on poorly developed and in some cases broken backend systems this can't be blamed on GDS. Until a sufficient number of those legacy contracts have expired and been picked up agile practitioners to develop in the right way, end-to-end digital is not possible I'm afraid.
Re: Great observations that correlate with mine
You are right though; some government agencies are still lumbered with the big contracts and have no choice. And positively, some are overcoming the contracts and doing things the right way: thinking about fully joined up end-to-end services owned by a single team who can quickly and cost-effectively iterate them.