BT

The Surprising Truth About DevOps in Banks

| Posted by Benjamin Wootton Follow 0 Followers on Nov 23, 2015. Estimated reading time: 18 minutes |

A note to our readers: As per your request we have developed a set of features that allow you to reduce the noise, while not losing sight of anything that is important. Get email and web notifications by choosing the topics you are interested in.

I have spent much of my career as a software engineer within four global Investment Banks, working on a mixture of trading, risk management, and middle office systems. Overall, I spent around around five years working on development and DevOps work in this industry.

Since establishing Contino, a DevOps transformation consultancy, I have worked across a much more diverse set of industries including retail, media, insurance, education and manufacturing, where I have been advising our clients on DevOps, Continuous Delivery and related agile practices with the aim of speeding up their software delivery and release cycles.

One thing that has been fascinating from this experience is looking at the contrast between banking and other industries. I have observed first hand that banks are a very different animal to the rest of enterprise IT when it comes to DevOps adoption, with their own challenges, but also their own unique opportunities.

Though many of the people within banking will be surprised by this, I actually think banks are in many ways relatively DevOps mature and in an even better situation for improving their DevOps capability going forward.

In this article I'll begin by describing where banks are typically effective, well placed, or have a head start on their DevOps transformation relative to other industries. Then I will follow up focusing on where banks have lower maturity and more to do on their DevOps transformation.

A Culture Of Engineering Collaboration

The key idea behind DevOps is breaking down the silos between development, operations, and other groups such as architecture and testing, and getting the engineers within these departments to collaborate more effectively on a person to person basis.

Banking IT is generally heavily siloed in terms of organizational structure and reporting lines, and also often suffers from the geographic distribution of teams or off-shoring.

Culturally however, I believe that there are not the same barriers and lack of cross department collaboration as I see elsewhere in other industries. On the whole, banking technology organizations seem less siloed than in other industries.

On my development teams, we generally had a positive relationship with testers, working together early in the development cycle in a highly collaborative way. Although offshore, our testers were highly skilled and had a high degree of understanding about the system we worked on together. We understood their world and they understood ours, and incentives, goals and KPIs were aligned across both groups - to deliver high quality software, early and often.

Likewise, developers in our team were in regular and effective collaboration with IT operations engineers, supporting day to day production activities, working with operations to safely deploy our patch releases, and incorporating operations feedback and requirements into our book of work.

With both test and operations, this effective collaboration happened in spite of the organizational barriers that were put in front of us – I believe an example of culture trumping process.

I do not see this level of collaboration in other industries, where the phased, throw it over the wall mentality is much more pervasive. I work within some industries who seem to have everything in their favour for DevOps collaboration compared to banks (e.g. co-location, flatter hierarchies, shared budgets, cross-over in roles), but culturally, development and operations seem much further apart.

Developer Ownership Of Production

Because banking systems are important and the consequences of failure are high, developers tend to have responsibility, understanding and a degree of ownership of production. They know their way around the servers, how their code will be deployed, and how the code and infrastructure interact. They know how the system performs, how to monitor the system, and what the failure modes are.

Again, this is in stark contrast to enterprise software delivery in other industries where often developers have no visibility, access or ownership into production at all. Instead they have to operate in the ‘throw it over the wall’ way due to restrictive business processes and locked down access.

Production stability was always the top priority for the development teams I worked on. This was baked into the culture and way of working, and accepted without question, even by the business and project managers keen to release their new features.

Good behaviours emerge from developer ownership of production. Developers build more operable software because they are closer to its support and maintenance. They build higher quality software because the consequences of their code or deployments going wrong impact them directly. When production incidents occur, they can be resolved faster because developers understand the system to a greater degree.

This is a wonderful basis for DevOps as everyone is aligned behind the same goals and priorities. Developers in banking IT seem to first and foremost believe that their responsibility is to deliver high quality, robust and reliable production systems rather than just producing code. This alignment is the optimal place to start a DevOps transformation from.

Equal Status Of IT Operations

In many organisations I work with, IT operations is a second class citizen – underfunded, overworked, and given little say in key technical decisions and direction. Operability of the system suffers as new features take priority, with the business and by extension the developers driving activities and priorities. All operations can do in this situation is try to run and keep up with an increasing flow of change and incidents as technical debt mounts and their job becomes harder and harder.

Because production stability is so critical, operations and development are more equally balanced within banking technology, with the operations teams having a real seat at the table. Operations are generally involved early in the development cycle to ensure that new systems and features are supportable and that environments are sized correctly. Operations manage to keep control over new code or technology going into their environments, and generally have a right of veto on technical decisions or releases based on a risk assessment.

This can definitely go too far and become a dynamic that is restrictive of change, but on the whole I believe a healthy tension between development and operations is better than the highly subservient, underfunded operations function that I see in other industries.

Full Stack Understanding

The typical banking development team usually screens their developers for a combination of development skills, unix skills, scripting, understanding of the database tier, and architectural elements around resiliency, performance and scalability.

This optimizes for engineers who are relatively ‘full stack’. Often developers in a banking technology team know their way around the server environments, know how to tweak the deployment code, and know how to manage and run their system in production from the front end to the application server, to the database.

In many cases, the same developers write the front end and the back end code, or the middle tier code and the database code. This has its downsides, but it does mean avoiding that internal mini-silo (front vs back end) within the development team which is more prevalent in other industries.

This full stack generalist understanding of code, system administration and infrastructure is great for DevOps, as it inherently eases communication, supports collaboration, and reduces the need for risky handovers in the broader team.

Modularisation Of Applications and Enterprise Architecture

I have been working with a number of organisations this year who are looking to break up their monolithic applications into Microservices as part of their move to DevOps and Continuous Delivery. I believe that banks will have a generally easier time on this journey because of their enterprise and application architectures.

There are lots of big, legacy applications within the average banking environment, but one thing that banking technology loves is message oriented middleware. It’s a rare banking application which isn’t built around middleware such as Tibco, Websphere MQ or Informatica.

This gives banks a head start when it comes to breaking up their application platforms into components which can be independently developed, tested and deployed. It is by no means all of the Microservices puzzle, but it’s a good basis to build from compared to some of the tightly coupled monolithic architectures which I come across in other industries.

A Bias For Automation

Though banking technology teams have not been quick to adopt modern DevOps infrastructure automation tools such as Puppet, Chef or Ansible, they do have a bias for automating the management of their systems, their environments and their releases.  

All of the banking technology platforms I worked on were held together with tens of thousands of lines of shell scripting, attempting to automate releases and operational activities, sometimes around middleware platforms which were challenging to automate.

When we patched our systems, we would also generally do this with automation - through scripts which were source controlled and passed through environments according to a correct SDLC.

Enterprise organisations in other industries are much more manual in their approach, clicking around in GUIs or following manual runbooks to implement changes. This leads to wasted effort, but also inconsistencies in environments and snowflake servers, which often become a source of production incidents.

Automation is a key tenet of DevOps. When we automate a process, it becomes faster, more repeatable, and more efficient. In a DevOps mature environment, anyone can run and modify the automation scripts, breaking down key person dependencies. Automation speeds up release cycles, but also captures necessary audit and control information over who has done what within the system. Culturally, skills and process wise, I think banking technology teams are ready to make this leap.

High Degree Of Scrum Maturity 

Banking technology teams are generally in what we call a water-SCRUM-fall situation. This means waterfall approaches to project planning and initiation at the front-end, and waterfall, serial approaches to getting code out of the door at the back-end. However, all of the teams I worked with within banking technology have been fairly mature in the middle, at the Scrum part purely within the context of the development team.

Sprints, standups, planning, estimates, showcases and retrospectives were all common place in the teams where I worked. We would be genuinely working in 2 week iterative and incremental sprints, and incorporating at least some degree of collaboration with the business.

Likewise, many of the technical practices associated with agile were also present in banks. Continuous Integration, Test Driven Development (TDD) and Behaviour Driven Development (BDD) were very widely adopted.

DevOps is about extending the principles of agile software development into operations and back into the business. Therefore, agile maturity within the development phase is an absolutely key building block. Banks have a lot to do to improve their water-Scrum-fall situation, but are generally ahead of enterprise organisations in other industries with regards to agile maturity.

 

Where Banks Have More Work To Do

In the previous sections, I've described why many retail and investment banks are actually in a relatively good position when it comes to some of the most difficult elements of a DevOps transformation, including culturally and architecturally, and the level of agile maturity and engineering quality in place. Banking technology is far from perfect, but these observations are made relative to other industries.

In the next sections, we turn this around and look into areas where banks have less DevOps maturity and have a ways to go on their journey.

Technical Legacy

Banks have a significant problem of technical legacy. Their platforms have evolved over decades, in many places still incorporating mainframe technology somewhere at the heart of their platform. Banks also suffer from technology sprawl, as they have deployed many applications built in many different technologies over the years, integrating a mixture of off the shelf and bespoke software. This technical legacy is like a ball and chain to banks, restricting their ability to move quickly and with agility.

Technical legacy can be a real challenge for DevOps adoption from both a collaboration and automation perspective. For instance, specialist, niche, legacy skills lead to silos of key man dependencies, resulting in the very opposite of the collaborative environment we are looking to build. It can also be difficult to get these system under build, test and deployment automation in order to support faster release cycles. Release cycles are slow, so we cannot move to the fast, lightweight, experimental approach that DevOps aims to facilitate.

The interesting thing about technical legacy is that there is often a real business case for investing in DevOps initiatives in the legacy estate. The systems which are often most expensive to develop, change and run are the legacy systems, which need big teams of hard to find resources to support them. Targeted interventions can secure significant cost savings and add some agility and innovation back into tired old oil tanker platforms and enterprise architectures.

Slow DevOps Tooling Adoption

Banks are always cautious in their take-up of new technology. Sometimes this is for good reason, as they need to govern the enterprise architecture and tools in order to drive consistency of approach and future sustainability. It is generally advisable for any enterprise to avoid the bleeding edge.

The problem however is that we have seen a massive amount of innovation in the DevOps tooling space in a relatively short time. Banks are missing out on this innovation due to low adoption rates of tools which can demonstrably deliver huge speed, efficiency and quality benefits.

For instance, Puppet, Chef, and Ansible are infrastructure automation tools which make it easier to manage large volumes of infrastructure efficiently and with consistency. Banks are not aggressive retro-fitting these into their established platforms, continuing to manage environments through scripting.

We are now seeing the next generation of application delivery tools such as Docker, Mesos and Kubernetes emerge and being aggressively adopted in other industries outside of banking. These are incredibly disruptive tools, but I fear it will take a number of years before they are commonly found within banking technology.

Banks now need to take a big leap forward in their tooling acceptance and maturity in order to catch up with the innovation that has been emerging from the DevOps and open source communities over the last few years.

Infrastructure Provisioning

As a developer in banking technology teams, infrastructure was consistently one of our top pain points. It would take months to secure virtual servers, with long winded approval processes and justifications for why we needed this infrastructure.

Environments became a huge bottleneck for teams as there were never enough of them. We would be delayed waiting for an environment, and projects would be impacted if another team slipped and didn’t vacate the environment in time.

DevOps is all about removing these constraints and easing the path to production so that we can iterate on code from development into production, more often and more predictably. The problem is that teams are starved for access to virtual modern infrastructure that they cannot aggressively put these headline DevOps initiatives into place, so remain restricted to legacy ways of doing things.

Banks need to move away from a legacy operating model for procuring infrastructure. They need to aggressively modernize their virtualization platforms and adopt private, hybrid and even public cloud where they are able to from a regulatory perspective. Without this, they are tying the hands of their software delivery teams.

Infrastructure Agility

Within the banking environments I worked at, we could not automate our infrastructure in any way to achieve agility and speed.

For instance, we couldn’t create servers through APIs, we couldn’t deploy servers based on base images or roll back to snapshots, and we certainly couldn’t do this at environment level to request transient temporary environments consisting of a number of servers.

These are common DevOps practices elsewhere in industry, particularly those organisations operating in the public cloud, but they are not currently feasible within banks. Virtualization platforms are always behind in their automation capability and are locked down behind a legacy operating model.

Heavyweight Enterprise Tools

There is a tendency towards implementing big enterprise class tools in banks supporting business processes such as change management, environment provisioning, monitoring, or source control. Banks trend towards enterprise software because they are overly motivated to centralise tools, resulting in the purchase of an all encompassing offering from a big enterprise vendor.

As well as being generally painful to use in comparison to lightweight, more focused open source tools, this is damaging specifically to DevOps for two reasons.

First, these tools tend to be silo specific. For instance, we have silo specific test tools which the developers can’t collaborate on, and source control systems which are complicated for testers and IT operations to use. Monitoring tools are sometimes very IT operations centric, and run only in production for licensing reasons. This means that monitoring is not tested or baked into the fabric of the application. These tooling situations re-enforce silos rather than becoming focal points which help drive collaboration across development and IT operations.

Secondly, these enterprise tools are often hard to automate. If anything, they tend to only have a basic API which is poorly documented or difficult to use. Mature DevOps tools, on the other hand, have automation baked in as a first principle and expose their functionality through simple REST APIs which can be built into an integrated toolchain.

In order to raise their level of DevOps maturity, they need to change their tooling governance, allowing for more fine grained, local control of tools; more niche, lightweight tools; and more liberal open source policies.

Developer Experience

The developer experience within banks is fairly painful for DevOps. It feels as though you are constantly fighting against your tools, your infrastructure and your access rather than these things enabling you to do a better job.

For example, a go-to DevOps tool is Vagrant, which makes it easy to build repeatable development environments on the desktop using virtual machines. I tried to leverage Vagrant in my last few banking roles, but often the desktop wasn’t powerful enough to do this. The desktop machines were also locked down in key ways which prevented me from using certain features of Vagrant which I couldn’t get approval for. I also couldn’t get the binaries for the Vagrant virtual machine images due to network restrictions, and couldn’t get the Vagrant machines I did manage to build to download dependencies from the Internet. In the end, I moved onto other things and the team lost out by not being able to use a really disruptive tool.

To move faster, and make use of modern DevOps automation tools, delivery teams need much more control over their environments from the desktop and through the path from development into production. By opening up the environments (whilst keeping necessary safeguards) and allowing developers to adopt these tools and approaches, DevOps will emerge from the bottom up.

Organisational Structure

As I discussed previously, I think culture often trumps process and organizational structure within banks to support DevOps. However, we don’t give DevOps the best possible chance of emerging due to the heavily siloed organisational structure that we build within banks.

The banks I worked within are sometimes divided into “change the bank” and “run the bank” initiatives, which report right up to different directors at the top level. These organisations will have different KPIs, different budgets, different priority initiatives, and most importantly, different cultures.

Organisations in other industries are making the move over to more cross-functional teams. Making these organizational structure changes is a move which banks should also be considering, particularly in their systems which need to be developed with the most pace and agility.

Shared Services Teams Within IT Operations

The IT operations half of the banks often introduces shared services teams for efficiency. For instance, if three or four teams have a DBA or a Websphere engineer, it may make sense to combine them into a centralised function which multiple teams raise work into. This approach seems to drives consistency and cost/delivery efficiency as it allows having the engineers within the team at full utilisation.

This looks efficient on paper. However, acting as a customer of shared services teams is a frustrating experience. Because these teams are working at full utilisation, lead time for requests increases, delaying the delivery team from completing their activities. Delivery teams start to interact with the shared services teams via tickets, moving away from the human collaboration which is at the heart of DevOps towards faceless communication through online systems. People within the delivery teams are de-skilled and begin feeling that areas of the platform are not their responsibility, creating a dangerous mis-alignment.

To move towards DevOps, banks need to unwind many of their shared services teams and put resources back into their development teams so they are more in control of their own destiny, ideally in full control of getting their own code from development into production.

Conclusion

When I work with clients on DevOps transformations, we talk about a range of improvement vectors under the headings of people, process and technology.

I believe banks score higher on the maturity scale on some of the more challenging areas - cultural, architecture readiness, agile maturity, and technical best practices.The areas where banks have more work to do are primarily technical (tooling and technology) and process based, so are more directly tractable. All in all, banks are well placed for modern DevOps ways of software delivery. That was certainly surprising to me. What do you think?

About the Author

Benjamin Wootton is the co-founder and Principal Consultant at Contino, a UK based consultancy who help organisations adopt DevOps and Continuous Delivery tools, practices and approaches. Twitter @benjaminwootton.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Not so soon! by sri ram

Not sure if your experience has been based on one bank. In my Banks have one of the most complex set of dependencies b/w diff. organizational units and central teams such as infra, release management etc. This may be primarily due to strict regulations, security and audit requirements.

Banking culture by Peter Idah

Great write-up! I think the specific problem in the banking domain stems from the nature of the business. One of the major objectives of a DevOps-approach is to increase velocity. The banking industry as a whole has a very low incentive to do so,defending their stance under the veneer of regulatory compliance, security, and stability constraints. This posture impacts the entire engineering organisation, as they are effectively incentivised to maintain the status quo; it is unfortunate to interact with some engineers so immersed in the banking culture, unable/unwilling to see viable alternatives.
My view is that there will be no major change, until the core banking business model practiced today is seriously challenged by emerging upstarts. I won't hold my breath on that though.

Nice article. Couple more.. by Ravi Ok

Nice article - I relate to most of the points.

Couple of additional items:

One is the paperwork aspect. Due to various regulatory/stability requirements, Banks require release planning weeks in advance. If you miss a step in the process, you need to start over. This prevents quick iterative cycles (i.e. you are forced towards a waterfall cycle).

Second is the risk averse culture - any production incidences will lead to punishing review and paperwork, again wasting valuable time for the development teams. While this is justified in the interest of keeping a stable production system, it also prevents adoption of newer technologies. An extreme example if this is that it is normal for most banks to use Java 6 even today.

We have solved this partially by building a hot-hot system. So we release to a secondary datacenter first. Once we verify the new feature fully, we switch datacenters and make it primary. This requires that you invest in making your system hot-hot. Another requirement is to be able to switch between primary and secondary easily (with minimal paperwork as well).

The missing link: Demand Management by Darragh O'Grady

Superb analysis, Ben - although most if not all the areas you cover remain significant challenges for banks to adopt and/or maintain.

In my experience, banks need to get much (MUCH) better at demand management before the full benefits of DevOps can truly be realized. This, in turn, means (as Anthony Jenkins, former CEO of Barclays, puts it) that banks should not have a 'technology strategy', but rather have a strategy with technology at its core (www.businessinsider.com/ex-barclays-boss-anthon...).

I agree that many of the elements are in place in banks, but the thread that weaves them all together is lacking...and more than anything else, banks need to pro-actively manage complexity (via enterprise architecture, portfolio management, etc).

Interesting reading by Jon Sleeper

Thanks for posting Benjamin. I've not been so close to DevOps for a while but it's something that I might be getting more involved in, in the new year, and at a bank as well, so this is very useful.

devops in the bank by yi gene

this is really true in the bank industry

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

6 Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT