DevOps - Pivoting Beyond Pockets
Today organizations that are on a Devops journey often end up implementing DevOps in pockets that are not scalable. The transformation to a DevOps centric “culture” can be a Hit or a Miss with this approach. This article offers a summary traps and tips to consider.
Traditional Infrastructure Operations roles are no longer scalable. The traditional system admin or the network engineer or the engineering roles such as storage engineers are rapidly changing. The difference between a developer and operations engineer is becoming more and more invisible and will eventually dissolve. This is part of a massive shift in the IT Infrastructure Industry called as DevOps.
As Dr. Ahmed Sidky puts it: “You can’t buy a culture transformation. It is hard work from within the Organization”. An uncalculated and solely engineering based approach to DevOps can be less effective and not so sustainable. Operations are generally locked in the fixed mindset with focus on "control" (controlling change, controlling risk, etc.) They often "harden" their controls, under the delusion that they can actually control change and risk. The more Agile the Dev mindset is (and it's always more agile than the ops mindset) - the more tension and the greater the friction.
Let’s start with potential traps:
- Lack of clear vision on outcomes: Right from the C level executives there must be a clear vision and outcomes expected from a DevOps transformation. The initiative must be treated as a major program with high visibility in the organization. Pivoting to DevOps centric enterprise is not a trivial activity. It involves a culture change and will need all hands on deck.
- Lack of a transformation roadmap: Design a transformation roadmap taking input from all impacted teams and not from individuals. Alignment across all impacted groups is vital and a critical success factor. For example, Product Management not being part of strategic roadmap could be a major problem during execution.
- Underestimating the scale of change and only thinking of it as a tooling or procedural change. Hands on Technical expertise are needed rather than PowerPoint decks. This is important from VP and below. Core enterprise and solution architecture teams must be formed with the resources exhibiting high empathy along with right skill set. Your architects must be terrific communicators, hands on coding skills and must blog regularly!!
- Lack of Management empathy to the ground challenges. The doers and executives must both have great listening and collaborating skills. This also implies letting teams form and empowering them.
- Lack of baseline measurement and clear definition of done as well as ongoing metrics for learning throughout the DevOps journey. Two of the easiest metrics to implement are customer satisfaction and cycle time. Some organizations may want to use cost of delay to prioritize amongst Devops priorities.
- Rigid process framework that create no value but needs to be complied for the sake of documentation or the delusion of control
- Operations Change management – extensive approval cycles for operations changes and an inability to make the bottlenecks visible creates waste and negative value.
- Lack of Agile & Lean Culture education for both leadership and staff – taking an individual approach to training is potentially low value and may be detrimental to rapid team storming, norming and forming.
SIMPLY PUT – IT IS HARD CHANGE MANAGEMENT!!
Recently, authors Gene Kim, Kevin Behr, and George Spafford developed a fiction novel called “The Phoenix Project” that outlines what a hypothetical DevOps transformation looks like. While the novel is a way for people to understand how DevOps change can happen, it might be great to hear about some checks and balances that go into this transformation. Here is a simple roadmap framework that may be customized to suit your organization’s needs.
Based on the foundational vision for a DevOps transformation and the gap between organizational culture, the organization may or may not be able to make the change sustainable.
Here is how we suggest Pivoting to DevOps should be thought of and implemented based on our experience:
- The leadership team must be ready for a real change in Infrastructure operations and ensure organizational awareness that ops is an “Internal Customer” to dev, portfolio and product management. Organizations need to realize DevOps is where the value gets realized. Think of it as ValueOps not just DevOps.
- The leadership must be willing to make hard decisions in staffing, restructuring the organization to be non-siloes, and also re-think major strategic vendor contracts if needed. Some of the strategic vendors that came with the era of mass outsourcing are not compatible or sufficient flexible be more agile.
- Remove excessive approval gates in terms of heavy bureaucratic change management meetings
- Invest in automation tools and role transformation. Engineers hardware skills need to be supplemented with cross training on technology, this takes care of situations where team/s are waiting on an individual to carry out task that could be performed by certain individuals.
- Form communities to share the emerging good practices that can be leveraged across the organization.
- Get ready to restructure entire departments. Once the agile teams are formed (Scrum/ Kanban) the waste will be exposed, which will bring about transparency in stories/ tasks delivered by the various teams. This cycle will further expose and elevate opportunities for improvement such as speeding up delivery, increasing and operations, and design and build.
- Invest in developing technical skills to include 3 dimensional thought process and ability to pair quickly with application development teams. The priority needs to be on the people above process in talent development. Organizations should experiment in enabling people to rotate on roles so they develop a greater empathy and experience.
- Code Quality profiling and Test Automation: Tools for configuration management and code quality will not provide full benefit if the Code base is not checked for quality based on standard quality profiles. The quality profiles must be part of the performance management system as a shared goal; and code quality analyzed with quality thresholds so that “bad” code is rejected (with technical debt eliminated at the source).Make it difficult for bad coding practices.
- Get ready to change the performance review process and make it more team centric than individual performance.
- Most of all start with a team that is open to embrace change, as a pilot that will meet success by learning quickly.
- Ensure your organization has a decent portfolio management framework that uses the unit of capacity as team not as headcount only. Portfolios should look at whole teams servicing a market/ P&L rather than just headcount and ideally these product-centric teams will remain intact beyond the “project”. This is especially important when you are dealing with large scale. Enable and empower engineers on the team to pick up cross functional skills for example a Sys Admin must be able to work on cloud compute as well as physical hardware, and on various operating systems.
- Developers and Infrastructure must start to talk in the same language through tools.Have a strategy for the deployment pipeline. Don’t just look at it as code line up but as a Value Stream right from Product Management to Deployment. Ensure every team and department managing a high quality deployment pipeline. Every time there is a code failure it must be fixed right then, validated and deployed in the respective environments, this will enable you to keep all your environments in sync with updated with code/bug fixes.
- Reduce batch size to reduce deployment risk and to improve overall cycle time. This needs an effective collaboration between engineering and product management. Define work packages as weekly value increments.
- Optimize the teams based on value streams. Optimize Product Owners too within the DevOps teams. Stretch the definition of the DevOps term to fit Product Owners, without them it is just not effective enough.
- Finally make work visible through teams. DevOps needs standups not within infrastructure but within the value stream. It is tricky to give status to every development team hence a simple visible board to reflect backlog / in-progress (WIP LIMIT) / Review & DONE will go a long way in improving customer satisfaction and visibility.
Don’t be afraid to fail fast to succeed sooner and adapt your traditional infrastructure management as it may be causing more damage rather than fostering a better performing organization. The traditional infrastructure and application developer roles are changing. It is time we adapt to the benefits of DevOps and scale it beyond the pockets to the top of the value stream starting with the Product owner.
As members of the broader DevOps community. We would like to acknowledge the thought leaders whose strategic direction, books and articles have helped our journey.
About the Authors
Kamal Manglani is an internal Agile Coach with Walmart Labs. His background is strong hands on practitioner experience delivering cutting edge technology products in fast paced fortune 500 companies and has successfully implemented Agile across global brands from the US, Europe and Indian Markets. Kamal has pioneered and customized agile practices within the IT Infrastructure portfolio applying Lean Kanban principles. He regularly coaches agile to non IT functional areas such as HR and Finance.
Gerald Bothello manages infrastructure delivery at Walmart Labs. He has over 14 years of experience in IT as a developer, architect, Technical Program Manager and Manager in software and Infrastructure. He has successfully delivered several large scale projects using cutting edge technology and incorporating agile methodology for fortune 500 corporations.
Out of 15 suggestion, i l feel below four are important
Remove excessive approval gate,
Developers and Infrastructure must start to talk in the same language through tools,
Optimize the teams based on value streams,
Invest in automation tools and role transformation.
Craig Motlin Sep 01, 2014