Key Takeaways
- A successful strategic change in a company must take the organizational culture into account
- A DevOps transformation can be one of the strategic initiatives to build a high performance organization
- You can measure and visualize organizational culture with the Competing Values Framework
- Adding a culture perspective to their DevOps transformation helped Seqr understand why certain initiatives did not work and how they should lead them
- For a change to happen it is crucial to create new valid stories which show different approach to already existing problems
In the '2016 State of DevOps Report' the Westrum Model [1] of organizational culture is proposed. It focuses on information flow, high cooperation and trust as predictive factors of DevOps success in a company. It is a perfect future state design tool which, however, tells little about where your company is at the moment. Moreover, it does not suggest how to influence an organizational culture and in which direction it should change.
This article proposes an alternative (perhaps complementary) model described by Cameron and Quinn [2] which takes into account whether an organization focuses on flexibility or stability and whether its operation is internally or externally focused. I will show the results of this model being applied in Seqr - a company I work in (which is constantly under DevOps transformation). My experience shows that the starting point (the culture that an organization has) determines the way to influence organizational culture to build a high performance organization. One reason why an alternative to Westrum's model was chosen is because of the available tools to measure and visualize culture.
Why organizational culture matters
In the past decades there have been four major change initiatives taking place in companies: strategic planning, Total Quality Management, reengineering and downsizing. Their aim was to increase economic effectiveness. However, 75% of them either failed or produced problems serious enough to threaten survival of the company [2]. Neglecting the organization's culture was the most frequently cited reason for failure.
This is confirmed by my observation when witnessing hierarchical and bureaucratic organizations trying to adopt Scrum on a team level without influencing the environment which supports the change (executive level). In such organizations people are frequently taught by agile coaches about courage, focus, commitment, respect and openness [3]. However, the system they work in very often promotes politics, blame games and escalation paths, which are in direct contradiction to the aforementioned values.
Organizational culture, however intangible it may seem, should not be underestimated by leaders of an organization. Culture may either support or doom strategic initiatives of a company. As proved later in this article, a DevOps transformation can be a strategy for building high performance organizations.
In Seqr, we refer to our organizational culture frequently. For example, we considered our organizational culture when selecting a person for a managerial role to help us influence a change in a desirable direction. Moreover, we took into account the cultural aspect when we had to regroup internally to meet new business needs defined by emerging opportunities ahead of our company. In general, awareness of our organizational culture suggests how we should do things once a certain business decision is made.
What is DevOps transformation and how to define a high performance organization
What is a high performance organization? According to 2016 State of DevOps Report [1] high performance organizations are described as the ones which deploy on demand (multiple deploys per day), where the time from code commit to code running on production is less than one hour, the mean time to recover takes less than one hour and the change failure rate is less than 15%. High performers do better at both throughput and stability, which proves that it is not a zero-sum game. This can be achieved because the techniques behind Continuous Delivery enable high quality products and more frequent deployments at the same time. [4]
According to my experience, a DevOps transformation is a comprehensive effort existing on all levels of an enterprise to build a high performance organization. One of the ways to start a DevOps transformation is by teaching an organization to build its product iteratively and incrementally. Usually, it means adoption of Scrum in a company (not only among engineers) to deliver business value every sprint. Frequently, there is a moment when companies realize thanks to Scrum that their main challenge is that they are unable to deliver at a pace needed for competitive advantage on the market.
Seqr defines high performance by examining how much time a rollout takes and how often it happens. There is a rollout train departing at regular intervals of time (in our case approximately once a week) which delivers any feature, improvement or bug fix completed after the previous rollout. Teams dedicate part of their time in a sprint for improvements to decrease the rollout time. If the rollout time becomes small enough, the rollout train is replaced with deployments on demand. Once we achieve one rollout a day, we will consider if we as a company are satisfied or if we should put more engineering effort into rolling out every development branch commit.
I agree with Jez Humble [4] that there are two main blockers why Continuous Delivery (which is a fundamental concept of DevOps) does not work: inadequate architecture and inadequate culture. The former means that very often testability, deployability and monitorability are not of primary concern in a system architecture. The latter stands for employees not feeling the need to deliver things more frequently, collect feedback faster and improve their work continuously. Topics concerning architecture transformation are beyond the scope of this article. Let us focus on how to measure culture and define the potential direction of change so that it supports a DevOps transformation.
Competing Values Framework explained
The Competing Values Framework [2] defines four types of culture: clan, adhocracy, market and hierarchy. All of them are characterized by distinctive values drivers, assumptions about sources of effectiveness, dominating types of leadership and orientation.
Clan culture revolves around collaboration. Its main assumption is that satisfied and committed employees are the source of organizational effectiveness. The fact that employees participate and are involved in decision-making processes in a company leads to their commitment and empowerment. Leaders in organizations having a clan culture are team builders, mentors or coaches helping people individually and teams to move forward.
Adhocratic culture is very often associated with entrepreneurship and innovation. What matters are cutting-edge ideas and creative solutions to problems which produce new opportunities. Leaders in this kind of companies are mainly innovators and visionaries who can inspire others.
Market culture pays attention to achieving goals, winning with the competition and increasing measurable outcomes like market share or ROI. The main assumption is that competition, both internal and external, are sources of productivity which leads to effectiveness. Leaders in market culture are people who enjoy competition and can deliver results despite many obstacles.
Hierarchical culture praises predictability, timeliness and efficiency. Effectiveness is obtained via control whose aim is to eliminate redundancy and waste. Hence, leaders are generally good organizers and coordinators who monitor if process is followed and rules are not being broken.
The four culture profiles can be summarized in Table 1.
Table 1: Four culture types according to Competing Values Framework, source: [2]
Apart from clear differences between the four cultures one needs to understand the two differentiating dimensions dividing those cultures. One of the differentiating criterion of effectiveness is whether companies are dynamic, agile and adaptable (clan, adhocratic culture) or predictable and stable (hierarchy, market). The former ones are considered effective if they can transform themselves to follow fluctuations of the environment they operate in. Many startups and NGOs possess such features. The latter ones are considered successful if the results they obtain are durable. Which is typical for government agencies or universities.
The other differentiating effectiveness criterion is internal orientation (clan and hierarchy cultures) versus external orientation (adhoc and market cultures). The former ones focus on integrity and unity as source of effectiveness. Whereas, the latter ones pay attention to the outside world and refer to competition and market to become successful.
Those two dimensions (flexibility versus stability and internal orientation versus external orientation) intersect to create a set of unique cultures. For example, clan culture represents flexibility and internal orientation. On the other hand, market culture stands for stability and external orientation. The key thing is that different cultures can coexist in an organization. For example, in a typical enterprise it is possible that the sales department has market culture, GRC department (governance, risk and compliance) a hierarchical culture, R&D department an adhocratic culture and Human Resources department a clan culture. According to my observations, this divergence in cultures may cause friction in an organization which may result in decreased effectiveness as energy of people is wasted in pursuing conflicts. Awareness of different cultures and empathy among leaders may help an organization to take advantage of such diversity and manage it in the right direction.
To measure what culture an organization has one can use a survey proposed by Cameron and Quinn [2] as shown in Figure 1.
Figure 1: The Organizational Culture Assessment Instrument, source: [2]
Employees are asked to answer six questions concerning: dominant characteristic of a company, organizational leadership, management of employees, organizational glue, strategic emphases and criteria of success. They have 100 points to distribute among answers A, B, C or D for each question. Answer A stands for clan culture, B for adhocratic culture, C for market culture and D for hierarchy culture. The column 'Now' represents how the company is perceived at the moment by a person filling out the survey. The column 'Preferred' represents what company should be like to become successful in future according to surveyee. The results for answer A, B, C and D are averaged for all questions and presented as a plot depicting a unique culture profile for a company.
Case study for Seqr
In Seqr members of Software Engineering, Operations and Product Management departments filled in the survey. Altogether it was approximately 60 people.
The structure of the company is as follows. Software Engineering consists of teams and management. The teams are cross-functional and consist of Software Engineers, QA Engineers and System Administrators. Their goal is to take a prioritized backlog prepared by a Product Owner and implement new features end-to-end – from an idea up to it being deployed on production and monitored in terms of business metrics and bugs. Full-time Scrum Masters/Agile Coaches are, on one hand, members of the teams and, on the other hand, part of the management team. They work with the teams but also with the remaining part of the organization like Product Management, Customer Service, Marketing or Sales. There are managers in Software Engineering whose responsibilities are vast: ranging from recruitment, selection and socialization of new employees, being people manager (figuring out if it is possible for people to align their personal goals with business goals of a company), to bridging the gap between technology and business worlds in the company. The Operations team (also known as Site Reliability Engineering team or SRE) consists of System Administrators who are mainly responsible for site reliability and environment provisioning; they are the backbone of the company's operations. The Product Management team consists of Product Owners who manage stakeholders (both internal and external) and map their requirements into a backlog available to the teams in Software Engineering. More details regarding Seqr (formerly known as Seamless) and its evolution can be found in the previous articles written for InfoQ: [5] and [6].
Separate culture profiles have been prepared for Operations (Figure 3) and Software Engineering (Figure 4) with a clear distinction between a team and their leaders.
Figure 3: Culture Profile for Site Reliable Team and their Leader
As one may observe the Operations team is fairly satisfied with the culture they have. They perceive it as mainly clan culture with a mix of adhocracy and market culture. However, their leader sees it differently. According to him there is a dominating clan culture in his team with a minor factor of hierarchy and market culture. The key thing to understand is that one should not try to find truth using those plots. They represent different perspectives which should be respected and understood.
An important conclusion based on the plot is that the leader of the Operations team does not see a need for introducing an adhocratic culture which is typical for experimenting. It is perfectly reasonable because the Site Reliability Engineering team is mainly responsible for smooth operation of the infrastructure which the whole company relies on. The domain the team operates in is susceptible to regulations stemming from financial regulations or imposed on us by third-party companies which rely on our product. Hence, it should not be assumed that DevOps transformation will originate from this team as - by definition and based on their culture profile – it is not their primary concern.
Figure 4: Culture Profile for Software Engineering department (Teams, Scrum Masters and Managers)
As far as the Software Engineering department is concerned, the good news is that people in teams are satisfied with the culture they have – a mix of clan and adhocratic culture with a minor existence of market culture. Scrum Masters/Agile Coaches share a similar perspective to teams. Managers see things differently though. They perceive a culture with a dominating clan factor and feel a need of changing it towards an equal mix of clan, adhocracy and market culture.
The difference in perceiving the current culture between the teams and the management was discussed. From the managers' point of view, Seqr is a fintech company operating on a highly unstable market. Threats may come from different directions: a regulator, a direct or indirect competitor, or by changing habits of our customers. Only high performance companies which are able to change direction fast enough can survive on such a market. It would not be possible if our product was not developed incrementally (Scrum) and if we were not able to deliver frequently (eg. once a week). Hence, in Seqr we believe that the DevOps transformation is one of the answers for building a high performance organization.
DevOps transformation in Seqr
Establishing Continuous Delivery is one of the fundamental milestones of a DevOps transformation and Seqr is no exception to that. Neglecting cultural aspect for now, achieving maturity in Continuous Delivery requires changes in design and architecture, build and deploy, test and verification, and information and reporting [7]. According to my observations as a consultant, I have never seen a company with the very same delivery pipeline. Moreover, I frequently see experts arguing what technological changes should be introduced to set up successful Continuous Delivery in a company. What we frequently observe in Seqr is that there are many solutions for the same problem. The fact that the practice of doing this emerges (instead of being well known upfront) suggests that one deals with a complex domain as defined in Cynefin [8]. The strategy of dealing with complex domains is by executing probe-sense-respond iterations frequently. What we have learnt is that experimenting is required as there is no evident best or good practice defined for the unique context of our company. Experimenting is typical for adhocratic cultures.
For a company like Seqr it is crucial to understand when market changes or new opportunity emerge. People in the company need to be curious about customers, competition and market. This kind of focus – ability to relate oneself to a business surrounding – is typical for a market culture.
As observed in Lean Enterprise [9] an organization can move fast at scale if it consists of small, decentralized and autonomous teams. In such an organization the principle of subsidiarity applies: 'by default, decisions should be made by the people who are directly affected by those decisions. Higher levels of bureaucracy should only perform tasks that cannot be performed effectively at the local levels'. The foundations of such an approach lies in confident teams who are empowered to experiment and make decisions concerning architecture etc. Our experience in Seqr shows that a clan culture supports that level of confidence. There is no single entity in an organization which makes crucial technological decisions – it is always a group effort. It may take more time in comparison to a decision made by, for example, an architect alone. However, we think it is more valuable if people who are affected by a decision can influence it, not resist it and support it, since it was their decision.
Eventually, we arrived in Seqr at a desirable culture profile tailored to the context of our organization. It is presented in Figure 5.
Figure 5: Desirable culture profile in Seqr supporting DevOps transformation
Two things need to be stressed out. Firstly, this is not a universal culture profile for DevOps transformation. It is what we believe in Seqr will help us be successful. That is why it should not be copied. However, the reasoning behind it is valuable to other companies and their contexts, hopefully.
Secondly, culture belongs to complex domain (Cynefin). Hence, 'cultural change is an evolutionary process from the present, not an idealized future state design' [10]. It means that the cultural profile for DevOps transformation in Seqr should be treated as a direction towards which we wish to change and not a precisely specified goal.
How do we perform the cultural change? It is naive to think that managers can change the culture in an organization on their own. They can influence it at most. Since culture is an emergent property of interactions one can influence it in several ways [10].
If one listens to stories being told in a company, one can decode patterns of dealing with reality and coping with problems among people. Hence, for a change to happen it is crucial to create new valid stories which show different approach to already existing problems. When it is done, they become a reference point in discussions and provide new set of solutions producing new set of behaviours and, thus, changing the culture.
The beginning of DevOps transformation in Seqr originated with an experiment in which two teams became responsible for not only developing a solution, but also delivering it to production and maintaining it. This experiment proved to be successful and encouraged us to repeat it with all the teams in the company. The results was the reorganization of teams in the whole department by making them truly cross-functional (Software Engineers, QA Engineers and System Administrators being part of one team). Moreover, we took advantage of people being fed up with lengthy meetings trying to solve a certain problem - very often they were ending up with no solution because of complexity of problems. Hence, we suggested to try one of the sensible solutions which were proposed during the meeting without discussing it too much. The only condition was that we would get back to the outcomes after two weeks with sincere intention of measuring the results. It would help us see if the experiment succeeded or failed.
It appeared, that those experiments (typical for adhocratic culture, by the way) helped us to improve our work faster as we were able to take actions more frequently instead of being stuck in endless discussions searching for perfect solution to a problem. Thanks to this approach we introduced the following technological concepts into our everyday work:
- containerization (Docker) of parts of the architecture,
- heading towards uniformity of environments (development, staging, production),
- introduction of feature toggles and canary releases [11],
- monitoring of the production environment using Appium on real mobile devices , etc.
Those experiments have also led to changes in the way we deliver things:
- we stopped starting and started finishing features [12],
- we introduced WIP limits when implementing sprint goals,
- we separated releases from rollouts,
- Software Engineering teams took over application monitoring from the SRE team to shorten the feedback loop,
- we introduced blameless post-mortems in case of significant failures on production environment, etc.
Another step we took towards influencing cultural change was by managing natural constraints via relaxing and increasing them. As an example, the aforementioned two teams had to take care of the full delivery process: development, testing, deployment, monitoring unlike other teams in Software Engineering who were responsible for development only. The true reason was that there were not enough System Administrators in the Operations team to help them with delivery and monitoring. Hence, teams reacted accordingly and proved that it is possible to create a piece of software which is testable, deployable and monitorable without a need of handovers between specialized teams.
Since culture is an emergent property of interactions, one needs to manage them as well. As an example, during the socialization of new employees we mix them with employees that hold the desirable culture. Hence, the new person initially learns the desirable patterns of coping with problems and interpreting the reality. This is particularly important for the Polish culture (Software Engineering, Operations and Product Management are located in our office in Lodz, Poland; the author of this article is Polish). Because of low indulgence Poles have a tendency to cynicism and pessimism as a way of dealing with uncertainty of running a business [13]. This tendency of national culture may be destructive if left alone.
The ability of identifying desirable culture pockets [14] popping up in a company is crucial as well. For example, when teams took over the responsibility of application monitoring on production from the SRE team, people in teams saw two solutions. Some of them asked management to appoint who should be on duty. Others said that whoever takes care of monitoring is a team's decision. The former solution asks for hierarchical culture (leader being a coordinator), whereas the latter requires clan culture (leader being a facilitator supporting the team in a decision). According to our desirable culture profile, management decided to support the latter initiative of solving the problem. By doing so management influences culture change towards clan direction.
Conclusions
DevOps transformations are sometimes limited to implementation of certain technologies and practices. By neglecting the culture in our approach initially, we experienced resistance among people to the introduced changes. This, in turn, resulted in the outcomes which were not satisfactory (eg. low deployment frequency). We observed that a natural engagement of people towards continuously improving the delivery pipeline was present only among some of engineers in our company. Once we added a culture perspective to our DevOps transformation, we started to understand why certain initiatives did not work and how we should lead them.
To build a high performance organization via DevOps one very often needs to change the organizational culture. According to my experience, it is one of the most difficult challenges during the transformation. Firstly, it requires leaders who understand how to influence culture as a complex adaptive system. Dealing with such intangible matter like culture requires a wide variety of soft skills among leaders to interact with people and teams. One needs to be able to merge this soft-skill world with engineering world and recognize how those two worlds complement one another. Eventually, the whole organization needs to be aligned in terms of goals and understanding value behind frequent experiments, short feedback loops, lack of competence silos and shared (not diluted) responsibility, etc.
One may get the impression that organizational culture is a solution to all the problems with DevOps transformations. Certainly, it is not. It is a cornerstone which either amplifies or dooms strategic initiatives in your company. If change in an organization is done with a culture in mind, most probably culture will reinforce the efforts.
Culture is one of the reasons behind the results we obtained. However, it would not have been possible without people (not only engineers, but also Scrum Masters), their expert knowledge and experience, to execute DevOps transformation. The results are presented in Figure 6.
Figure 6: Rollout statistics during DevOps transformation (time span: approx. 1 year)
Each horizontal block in Fig. 6 represents a step in the rollout procedure. Their meaning is not described intentionally in order not to reveal technical details of our production environment. What is important, however, is that the average rollout time decreased significantly from approximately 80 hours to 5-10 hours. Due to such improvement, rollouts have become more frequent. Moreover, they became more stable and thus predictable in terms of how much time they may take. It is essential as a certain degree of predictability is required from Scrum teams regarding their commitment after the planning session. Rollout is responsibility of a Scrum team and may influence other Sprint goals.
There are two main reasons why some blocks disappeared with consecutive rollouts: (1) simplification of the rollout procedure (including the architecture) and (2) automation of manual processes. As far as simplification is concerned, the rollout process was redesigned to reduce the number of steps and, thus, reducing its complexity. Automation of manual processes, on the other hand, applied mainly to testing and deployment. The failure rate has been kept low throughout the whole transformation.
As mentioned earlier, we are still under DevOps transformation aiming at improved rollout frequency and rollout quality. Hence, the process is still ongoing, but the direction we decided to take seems to be promising.
Bibliography
[1] Puppet, Inc., DevOps Research and Assesment, 2016 State of DevOps Report
[2] Kim Cameron, Robert Quinn, Diagnosing and Changing Organizational Culture: based on the competing values framework, Jossey-Bass, 2006
[3] Dave West, Updates to the Scrum Guide – The 5 scrum values take center stage
[4] Jez Humble, Continuous Delivery Sounds Great But It Won't Work Here, Lean Agile Scotland 2016
[5] M.Lundgren, T.Pajak, Downscaling SAFe
[6] T.Pajak, DevOps at Seamless: The Why, How, and What
[7] A.Rehn, T.Palmborg, P.Bostrom, The Continuous Delivery Maturity Model
[8] D.Snowden, M.Boone, A Leader's Framework for Decision Making
[9] J.Humble, J.Molesky, B.O'Reilly, Lean Enterprise: How High Performance Organizations Innovate At Scale, O'Reilly Media, 2014
[10] D.Snoweden, OMG, it's culture change time
[11] D.Sato, CanaryRelease
[12] H.Kniberg, Focus (or Stop Starting, Start Finishing), Agile By Example 2016
[13] G.Hofstede, National Culture
[14] P.Brodzinski, Culture Pockets
About the Author
Tomek Pająk (Lodz, Poland) has held several roles in IT: Software Engineer, IT Architect, and Engineering Team Lead. At the moment, he is Software Engineering Manager at fintech company SEQR which creates its own product- a mobile-payments solution available in Europe and USA. To obtain competitive advantage, he uses Lean Startup and Scrum (for product development) and DevOps (for strong IT performance). He is also a coach at Sages, helping companies to improve their businesses through the adoption of agile/lean concepts and certain technologies. Tomek received his MBA from Akademia Leona Kozminskiego and a MSc in Telecommunications and Computer Science from Technical University of Lodz. He speaks at international conferences such as Agile Lean Europe, Agile Cambridge, Agile By Example, Agile Central Europe, DevOpsDays, Agile Eastern Europe, Atmosphere etc. You can reach him at LinkedIn and twitter (@tomekatwork).