BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles Downscaling SAFe

Downscaling SAFe

Bookmarks

 

Whenever one starts to talk about scaling up an organization, it is important to define the areas of growth. What causes the organization to grow, where is it growing, where does it need to grow, how - and why?

In the case of Seamless Payments - a Swedish company behind a mobile payment solution called SEQR - three definitions of growth in two years’ time could be identified.

  • First of all, the number of employees in the company has grown from 50 to 200.
  • Second of all, at the same time the number of countries in which the product is available has increased from 1 to 12 (USA, UK, part of Eurozone, Sweden and Romania).
  • Third of all, the potential number of consumers which could use the product has grown from 10 millions to approximately 600 millions.

Consequently, the challenges which the company faced were not only technical (how to deal with: multiple environments, feature requests coming from different markets, synchronizing work between teams?) but also organizational (how to deal with inevitable change of culture in organization which grew 4 times in 2 years?). The Scaled Agile Framework (SAFe) with custom modifications to it in accordance with Agile and Lean values helped the company to go through this period and prepare for further growth. This article describes the change that was done using a slimmed down version of SAFe that still maintained its core ideas. However, one needs to remember that the change was also possible due to major technical investments in the deployment pipeline, a Lean Startup way of thinking about building products etc., which are beyond the scope of this article.

To better understand the consequences of the change one should look at this from two perspectives: marketing and management. Geoffrey Moore in his book “Crossing the chasm” describes how companies with disruptive innovations (which SEQR undeniably is) may fail at early startup phase. The majority of such companies, according to the research, fails at transforming from early adopters of a product to early majority. Although mobile payments are not in early majority phase globally yet, Seamless is at the moment flying over the chasm by being already present in 12 countries. On the other hand, from a management point of view it is worth understanding Greiner’s growth model. With a fourfold increase of the number of employees the company was growing through direction. However, new markets and new features became so numerous that the employees started to experience a lack of situational awareness and purpose which resulted in frustration. This could be interpreted as an autonomy crisis in the aforementioned model for which a solution had to be found.

Implementation of SAFe at Seamless

The Scaled Agile Framework - SAFe for short - adds a business framework to support agile teams. It addresses the inherent shortcomings of scaling Scrum to large corporations by introducing tools and structure for creating a WIP-limited pull system from prioritized pipelines of high-level requirements to rolling-wave planned release trains.

SAFe is not only a way to scale agile organizations, but it also offers an interesting perspective on how to do it. In particular, it does away with the often chaotic and sub-optimized multi-project environment and creates clear and simple top-level backlogs from which agile teams produce a sprint plan and a slightly longer forecast. In theory, this should give teams a better heads-up on upcoming work, thus avoiding the dreaded "one-sprint planning horizon" often prevalent in agile organizations, and also give the teams their much needed focus and time to do the work properly.

But SAFe was created with the primary intent of introducing agile and lean thinking into big corporations. How can it be applied to a fast-moving small company that wants to introduce structure without unnecessary bureaucracy?

The core principles of SAFe are present in this modified version:

  • The planning horizon covers about three months, being re-planned every sprint planning.
  • A tactical, or program, level is introduced where teams show off how they coordinate their work and what their focus is in the upcoming sprints
  • A strategic, or portfolio level is added where features and epics are funneled into one or more backlogs, which are managed by product managers and product owners.
  • Finally, a technical/infrastructure backlog is created on the top level, managed by the architect who ensures that technical investments are motivated from business features, with business cases if needed.

From previous experience with small organizations, this system does indeed apply to their reality, reducing chaos and waste quite a bit, offering a good tradeoff between reduced waste and a slight increase in administrative overhead. For various reasons, the visualization of the top and mid tier of SAFe was implemented using a set of slightly different tools than prescribed by SAFe, which will be described next.

The backlogs of features, epics and stories were managed in Confluence and Jira, which proved sufficient for distributed teams to pull work from. A blueprint was prepared in Confluence to make it easy to describe ideas, needs and features in a way that could be understood at a glance, with further details available at demand. It was also required to describe early how the benefits of the realized blueprint would be realized and which decisions and lessons learned should be promoted to future releases. An added bonus with Confluence proved to be the discussion threads, often increasing the involvement from distributed experts without the need of extra meetings, meaning that the blueprint could start from an idea scribbled down on paper, or interview notes with users, growing over time into a business case complete with links to user stories, acceptance test criteria, design documentation etc. providing full traceability in a simple format once new features are launched

The overall WIP was managed using a Capacity Matrix. This is a visual board that lists all line functions providing capacity on one axis and projects/assignments/other work withdrawing capacity on the other. Whenever an intersection between a function and ongoing work was being over-utilized, a red signal would alert management that decisions have to be made to help the teams to move forward. Over time, a mindset change started to occur on the managerial level. While management had the clear goal to help teams and not only express their expectations it is also expected by teams to find a solution to a problem on their own (possibly aided by a ScrumMaster) before announcing a need for help. In essence, a red signal means that the teams have exhausted their options and will miss the next goal unless management support is provided.

Capacity Matrix in Google Spreadsheet

By monitoring the capacity matrix, which is updated weekly by team leads and product owners, managers can focus on the right decisions at the right time such as reassigning priorities or cutting scope. This concept does scale well because management can focus on strategic decisions and get involved into operational activities only when it is needed and also avoids the temptation of moving people between projects and tasks. Furthermore, by using this kind of visual planning micro management is avoided as the status and ongoing efforts are always visible in a format that can be understood by everyone. But before even reaching status red in the Matrix, teams have already tried to solve the issue through their primary planning tool - the Tetris board.

Tetris Board in Google Spreadsheet

The Tetris Board - no connection to a popular game with the same name - consists of a number of swim lanes - one per team - divided by time markers in the form of sprint boundaries. Here, teams illustrate through the use of colored sticky notes of different shapes which release or epic they will be working on in a particular sprint, and dependencies between teams are illustrated by allowing the colored notes to "overflow" swimlanes, occasionally resembling “Tetris” blocks. In Scrum terminology, the current sprint represents a commitment and the rest of the board displays the teams combined forecast on how and when they will focus on different types of work. For this case, it was also decided to color-code the type of work, meaning that epics, architecture / infrastructure, risk management and spikes all got individual colors, making it possible for teams and managers to assess the “mix” of upcoming work at a glance.

At Seamless, due to the distributed nature of the company (the Software Engineering department being present in 4 countries) the Tetris Board was first replicated by managers and ScrumMasters on physical boards, one copy at each location. After six months, the Tetris Board was moved online into a Google Spreadsheet document which can be accessed by anyone in the company. Having the physical boards for such a long time with duplicated information did cause some initial complaints over extra work and “old-school tools”, but when the boards were finally moved to a digital form, it was concluded from all sites that the purpose and joint ownership of the boards had not been made as clear had they not been present in a physical version to begin with.

What are the important points of this planning board? As mentioned before, the Tetris Board is updated directly by teams. Thanks to this they gain situational awareness and an understanding of the common goals - lack of these were one of the most significant problems during the fast growth of the company. In addition, people in the teams can, due to their collective knowledge and experience, identify dependencies more precisely between epics being implemented to avoid problems in future. This kind of ability is hard to find in a single person such as a Product Owner, Product Manager or Architect, although this is often expected by an organization and can be wrongly deduced from the agile principles.

Another very useful thing is that the Tetris Board provides a quick way of assessing the consequences of adding unplanned work, or stories going on far longer than expected. Teams and team members can easily see the critical chains of their work, adding buffers and managing risk accordingly.

Scrum and ScrumMasters in SAFe

SAFe uses Scrum on the team tier of the organization. As Seamless had tried Scrum on team level for some time, the concept of working in sprints with planning sessions, synchronization meetings, review and retrospectives were already in place. The role of ScrumMasters is toned down in SAFe compared to Scrum, and implementation of the framework is largely left to management. Coming from a previous good experience of having ScrumMasters act as agile coaches for the entire organization, an approach with dedicated ScrumMasters being authorized to coach the implementation of Scrum on team level was introduced, but the role was also further extended to facilitate the rolling Tetris planning and review. Interviews were conducted with the current ScrumMasters who for the most part were enthusiastic to the proposition of working as full-time coaches. In the end, only one of the current ScrumMasters opted for a technical career instead of a full-time coach position. (In practice, some ScrumMasters did help out with technical issues when needed, without compromising their primary job, but this was never required of them)

Other roles

To facilitate Capacity meetings and Tetris reviews (where the Tetris board is discussed weekly by stakeholders) the role of Program Manager was created. This role ensures that information is kept updated by teams and individuals and that the meetings are kept short and focused. The role of Release Train Engineer as described by SAFe was not used, responsibilities instead being shared between the Program Manager and a System Team (aptly named: “Rollout Team”) headed by a skilled test- and integration engineer. As was the case at Seamless, this was implemented as a trade-off, as it may seem suboptimal to the reader. There are actually two handovers which take place: at first, teams pass their work to the Rollout Team for testing on staging and, subsequently, they pass it on to Operations team to have it deployed onto the production environment. Problems connected with loss of information and context and little feedback from production environment back to developers may arise, something the DevOps approach tries to address. However, the transformation to a DevOps organization requires a continued investment in further cultural change and automation, which is beyond the scope of this article.

To raise the level of technical and integration awareness in teams, the role of Tech Lead was created. As an informal leader, the Tech Lead is responsible for ensuring that teams will pull relevant stories from the architecture runway and Tech Leads bring integration and architecture issues and design suggestions back from the team to the runway. To ensure that learning goes both ways, even when time is short, architects would regularly pull the tech leads together for discussion and learning.

From Projects to Release Trains

It is common for many product development organizations to be stuck in the traditional model of project budgeting and execution. Capacity/resource management is often performed ad hoc, projects being funded when needed, and project managers often rewarded for delivering results on time and budget, without concern for other ongoing work. As we know from W. Edwards Deming, optimizing a single workflow, project or business leads to sub-optimizing the whole system. This very often means that in reality people are pulled between work, which is re-prioritized without much afterthought, leading to a lot of time (and money) being wasted in context-switching, also leading to increased costs from poor release quality, poor design, and increased variability in forecasts.

Introducing a WIP-limited program execution where work is planned in release trains and budgeted according to the number of needed, and possible, parallel trains, brings an end to the costly fire-fighting individual scenario.

The transformation is often contested and difficult in practice, something that also proved true in this case. Initially, project managers opposed the idea, advocating the need for projects to control the introduction of new markets and products. The strategy used was to reduce the focus of a project to all activities surrounding a market launch or a new product launch (including engagement from Marketing and Sales team), providing input to and getting synchronization and forecasts from release trains. This revised scope was initially adopted by Project Managers, and later taken over by Product Owners in order to transfer execution to roles with a lifecycle view of the products and markets.

As far as budgeting is concerned, Product Management (consisting of a Head of Product Management and Product Owners) is responsible for this. They collect requests from all the departments in the company and create a prioritized backlog from which teams can pull work. No specific formula can be given in this article for how important a given epic is. However, things taken into account are: deadlines for entering new markets, CEO’s vision, investor’s expectations, ROI (in which income, attracting new consumers and increasing loyalty of existing consumers play major roles) and effort estimation from the teams. It is worth emphasizing that a partnership of trust between Product Management and Software Engineering has been built which results in scope modification discussions whenever the estimates provided by the teams are beyond a planned budget. It is not uncommon in other companies that engineering becomes a servant to a specific business unit and delivers what is ordered. At Seamless however, mainly due to the aforementioned partnership, empowerment of teams and pull system (obtained via the mechanics of Tetris Board and Capacity Matrix) the Software Engineering department takes an active part in driving the business of the company.

As far as the Release Train is concerned, it brought a necessary cadence to the way the organization works. It was vital during the initial chaos that was caused by the fast growth of the company. However, it is an intermediary step rather than ultimate goal of delivering new features. The ultimate goal is to deliver as frequently as possible so that any release ceremonies (like rollout planning meetings) become pointless as they would have to take place all the time becoming a nuisance eventually. That can be obtained using technical investments stemming from DevOps culture in the organization. Significant amount of attention is being paid at the moment to that change.

Conclusions

The transformation which happened at Seamless can be summarized with one sentence: scaling up is about utilization of knowledge and experience of teams because they deal with the scaling consequences on a daily basis and very often understand it much better than management. However, management responsibility is to create a system which supports this setup, empowers teams and influences the appropriate organization culture.

The system for teams is obtained using tools like the Tetris Board, Capacity Matrix and Release Trains which provide sufficient level of visualization for the whole organization to perform the planning and forecast the future, understand complex dependencies and being able to find an optimal WIP limit not only for teams but most importantly for the whole company. ScrumMasters as Agile Coaches become change agents in the organization and should ensure that the vision is clear on all levels.

As far as empowerment of the teams is concerned, it is obtained by a pull system in which teams pull features to work on from backlog created by Product Management. It is the teams which influence the order of things to be done due to collaborative work on Tetris Board. Meetings on which the Tetris Board is discussed between teams and Product Management (held once a week for half an hour) provide an opportunity to discuss priorities, sync in terms of goals and vision and increase situational awareness. This, in turn, gives teams the necessary context for making better and more conscious decisions every day. Moreover, the mechanics behind the Capacity Matrix put emphasis on teams having the freedom to make decisions and, what is most important, take responsibility for it. On the other hand, it prevents management from falling for the temptation of micromanaging their work and lets them focus on giving enough support for teams to do their job the best they can.

The last factor is nurturing a proper culture in an organization which supports the system and empowerment of teams. There are no easy recipes for this apart from the ideas and principles one should follow. Those ideas are: trust, respect, a common goal and partnership between different departments to achieve this goal. That is why it is so crucial that change agents are mostly recruited from ScrumMasters since they have the inherent willingness and enthusiasm for transforming an organization and helping it improve at all times.

To sum up, the case study of Seamless can be an evidence that small or medium-sized companies can benefit from a scaled agile framework with custom modifications. Moreover, if the company succeeds and grows even more it will, hopefully, be ready to absorb the turbulences connected with it. The future will verify this assumption. However counterintuitive it may seem, the response to scaling up an organization should be empowering teams and creating supportive system for them rather than creating a multi-layer management structure in which a lot depends on single people who very often become overwhelmed in trying to grasp the complexity of the world they work in.

About the Authors

Mikael Lundgren started using Scrum and Lean Product Development in 2001, after experiencing six years of increasing frustration with the contemporary means of directing work in the software industry. A former software developer, project- and development manager, Mikael became a Certified Scrum Trainer in 2006, and later founded Levla AB as a platform to help entrepreneurs and leaders create competitive, creative and sustainable organizations. Examples of companies that have worked together with Mikael in the past to transform their organization include Spotify, bwin, ABB and Assa ABLOY. As an author of books on agile product management and the ScrumMaster profession, Mikael is also a popular lecturer and speaker. He holds an MSc in Computer Science from Uppsala University. Visit Levla and Leanspired to learn more, and follow Mikael on Twitter (@leanspired).

Tomek Pająk (Lodz, Poland) went through several roles in IT: Software Engineer, IT Architect and Engineering Team Lead. At the moment he is Software Engineering Manager at Seamless Payments responsible for services built on top of the core product SEQR - the mobile payments solution available in 12 countries (including USA, UK, Sweden and most of the Eurozone). To obtain competitive advantage of the products he utilizes Lean Startup and Scrum (for product development) and DevOps (for strong IT performance). At the same time he is Coach at Sages helping other companies to improve their businesses by adoption of agile/lean concepts and certain technologies. Tomek received his MBA title from Akademia Leona Kozminskiego and MSc in Telecommunications and Computer Science from Technical University of Lodz. He speaks at international conferences: Agile Lean Europe, Agile Eastern Europe, Atmosphere, Agile By Example. You can reach him at LinkedIn and twitter (@tomekatwork).

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Interesting adaptions

    by Martin Burns,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Very interesting case study, showing that SAFe's operational effectiveness footprint very much covers most Team-of-Teams delivery scenarios, even if it does have a number of features and ways of articulating them that are targeted towards existing large organisations in established industries.

    The visualisations are particularly interesting: I'm going to spend some time studying them and working out how they might work in other contexts.

    Overall, this is fairly orthodox SAFe, which absolutely supports and encourages customisation to meet your own contextual needs (compare to the rhetoric of "ScrumBUT").

    The devolution of authority to those who know content best is absolutely spot on SAFe's messaging to leaders. The principle is to be thoughtful about central/decentral but to understand that the people who do the work generally understand it best. If they also have a very clear understanding of purpose, then the argument is even stronger.

    It's great to have Swedish case studies!

  • Lead and Cycle time

    by Grgic Viktor,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I wonder how long is the actual lead time of items requested by business. The same for cycle time. Also, how long it was.
    I also wonder what exactly is downscaled?

  • Re: Lead and Cycle time

    by Tomek Pająk,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Viktor, thanks for your comment and sorry for the late reply.

    The most significant problem we tried to solve was ability to roll out to new markets fast because that was the strategy at the time. Originally, we launched our product in Sweden in 06.2012. Then, it was Romania on 10.2013. Everyone in the company felt it is too slow and one of the reasons for that was that we grew as a company (in terms of people - 4 times) in a really short time. We felt it most severely in syncing work between Software Engineering, Operations, Product Management, Sales and Marketing departments and when mapping company strategy into operational activities of all of us.

    Then, due to Mikael (co-author of this article) and his experience from previous organizations, we changed the way we work by introduction of slimmed-down version of SAFe with Agile and Lean concepts back in our heads (Scrum on team level, Tetris Board and Capacity Matrix as visualisation tools etc.). What we measured was pace at which we entered markets with our product: Finland (5.2014), Belgium (6.2014), Portugal (12.2014), Netherlands (3.2015), Germany (3.2015), Spain (3.2015), France (3.2015), Italy (3.2015), USA (6.2015) and UK (6.2015). I think the difference is obvious.

    Of course, it's not only introduction of SAFe which helped it happen, however, it was one of the most significant factors to achieve this. Among other things one should mention at least: investment in test automation and deployment pipeline, introduction of Lean Startup concepts (like MVP way of thinking when creating new products) and experiments with DevOps.

    As far as downscaling is concerned, we decided to use this term because Seamless is not a corporation (<200 employees). However, very often people do consider SAFe with a framework for big companies. Hence, in our humble opinion, using a slimmed-down version of it could be described as 'downscaling SAFe'.

  • Re: Lead and Cycle time

    by Grgic Viktor,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Ok, thanks for answering. It is interesting

  • Had the company always used Scrum?

    by Andrew Bennett,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I noticed that you had been using Scrum for some time, but had you done a previous transformation to scrum as well, or were you always using that methodology?

  • Re: Had the company always used Scrum?

    by Tomek Pająk,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Andrew, thanks for your question. Scrum appeared in our company somehow naturally around 2008/2009 and was not a part of long-term strategy but rather an effect of evolution. I think I will never say that 'Scrum transformation is completed' as it is hard to define success criteria ultimately. Should it be gauged by how mature a role of Scrum Master is in an organization (Scrum Mom or Agile Coach?) or perhaps by how fast and accurately we are able to deliver business value? I would say that everytime we think we are pretty good at 'doing Scrum', something changes internally or externally and we realize we have flaws in our value delivery chain and try to react accordingly. Perhaps this is a definition of succesful Scrum implementation: that we are able to observe fast what we do wrong and try to adjust the system - continuous improvement.

  • Re: Had the company always used Scrum?

    by Andrew Bennett,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Thanks for the response. I agree that Agile is a journey with no end, but I was curious as to when you were starting to use Agile methods, and then when you transitioned to the scaled down SAFe.

  • Training?

    by Matthew Koch,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hi! Thanks for this article. I realize it's several years later at this point but I am wondering what training you felt was appropriate for this scale, and what you ended up pursuing for teams. Normally SAFe prescribes the "Implementation Roadmap" with several points along the way, indicating trainings at each stop - how did you align or diverge from that? What trainings, if any, were most valuable?

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT