Large Scaled-Scrum Development Does Work!
Agile Scrum development is nothing new or extraordinary. But when putting a hundred professionals from related development and product areas all in the same boat to develop a product, it becomes a challenge — a task we at Ericsson Eurolab in Herzogenrath, Germany (near Aachen) have tackled and conquered with the help of kaizen and other adjustments to agile practices. We can proudly claim that large-scale Scrum development DOES work!
Introducing Ericsson Eurolab
When you make a call or wirelessly browse the Internet, you most likely use an Ericsson solution. About 40% of the world’s mobile traffic uses networks that are built by Ericsson. The 10 largest operators in the world use our solutions.
Today, Ericsson employs roughly 118,000 people and is the fifth-largest software company in the world. Ericsson Eurolab, an ICT development center near Aachen Germany, has made key contributions to GSM, UMTS, and LTE network standardization and development, and drives research in multimedia technologies. Some 510 employees currently work at Eurolab.
In order to build a successful product, we need a hundred people to work as one team. Having an end-to-end functional team is a challenge not only because of the number of agile teams involved but also because of the geographical distribution across several countries. We nevertheless have managed to successfully unite product-development and operations teams in the same boat, allowing all members to embrace the same vision, to keep an end-to-end view, and to avoid any silos.
This boat, or perhaps “ship”, carries a crew of a hundred professionals who are responsible for product development, network integration and verification, operation and business, standardization, global services, and sales support — and a most important passenger, the customer, to the goal.
What do we do different than past large-scale Scrum development projects? How can we reach flow efficiency, shorten the delivery cycles and keep everybody focussed on a common goal with so many people aboard?
Key factors to success
Communication and collaboration
Our one, large team is composed of several Scrum, kanban, and agile teams: it is a team of teams!
Regular Scrum meetings occur more or less by the book, with team stand-ups, retrospectives, and grooming sessions. Depending on the team’s maturity, each agile team might adapt the number of team stand-up meetings to their needs.
Furthermore, there are diverse Scrum of Scrum (SoS) coordination meetings on regular bases like:
- SoS meetings for concrete issues:
- continuous integration on feature level;
- continuous integration on network level; and
- code quality and design expertise.
- architectural and software-anatomy SoS:
- requirement/features workshops;
- grooming sessions; and
- regular architectural SoS.
- ad hoc SoS on request, to coordinate issues like:
- overall retrospectives or kaizen events;
- competence sharing meetings; and
- re-organizing new seating.
Overall retrospectives are essential for our kaizen culture, so it is crucial to keep them alive, and on a regular basis! With a crew this size, a continuous-improvement mindset requires frequent checks of “what can we do better?”
The key meeting that keeps the entire crew synchronized and working as one team is the daily all stand-up meeting.
This is a daily face-to-face gathering for the entire crew – all 100 or so people. This meeting uses video conferencing so that teams can participate from any location. Coordinating this number of people every day at the same time to guarantee an effective meeting requires forceful moderation. Our setup does not include ScrumMasters; the moderator role is rotated among the development teams, so that each week a different team takes on that responsibility. It’s the best way to build skill in moderating! Information packages about concrete issues regarding sales news, line strategy, or celebrations are scheduled on a specific weekday.
General development and operations coordination takes place during this daily all stand-up: work on bug fixes is distributed among agile teams, hardware and software issues are highlighted, any problems are considered, and decisions that affect the whole crew are taken.
Transparency and trust are important when navigating towards a common goal: there are no secrets! Operations and marketing colleagues keep the crew informed about contracts, competitors, market trends, demos, etc. Customers have even visited our crew stand-up meetings a few times, joining us to celebrate successful deliveries or sitting with the teams to give us feedback on our product.
We do not make the daily stand up meeting mandatory, instead we make it attractive. We recommend structuring it, moderating it, and keeping it short — remember, the attendees are standing! Improving this gathering should be part of the overall retrospective.
In our development concept, any customer need is a feature. Any feature with a go-ahead from product management goes into the product backlog, which means that we do not have a tollgate decision per release but a go-ahead decision per feature. Several features are developed in parallel and each follows its own development flow. Every few months, we combine completed features into a new release package that we prepare for the costumer.
Feature development starts with a workshop at which representatives of all teams, included product management, gather in order to reach a common understanding of the new feature, keep the view on the business purpose, and to outline the solution to be developed.
The following paragraphs highlight aspects of our development flow, which is key to our success.
We have a single overall product backlog for the entire product development, including all user stories and spikes, from feature concept to product deployment. The product backlog distribution is completely based on pull strategy. All product owners (POs) work together on backlog prioritization and content so that the development teams perceive the POs as one person. For this purpose, all POs get together at a daily PO alignment meeting and distribute tasks related to their role, discuss any backlog items, hot issues, new user stories, etc. Before each sprint, after the grooming session, the POs also gather for a backlog prioritization meeting. Considering how geographically distributed our POs are, PO coordination is an impressive achievement within our large-scale development process.
The development teams apply Scrum methodology whereas the solution teams and the product-introduction teams use scrumban boards. Scrumban teams pull user stories anytime during the sprint, as unexpected, high-priority customer user stories may arrive. However, all teams keep the same sprint rhythm. For scrumban teams, that means that the demos also cover status presentations of ongoing user stories.
Management enthusiastically supports competence development and continuous improvement of technical skills: any agile team is welcome to take on any user story, regardless of the competence required. Having, for example, members of the software-upgrade team taking development or solution-integration user stories strongly encourages competence building in different areas.
Furthermore, we practice temporary rotation of team members in order to facilitate people growth. For example, we challenge our agile developers to spend some sprints working within solution-integration or upgrade teams — a perfect way to create end-to-end-capable teams. We continuously seek a balance between increasing the competence of our crew and agile team effectiveness: people growth is an important asset but should not hamper product development.
Continuous integration is the foundation of our agile development, as we integrate code generated from different features into the base software. We not only integrate new code daily and have test suites with different priorities running overnight, but we also extend our automated load tests for non-functional testing of our product: stability tests, high-load tests, overload tests, etc. This way we can check the robustness of our generated code on a daily basis.
We also use continuous integration to test the interaction between our application code and new platform releases, so that we can foresee any compatibility issues.
Our decision to run automated continuous integration on different levels has significantly improved our integration time and drastically reduced the number of faults found by the customer.
Scrum and scrumban teams develop, integrate, and verify ongoing features. Software-upgrade assignments are included and prioritized as user stories in our overall product backlog in order to make sure that any issues and tasks related to software upgrades do not cause bottlenecks towards the end of the development flow. This way, through the close collaboration of development, upgrade and solution-integration teams during product development, we reach significant flow efficiency in our process.
During the solution and integration tests for network security, vulnerability, robustness, characteristics, etc., our customers are also welcome to join and be part of the product verification.
Acceptance testing takes place at the customer site, first in a customer’s lab and then in the customer’s live network. At this stage, we follow a cross-functional culture, so that any agile developer can participate in the deployment of new features and experience the customer’s passion.
Risk and quality: Spikes are welcome!
Within our agile culture, risk management and code quality are no longer administrative tasks. They are present in our development from the very first line of code.
In our crew, we do not have any role dedicated to quality. Each team member takes responsibility for the quality of our product.
Spikes and refactoring are part of our daily work! Spike user stories are included in the overall product backlog to trigger investigations into issues for which we do not have enough knowledge available for the development.
Code refactoring like renaming or code clean-up is done by everyone — everyday. Any member of the crew can request a rework of unreadable or badly structured bigger chunks of code, in order to minimize technical debt. We keep a prioritized list of technical-debt issues, which are constantly flowing into our overall product backlog in form of user stories. Such items can improve usability, robustness, and, most of all, maintainability of the code.
Having the customer on board is essential!
Close interaction with the customer is one of our cornerstones and has been a significant factor in our success. Our customers are welcome during the entire flow of product development: we allow some customers to directly influence our overall product backlog, we demonstrate existing and new features per customer demand, and customers are invited to join the test labs within the development process. Any feedback given by customers at demos or test labs flows into the development: this way we can identify early on whether the developed functionality will lead to true customer value.
Our product-deployment support provides support to any customer; a kanban team composed of dedicated deployment managers and team members from the development teams drives this initiative.
Reaching one port… and planning for the next
We have successfully managed to create a geographically distributed, end-to-end functional agile team of around a hundred professionals to deliver a product with best quality and highest customer satisfaction.
We got a hundred people to work as one team, based on an end-to-end cross-functional team setup, so that colleagues in product management, product introduction, and sales and marketing work closely with software developers. We share a single backlog for the entire product development and develop features in parallel. We strongly encourage team members to build competence in any related product area. Collaboration between development, upgrade, and solution-integration teams ensures significant flow efficiency. And the most important person, the customer, is always welcome to join at any stage of our development flow.
Kaizen, or continuous improvement, in form of:
- overall team-of-teams retrospectives,
- flow efficiency,
- continuous learning culture, and
- continuous customer feedback
has significantly helped our organization to constantly improve our agile development. Having tackled these challenges, we will surely further evolve our large-scale Scrum development success story.
About the Authors
Almudena Rodriguez Pardo, born in Bilbao, Spain, studied computer science at RWTH Aachen and started working for Ericsson in 1995. She worked several years as a software developer and quality coordinator in different development departments within Ericsson Eurolab. Since 2013, Almudena has worked as a Scrum developer and agile coach, involved in the agile deployment at Eurolab. She is committed to the agile community, and with Norma had the pleasure to speak at ScanAgile 2015 and London Agile Tour 2015. This year, she will participate in the Delivery of Things World 2016 in Berlin.
Norma Acevedo Lopez was born in Santillo, Mexico and started at Ericsson in 1990 as a software developer and software-development trainer, afterwards working for several years as a process manager. Since 2010, she has worked as ScrumMaster and agile trainer, and was intensively involved in the agile transformation at Ericsson Eurolab. At the beginning of 2015, Norma took over the role of product-introduction manager. She is committed to the agile community, and with Norma had the pleasure to speak at ScanAgile 2015 and London Agile Tour 2015. This year, she will participate in the Delivery of Things World 2016 in Berlin.
Big thanks to our colleagues Joerg Koettig, Hendrik Esser, Maria Larsson, Maria L. Medrano Cuadro, and Tyrone Castelanelli for their excellent support and advice when writing this article.
Robert Van Dell II
Almudena Rodriguez Pardo
Newcomers are distributed among the existing Agile teams, depending on the competence and background they bring along.
Pairing with experienced developers is the best practice to get the new employees involved in our activities:
In these cases we look for user stories including some easier tasks, which can be taken by the newcomers, with the supervision of experienced colleges.
Great article to get a lot out of it!
Mehmet Metin Altuntas
Re: Great article to get a lot out of it!
Almudena Rodriguez Pardo
It is important to follow up why user stories are not finished within the corresponding sprint: could be external impediments, or underestimation of user story size, or competence issues within the team… and it belongs to kaizen culture to observe if there is any pattern, which should be improved.