Scaling Mobile at XING: Platform, Framework and Domain Teams
Dedicated to Piet.
And others who just do it.
Here at XING we've been having quite an exciting challenge.
Learning how to scale mobile development in a way so that as many teams as needed could contribute to development of mobile apps (on both iOS and Android platforms) and at the same time keep the apps consistent, stable and shiny... That’s not a thing you can do by a book (is there actually a book on this subject?).
So we had to come up with our way of solving the complex matter.
This paper is envisioned as a quick-read guide of our last years’ journey. It summarizes all key decisions and structural changes we made in order to enable the scale on mobile from 2 to 10 teams.
If your company’s next challenge is to take mobile development to the next level, you might find some of the ideas listed below worth trying.
XING is the social network for business contacts. As of June 2015, it has around 9 million users in its core German-speaking market. XING is a platform where professionals from all kinds of different industries can meet up, find jobs, colleagues, new assignments, cooperation partners, experts and generate business ideas.
The following diagram depicts a simplified structure of development teams and their dependencies that were in place when we started the transition.
Illustration: before scaling
We had a dozen of what we call ‘domain teams’ which were owning functional parts of our platform - www.xing.com and related services. There was normally a team owning each of the domains: profile, start page, groups, messages, jobs, news feed - to name a few. Each of these teams was fully responsible (and hence stuffed cross-functionally) for designing, building and maintaining features in its domain. So for instance the Groups team had a dedicated Product Owner, UX and visual designers, front-end and back-end developers, test engineers. That allowed the team to take care of all needs of the Groups product… Unless it had something to do with mobile.
If it was the case, then the Product Owner for Groups had to engage with the corresponding Product Owner for Mobile (iOS or Android, or both) and help plan development of the Groups-related features on mobile. Not as easy as this might sound - consider a dozen of teams wanting their stuff to “go mobile”... Eventually we ended up with long queues and waiting times.
Such process had its historical roots: back in the days when XING was found and the first platform was built, all users were web users (ah, good ol’ times), and the new mobile thing was yet to come. So the approach of having multiple strong cross-functional teams for the web and several app teams for the mobile platforms sounded reasonable. It also worked.
By the time we started the ‘mobile scaling’ initiative described in the next chapters (beginning 2014), we had already over a dozen of domain teams (each with its own needs, ideas and roadmaps). Also mobile usage was rapidly claiming to take a half of the overall platform traffic. It started to become more obvious day after day that in order to unleash the locked potential, we had to change the way we were approaching mobile development.
(Our internal name for the initiative was ‘Unleashing Mobile’. In 2015 we are already talking about unleashing data science and API. The name definitely sticked)
By the way, this kind of a challenge is quite common for companies that have been started in the pre-mobile era (so-called "desktop companies"). And XING is a perfect example here. We have excelled in scaling web product development at the enterprise level with thinking adopted from Agile, Scrum, Lean UX. We’ve been applying internal open-source model, continuous delivery, extensive test automation and many other good practices of the days... And now it was the time to embrace mobility.
So what did we change? Roughly speaking almost everything, but it all boils down to the following five strategic steps:
- Created sense of urgency and rapid environment for experimentation.
- Changed our apps’ architecture to support modularization.
- Restructured our existing mobile teams and split them into smaller work groups with sharper focus on a) platform development and b) infrastructure support.
- Onboarded new mobile developers and seeded them to other domain teams throughout the company.
- Introduced release trains and other means of release coordination.
There are definitely numerous ways on how to bring mobile development to the scale we wanted. Though looking back I can confirm that the success of the overall initiative is based on success of each of the mentioned strategical steps.
If we would do the transition again, we would repeat these steps, though of course specifics of implementation would likely differ based on the context.
1: Managing for Innovation
Most of us have heard about Conway’s law. It claims relatedness of organizational structure (with its related processes) and produced system architecture - they go hand in hand. And that’s of course not a surprise. Consider a company with highly strict functional departments and lack of interdepartmental collaboration. Which kind of system would it produce? It would likely end up designing a set of isolated components, each exposing a unique and complicated interface. That’s an example of a causal connection between organizational structure and system architecture.
What is actually interesting here is that this connection can be reversed! Meaning: you can influence changes in the organizational structures by reshaping your system architecture. So now looking back at the last 1.5 years we can clearly see how the reversed Conway’s law got proven at XING by our transition. In fact the overall scaling initiative (namely: having more teams own and contribute to app development) was only possible because of the technical breakthrough in the mobile architecture (namely: breaking our monolithic apps into sets of independent modules).
Surprisingly, we didn’t start with a master plan of breaking all apps down to modules. It came much later when we realized it was the way to scale.
Also, the innovation came from the bottom (as you would expect it, right?). While management was trying to craft a strategy to the scale (and I was a part of those talks), several developers have sketched an architectural idea and one of them coded it during his slack time.
That sounds like a chapter from one of those technical novels you might have read. But simply speaking, this is exactly how it all started: while others are talking, a small group is simply making things real.
Of course, it was not all that trivial and hippie. One could even say the breakthrough was only possible because of the modern management approach applied at XING. I can think of at least two key management practices that let the innovation emerge in the first place:
- A loud need for change was advertised by the management - everybody felt a strong sense of urgency and some people started to try new things out.
- Engineers had slack time to innovate - here at XING every 8th week is what we call the Innovation Week where you as an engineer can work on anything and with anyone you’d like.
So in fact the first module was cleaved out of the iOS app during one of the Innovation Weeks. It was the login module. Eighteen months later it is still one of the 25 modules composing our current iOS app that has over 1.5 million downloads. Despite the number of the modules and the overall app complexity (200 - 350 KLOCs per an app) we are quite proud with the level of consistency we have between the different parts of the apps that are developed by different teams: you can download and see for youself here.
2: Breaking Up the Apps
Once we realized the modules are the way to go, we started to extract domain logic from our monolithic apps into modules. All the heavy-lifting of splitting the apps was performed by the framework teams (our new team structure described in the next chapter; for now you just need to know that these are small teams of developers focusing mainly on infrastructure).
The clear scaling strategy started to emerge: extracting all domain logic into separate modules and giving them away to the domain teams. Before these teams had been developing for the web only, but now there were given mobile modules of their domain to own, maintain and develop further.
This is what we appeared to have: feed module, messages module, jobs module, and so forth. Now our releasable apps were becoming a kind of containers of modules that are packaged together. Independent development and versioning of the modules now became possible.
A bit of tech bits (actual as of writing the article):
Our iOS modularization is based on CocoaPods and each of the extracted module (pod) sits in its own GitHub repository. We have developed a dependency resolving script (similar to a ruby bundler) that assembles an app based on releasable versions of the modules.
On Android all modules reside in one common repository as libraries and all dependencies between them are managed by Gradle build engine.
Both approaches proven to work at the level of scale we are experiencing. Both of course come with their pros and cons. Interested to know more about techy stuff? Contact our devs or ping me @alexeykri and I’ll get you through to them.
As of writing the article (summer 2015) we’re still extracting app parts into new modules, also some modules got merged together. Seems like it is a natural process of adopting architecture to specific needs and pain points.
In fact now we differentiate different types of modules: domain modules, reusable libraries and UI components.
If the domain modules are logically owned by the respective domain teams, the libraries and components are in the hands of corresponding iOS and Android platform teams (more about these teams below).
3: Forming Framework Teams
This happened during the very early days of our transition process: we realized that if we need to make significant progress on the front of scaling mobile, we need to allow our folks to keep a sharp focus. Hence, after some debates we decided to equip special squads - teams whose only mission would be on trying new architectures and supporting other teams in scaling. These teams will not be contributing to development or support of app features, rather creating and improving environment to make it easier for other teams build features and release the apps.
Basically this is how we formed what we call now ‘framework teams’: small special teams within our mobile family of teams who focus on the infrastructural topics.
To be honest, I personally didn’t buy the idea of forming special teams to deal with the infrastructure. Being a fan of feature teams over component teams, that felt somehow wrong to me. The pure mobile teams we had in the company became bottleneck of mobile development, that’s why we have started the whole initiative of unleashing mobile. And now we were seem to create another special set of teams. Will they become bottlenecks in the nearest future? Are we really solving the root-cause and improving the system?
While I was asking myself and others these questions, we decided to try and see how it would go. The whole mobile transition was a big experiment, so why not make another smaller one inside? Inspect and adapt, right?
So that what we did: the small number of mobile developers (two people per a platform) decided to join the newly formed framework teams and dedicate themselves to defining and building infrastructure and probing new modular architecture. In the very beginning of our transition responsibilities of the framework teams covered things like: prototyping and introducing modular architecture, extracting domain modules, redesigning deployment pipelines, automating deployment and testing routines.
Over the course of the mobile scale transition the mission of framework teams has changed once the basis had been laid out (described later in the article). More and more often I hear people within our company challenging our current team constellation. And I would not be surprised if within the next 6 months the framework teams would call their mission ‘done’ and join back to app development. Or maybe a new experiment will be planned. I’ll keep you posted.
Anyways, once we saw the steady progress with the new scalable architecture, were we able to enter the next phase of our transition project - scaling. Yay! Apparently everything we’ve done so far was only laying a stable foundation to allow the scale. Scaling has to be well-prepared and done slowly and it took us about 9 months to get to the stage when we could start adding more mobile developers to the equation.
4: Onboarding, Bootcamping, Bootstrapping and Seeding
About nine months after our first modules had been introduced, we started to onboard new developers into our existing mobile teams. That made our initial mobile teams grow quite largely. And they started to play a role of boot-camps. In 2014 we have doubled the size of our mobile staff crossing the edge of 30 engineers for both platforms together.
The extended teams have been actively exploring new development practices that would suit the new vision: multiple teams should be able to contribute to the apps, while maintaining decent level of UX and code consistency
Just for the record: in just a few couple of months the teams have experimented with several branching models, substituting daily code reviews with pull requests, introducing release trains (regular pre-planned release cadences), forming communities of practice... Once you start scaling your development processes, speeding up learnings becomes vital.
The active boot-camping lasted for about 3-4 months and after completing several projects together we started to seed newly onboarded mobile developers into the existing domain teams.
5: Living the Scale
By the spring 2015 (about 18 months in transition), we can say we have seriously stepped into the land of scaled mobile development. Currently we have 4-5 domain teams contributing to our core mobile apps on two platforms. And every two weeks new app versions get submitted to the stores. That’s quite a rhythm for this many teams and this level of app changes.
Illustration: scaled teams
As you can see we have ended up having two kinds of mobile teams: platform and framework. The platform teams are responsible for maximizing the apps’ consistency, usability and quality. And consistency comes in two flavors: 1) across domains and 2) within the given mobile platform.
In order to achieve this, our platform teams spend quite some time in joint discoveries with the domain teams. They help out reach desirable UX by maximizing existing pattern and code reuse (emergence of the common pattern library deserves a dedicated article). It is a full-time job. And it is a new role. I think only now we are starting to understand all responsibilities this new role implies.
On daily basis this results in some of the following continuous activities:
- Foster other teams using best coding practices, guidelines and maximize code reuse. This results in running shared code reviews and regular dev exchanges. Holding workshops on test automation, feature tracking, usage of our standard component libraries - all topics that each domain team should master in order to be a good citizen within our mobile ecosystem.
- Participate in the early phases of product discoveries. Representatives of the platform teams are acting as sparring partners to the domain teams to help them design solutions that not only perform best for the domain but also consistent across the apps. Participating in other teams’ backlog refinement meetings, design days, API reviews, tech spikes are just some example of these activities.
- Coordinate release trains. Since we release new app versions every two weeks this usually results in running a weekly stand-up with representatives of all contributing teams, collecting release notes (we use wiki for this), automating release testing, communicating to App Stores.
- Monitor and troubleshoot beta and live apps. This covers daily tasks of observing app live behaviors, crash rates, app ratings, beta testers feedback, helping out with customer care topics.
The focus of the framework teams is on making the domain teams contribute to the apps as fast as they need to. Hence the framework teams can be seen as road workers: ever improving the delivery pipeline - our highway to the users. They are dealing with such things like: build scripting, automated test coverage, library upgrades, release train coordination, automated app monitoring - anything that would otherwise make other teams slower or lose their domain focus.
You may ask what kind of specialists did we recruit to join our frameworks teams? Which special skills were we looking for? How did we select people?
We actually didn’t do any special promotion or hiring. Indeed some of our mobile folks have been already involved in automating build pipelines and all that by spending most of their time in a ‘black hole of a linux console’. So all that happened is that management gave clearer naming and helped shape what had been emerging. Is that not what true servant leadership is all about?
One should not forget that the architectural change came bottom-up and now the bunch of guys who had started in a guerilla-style mode were receiving a formal acknowledgement of their contribution and a clear green light to continue and bring it to the next level.
Who wouldn’t agree?
And what about those guys? Well, they do as they have been doing. But now instead of thinking "just web", they are designing and building their solutions on several platforms. This of course implies extending their competencies and skills to mobile. And not only on the coding side, but as well for QA, UX, product management. That's a big mind shift.
As an example of the shift: in the Q3 of 2014 first in the XING history we developed a pure multi-platform product. The so called "Industry News" has been first released on mobile as an MVP. And then, once validated, have been brought onto the web. It all has been done by a single domain team. That’s the outcome we had been striving for during the long months of transition.
It is probably not worth mentioning that despite our breakthroughs, our journey is far from being over. In fact we are in the very beginning of the new multi-platform era and are very excited about it. The users have just started to see the impact of the changes. And there is so much more to come!
Key Learnings and Why You Should Not Repeat It
If I were asked to summarize our key learnings in five simple bullet points it would be statements like:
1. Alignment and sense of importance.
Create enough understanding of the organizational problem and give space for engineerings to attack it.
2. Patience as a sign of respect.
Be patient, supportive and trust that your teams can do it. Complex problems require non-trivial solutions. Those can’t be simply bought, installed or managed. It is more like fishing - you can do your best to come to the right spot, right on time and with the right tooling. But you can’t speed up the fishing process. Or if you’re a vegetarian and fishing is not your thing - think of seeding a flower in a pot. You can do your best but in the end it takes as much time as it needs to take.
3. Autonomy in the moments of silence.
Innovations can’t be planned or facilitated as group activities. Make sure your team members have individual slack time and can work independently on the things they do believe can potentially work. Design is an individual process.
4. Amplification as a key management practice.
When finally your teams are onto something promising, help it grow and be taken care of. This is when team coaching and project management practices become handy. Converge and join everybody’s efforts to bring the thing to life.
And hence we are specifically speaking here on scaling, here’s my fifth take-away:
5. Scaling is only the last phase.
Don’t scale until you know what you’re doing. Don’t throw people to let them ‘figure it out’. Before you start adding more team members and teams, make sure you have a stable technical foundation and everybody knows what they’re doing.
I don’t think we found the most efficient process. It took us what it took us - 18 months and we are still miles to go. Some signs on our way proof to us that we’re moving into the right direction. It is getting sunnier and warmer (a constant dream of all people living in Hamburg and Northern Germany in general). But we don’t know the other way.
Still it doesn’t mean there is none. Who knows? Maybe there is a very well soundproof autobahn (with no speed limits) right behind the bushes we’re cutting through… If you find it, give us a friendly beep.
About the Author
Alexey Krivitsky is a Lean-Agile practitioners with deep hands-on experience and a Certified Scrum Trainer®. The first ScrumMaster experience of Alexey dates back to 2003. Since then Alexey has been actively engaged in coaching teams and organizations, by working as a ScrumMaster, Agile coach and business trainer. He has co-produced a number of Agile events, including the Agile Eastern Europe Conferences 2009-2015. Alexey is also known for inventing the Scrum Simulation with Lego that is translated to more than 15 languages. More about Alexey on his personal homepage.
Platform team members