Engineering Culture and Distributed Agile Teams

Distributed teams are the norm for many organisations today. Companies are global, communications technologies allow people to live away from the "office" location and many of the new workforce are nomads. Becoming a high-performing team is possible in a distributed group, it just takes more effort to overcome the inherent challenges of distance.

This is the fourth in a series of articles which explores the impact of cultural differences on distributed teams - "Distributed Agile - how to work with teams across the globe". You can subscribe to receive notifications via RSS.

Having a high productive engineering culture should be the same whether your organisation is centered around one physical location for the whole company or when people work in globally distributed teams.

Most of the time distributed teams are a fact of work-life: many organisations use distributed teams in their search for talent. The search always starts in the area where the headquarters is located but then quickly expands to the rest of the world. In some cases setting up these distributed teams causes undesirable situations.

Most of the distribution starts with how the contract is setup between the head office and a supplier. Most common problem is the decoupling of different types of work; i.e: "Design is done by HQ and supplier only writes code". This form of distributed work has had far reaching negative consequences for both companies in the contract. You can even say that this type of decoupling is a cause for reputation damage for the whole industry of "outsourcing companies".

Unfortunately, on the quality of work side of things; nothing good can come from this structure. Because of the decoupling of different aspects of the work, it is virtually impossible to have a value chain wide improvement initiative. There are too many opportunities for noise during production releases because of all the hand-overs.  

In addition to that, Distributed teams impact the product architecture. As per Conway's law:

Organisations which design systems are constrained to produce designs which are copies of the communication structures of these organisations.

The law is based on the reasoning that in order for a software module to function, multiple authors must communicate frequently with each other. Therefore, the software interface structure of a system will reflect the social boundaries of the organisation(s) that produced it, across which communication is more difficult.

Evidence in support of Conway's law has been published by a team of MIT and Harvard Business School researchers, who found "strong evidence to support [what they term the equivalent]… mirroring hypothesis", and that "significant differences in [product] modularity" were "consistent with a view that distributed teams tend to develop more modular products".

Hence, software quality is dependent on the organisational structure and degree of collaboration among team members.

There are multiple challenges associated with the distributed teams which could be overcome by using some techniques, which we are going to explain in this article.

I've started developing a distributed agile framework together with 3 others: Hugo Messer, John Okoro, and Arjan Franzen.

We are still in the inception phase. Our belief is that we should open source a big part of the framework. The framework we have developed consists of 9 'bubbles':

1. Culture
2. Organization
3. Product
4. Team
5. Architecture
6. Engineering Practices
7. Communication
8. Tools
9. Leadership

Each of the bubbles has 3 elements:

• Questions: a set of questions that organisations can use to assess their current state
• Virtues: behaviours that foster distributed collaboration
• Practices: things that have worked, shared by practitioners

Following questions could be asked to the distributed teams to find out the health of engineering culture and associated challenges.

Engineering Culture

A good engineering culture is the heart of an agile team. A healthy engineering culture enables teams to build a product that means something to them.

Questions

• Do we follow any engineering based practices as opposed to planning based practices
• Is the adoption spread equally across our distributed teams?
• How do we spread adoption?
• Are our systems designed to support adoption across our locations?

Virtues

• Strive for excellence - Teams should own their code and always strive to write clean, readable, maintable and extendable code. Good agile teams strive for excellence in their work and think from the long term perspective. That means, they forsee how long, on low maintainance, code would be easy for change.One of my team in a startup always used to think the architecture from the perspective of increasing the user base gradually and how code should be flexible to achieve their goal. I have seen developers and managers talking about writing the comments or documents to explain the code's behaviour. This a smell of not properly written clean code.
• Build quality in - Good agile teams hold accountable for software quality. They strive ways to build quality software as per client's needs and  focus on avoiding quality issues in the first place. Agile teams should take care of the quality in mind since very beginning of the product development and continue throughout the development process.  
• Incremental process improvement - Inspect and adapt is the core of agile teams. They inspect, learn and improve continously. Organisations just need to create an environment where people can freely share their opinions and experiment new stuff so that they can improve their products and processes continously. Moreover, often people have great ideas but when they are not empowered, they do not share any improvement ideas or take responsibility of working on these ideas. True leaders empower the team and encourage them to learn from fauilures. This kind of fail safe environment is very important to build a good engineering culture.
• Learning: Get better at what we're doing. Learning mindset leads to successful products. Teams learn from various feedbacks. The very first feedback they can get from the unit tests associated to the code and then from the product and users. The shorter the feedback loop, more the learning. Also, good engineering teams share with others, whatever they learn. That's how the whole engineering culture thrives in the learning environment.
• Simplicity - This virtue is one of the most important values of Extreme Programming. The methodology favours simple designs, not thinking too far ahead into the future, but focusing on the requirements of today, while making the code extensible enough to add new requirement in the future.

Practices

Remote Pair Programming

Pair programming is an agile software development technique in which two programmers work together at one workstation. One, the driver, writes code while the other, the observer or navigator, reviews each line of code as it is typed in. The two programmers switch roles frequently. We know very well the benefits of pair programming as:

• Better quality
• Fewer defects,
• Improved skills, and
• Faster delivery.

But what happens when you want to pair with someone in another city or country? Yes, you can still perform pair programming by using some remote pairing techniques and tools. Use tools which provide collaborative editors, screen sharing, secure networking, and virtualisation to create a remote pairing environment that feels as if your pair is sitting right next to you.

The below mentioned tools can be used to do remote pairing:

• SelfLanguage

• SqueakSmalltalk

• VirtualNetworkComputing

• RemoteAdmin

• NetMeeting

• SpeakFreely

• ScreenMultiplexor

You can also use some collaborative editors for during coding with your remote pair:

• EmacsEditor

• QuantumPairs

• Sangam - PairProgramming-PlugIn for the EclipseIde(http://sangam.sourceforge.net/)

• Saros - PairProgramming-PlugIn for the EclipseIde (http://www.inf.fu-berlin.de/w/SE/DPP)

• SubEthaEdit and AppleiChat

• gobby (GTK) http://gobby.0x539.de/

• MoonEdit

• MultitaneousEditor (Java)

• UNA  (http://www.n-brain.net/)

• beWeeVee - A real-time collaborative notepad based on Silverlight.

Micro services and Distributed teams

Micro services is a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services.

Many times people share technical challenges in distributed teams. For instance, people in different locations work on same code base and this lead to overhead of overwriting or integration. How we design an architecture that multiple teams can generate value continuous without any hassles. To achieve this use loosely coupled code or architecture. One of the best options is to use micro services. This is not applicable in all type and size of applications but domain driven service oriented applications can benefit from it very well.  

How micro services can influence a good distributed team structure? Following Conway's law, breaking isolation of your services breaks the isolation of your teams, so be sure that your organisation is able to cope with that level of communication and integration. Hence, having small cross functional teams working on different services in different locations works well.

Multiple remote cross-functional teams working on different services based on business capabilities

One more benefit of using micro services in distributed team is that different services can be built using different technology stack. For example, your team in one country is strong in C++ and other team in different location in Java, you can take a call based on suitability to build one product based on skills availability as well. That means decentralised governance, hence, multiple small colocated teams in different locations.

Virtual Office

Distributed teams need some face time with members on other loaction. In one my teams in India we used to have active skype video call throughout day with people in Amsterdam. This way of working increased collaboration between team members. They felt more open to ask questions and help each other.

Along with the skype, you should use a big screen in the team area so that people can see the whole team. Also, in this way you can survive with only one active call.

SOCOCO is also a good tool for creating a virtual co-located office environment.  Through this tool you can create real rooms on a real map that are always there – not an abstract notion of a “channel” or “message thread” or “work group”. Also, with a click, video, audio and screen share are instantly up, and a spontaneous meeting will just frictionlessly and immediately start.

(Click on the image to enlarge it)

A Sococo online office with an Agile team meeting in progress.

Ad-hoc coordination, Team as one, t-shaped team-members - by Arjan Franzen

It is normal to start in a situation where each person in a team is most focused on his own functional expertise. They pay attention to estimate that are appropriate to them and work on a 'personal work-list'. Working this way will likely push back possible obstacles, they are filed as 'distractions for which I have no time'. By working distributed there is an extra penalty for coordination with your teammates: think of setting up tools or having coordination meetings. These penalties are the cause of a lot of rework.

By deferring communication or coordination you are guaranteeing that the learning is delayed and so the wrong things are produced/delivered. You can say that having all sorts of coordination meetings is counterproductive to cooperation. When you spend your days in coordination meetings, no work gets done. This is true also, however people 'save up' discussions and this is equivalent to delaying troubleshooting. What works well in a distributed setting is having the facilities for ad-hoc communication; Things like always-open-Zoom-meeting-rooms, Mirror-rooms or virtual offices like sococo make it easier for teams to meet and have internal coordination.

Building successful teams has a lot to do with improving the internal coordination. The team indicates (to each-other and to external stakeholder) when there should be coordinated efforts or that somehow they "need to sit down with". These are important ad-hoc coordination moments that do not need a wait for a weekly coordination meeting.  Agile and Scrum daily stand-up meetings are also a great opportunity for daily coordination within the team.

Next to the ad-hoc coordination, the team must be seen as one by the rest of the organization. Too often you see that local experts (heroes) have multiple assignments that are seen as perks by the organization. This means they can slack off at certain moments from the mundane team work. This culture should be countered with a culture in which the team is responsible as one and is lead as one. Approaching a team as one means that external stakeholders do not interfere with the day-to-day planning activities of the team.

Approaching the team by indicating that learning inside the team is not waste, it does take time so please be aware that you must complete in time and spread knowledge within the team. This makes for a nice planning puzzle for the teams to solve: once they solve it they've made a big step in becoming more autonomous and more mature in terms of being a self-organizing team. A mature team that act-as-one will see obstacles that risk the constant production flow (flow is a practice that comes from Kanban and Lean) being harmed and will start experimenting with ways to avoid this 'pain'.

The mantra of 'bring the pain forward' or "fail fast" is in essence an informal risk management technique practiced by teams. These teams will attempt to prevent blockage of the delivery pipeline by mitigating perceived risks on the backlog. Tools to do this are 'proof of concepts' and/or 'spikes' both are work items that do not necessarily deliver value to the customer immediately but do make sure that work can continue to flow through the system in the future.

The flow, Continuous Integration and Early Testing - by Arjan Franzen

The book "The DevOps Handbook" describes how the "three ways" can be used to implement DevOps. The first way: Optimizing flow. These concepts stem from Toyota's Lean and are central to the control of the process. The flow can be seen as the work packages that flow through the system. " This way of management is a complete shift when compared to traditional planning.

The traditional way first creates a statement of work; it is estimated, implemented and tested. All controls are centered around capacity.

The management of a process flow to be best at the moment that there is a central view of work packages. This central view of open and current work is needed and can be approached the same as if a team was entirely co-located.The overview here is most important so a central JIRA board (or other Kanban board / tool) is good on a co-located wall display as well as a bookmark on a distributed team-member's desktop.

The first practice the team must engage is continuous integration. Essentially this attempts to remove as much of the delayed integration from the process.

Delayed integration means that people in teams work in isolation on their own or group based branches. Sometimes this is a much needed solution that prevents the software from complete failure. However these types of branches should be viewed as a necessary evil and not "a good way to structure development". Continuous Integration mandates that you kill as many of your long lived branches as possible and that you merge into mainline at least once a day. This means a lot of coordination and possibly pair-programming.

This integration work if done consistently will allow the team to change the design of the system in such a way that it is more open to change. Robustness must be built in (engineered in step by step) in order to have a reliable release every time you release. Distributed aspects of the team do not block the team being successful in changing toward a Continuous Integration (CI) environment. So with the branches minimized and the merging done we make sure we install a buildserver (CI server) to keep watch and start the first of many quality checks. The start of the delivery pipeline

With the CI server in place we build on this orchestrating the automated tests. It is crucial that the team works together internally (Dev-Test) to make sure tests are coded in such a way that it fits nicely with the rest of the codebase and that what is being tested makes sense from a business perspective.

Coordination with external stakeholders is needed to assess what are the scenarios that will have the most negative impact in production once they fail. To facilitate this coordination make sure the team does regular demos to external stakeholders so that a part of the team (Dev+Test specialist) have regular opportunity to learn about the business context.  If your team uses Scrum, demos would be a standard part of the team's work during each sprint.

In distributed teams, travel may be needed to facilitate this as some parts of the context may not be easily transferable through an internet based communication platform. Make sure you structure the automated tests along the lines of Continuous Delivery. First start with the basic unit-testing. Continue with automated regression and finish with automated capacity tests.

Some automated test tools include:  Selenium, Cucumber, JUnit, Mocha and Jasmine.  A fairly comprehensive list of automated testing (and other DevOps tools) can be found in the Xebia Labs Periodic Table of DevOps.

Cloud based Distributed Software Development

Distributed Software Development (DSD) in the cloud is increasing exponentially. In many cases, it seems an obvious choice, especially for new, emerging ecosystems where companies want to involve external developers or external development teams in their own development projects or programs.  Ramping up cloud infrastructure is inherent to cloud-based services, but adding a new offshore team to the existing cloud-based DSD project may pose several technical and non-technical challenges.

Technically, cloud enables elasticity, scalability, and flexibility in DSD teams, resulting in overall increased productivity. For example, cloud-based DSD allows teams to perform rapid testing, dynamically scale up or down the required computing infrastructure, and produce working software updates rapidly.

One of the major benefits experienced by the teams in the project was sharing cloud-based tools across distributed teams. It enables rapid development in the distributed teams without the hassle of replicating technical environments.

Teams can use cloud programming environments which are online web-based applications designed to offer development capabilities to developers. They usually consist of a source code editor, a number of compilers or interpreters depending on the programming language, a debugger and a project/solution viewer for managing the independent subcomponents. On top of that, programming environments also include connections to code repositories, collaboration features for code sharing, VMs for instant application deployment and even monitoring tools.

Cloud repositories are web-hosting facilities that leverage the strengths of well-known version control systems such as Git, Subversion and Mercurial. In addition, they often support services and tools, including bug tracking, release management, mailing lists, and wiki-based documentation.

Conclusion

Being successful in a distributed Agile team is equally difficult from being successful in a non-distributed (co-located) agile team. – Distributed/Non-Distributed --There are some differences and these center around the internal coordination of the team. Since a team may be non-co-located focusing on easy ways to communicate and coordinate are essential in a distributed setting. Managing teams as one is a second step towards success. With this anti-local-heroes culture in place you will find that the engineering practices of continuous integration and early automated testing will be much more fruitful if compared to a setting where a developer's work is only inside an IDE and a tester is only fed test scripts and requirements and does all the testing work manually. Laying this foundation around open and early coordination is entirely possible because of the large numbers of tools, platforms available to support this. When combined with the right Agile engineering practices your teams will become engaged and highly productive.

About the Authors

Savita Pahuja has expertise in Scrum, Lean, kanbanareas and other visual discovery methods. She started her journey in IT industry as a developer and contributed the organization in agile adoption. Then she moved into agile world as a consultant and trainer. Since then she has been working with different clients for agile adoption and giving trainings on scrum and kanban.

Since Arjan Franzen became familiar with Agile, DevOps and Lean, he's been very excited about these innovations in software development and try to apply them in his professional environment as much as possible. Because he is passionate about this new way of software development he started his own software company: ZEN Software. By providing consultancy services on topics like Delivery pipelines, DevOps, Continuous Delivery and Lean software development, Arjan wants to enable companies to minimize risks of software releases. With minimum risk per release, the release frequency can increase. By applying Lean (and Lean startup) principles, it must be possible for companies to measure effectiveness and value in software changes: "Build the right thing". If companies build the right software (demand), and build them correctly (quality) they get meaningful software deliveries; "Purposeful software delivery" - The Motto of ZEN Software.

Distributed teams are the norm for many organisations today. Companies are global, communications technologies allow people to live away from the "office" location and many of the new workforce are nomads. Becoming a high-performing team is possible in a distributed group, it just takes more effort to overcome the inherent challenges of distance.

This is the first in a series of articles which explores the impact of cultural differences on distributed teams - "Distributed Agile - how to work with teams across the globe". You can subscribe to receive notifications via RSS.