Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News How to Remain Agile While Scaling

How to Remain Agile While Scaling

Leia em Português

Managing the complexity and scale of both the business and the software were key challenges as Yext started to grow, said Sean MacIsaac at Agile Business Day 2019. Yext addresses this by having product solutions that are broadly applicable and configurable by their customer team, and by hiding complexity with independently deployable services with clearly defined interfaces.

Agility is all about communication and alignment, said MacIsaac. Startups have less information and there are fewer people to communicate with. Everyone has personal connections, making it much easier to get alignment and have a high trust environment. This helps simplify both large decisions, like strategic direction, as well as small ones like implementation details of particular product features, as MacIsaac explained:

The early days of Yext involved moving from a product that was focused solely on health clubs into one that worked for many local businesses. While the decision was led by the CEO, every engineer and all of sales leadership were heavily involved in the process. The decision was made quickly, and once it was made, everyone already understood the context of the change, its goals, and key features needed to make the product work at a bigger scale without additional effort around communication.

When Yext started to grow, employees started to specialize on the business side. Different people focused on different customer industries and sizes, hence more stakeholders had to be involved in understanding and solving key customer needs, said MacIsaac.

On the technical side, Yext addressed challenges like being available 24/7 and having answers for key scalability problems:

When we had ten to fifteen engineers, we introduced technical mechanisms to solve these availability and scalability challenges like load balancers and independently deployable services. It did require all changes to services to be backwards compatible until all consumers of the service had also been updated, but this was a small price to pay for the isolation of the complexity.

MacIsaac mentioned that for core product functionality, the use of mainline development combined with automated testing, continuous integration, and feature flagging enables rapid iteration. Engineers can change the system and have the confidence that their changes will not break other parts of the system that they may not be as familiar with, he said. Code changes are integrated constantly; there are no surprises at the end of projects, such as two teams having built functionality in an incompatible way. Feature flagging lets them put new code into the mainline, activating it only for test accounts or early beta customers.

Yext adopted a model for breakthrough R&D that mimics practices from their early startup days. They have formed cross functional teams that include strategy, product, sales, business development, engineering, and operations for new products that are pre-product market fit, to keep the advantages they had in communication and alignment as a startup. MacIsaac mentioned that it is difficult for leaders to sacrifice the members for new product development teams, but the payoff is worth it, as having trusted team members who are given the autonomy to make decisions and implement them has been key for their major R&D efforts.

Sean MacIsaac presented how Yext navigated their scaling while keeping their agility at Business Agility Day 2019. InfoQ interviewed him afterwards.

InfoQ: How did you go from project teams to stable teams and what benefits did that bring you?

Sean MacIsaac: Even with the logical separation of services, there came a point where engineers couldn’t bounce between different parts of the system without a steep learning curve. It was at this time that we went from whole team ownership of the system to individual teams that owned parts of the system. This required more project management effort, as some projects spanned teams.

There were a few key things we did to minimize this overhead. The first was having teams own services that were closely related, so many projects still involved only one or two teams. The second was getting early agreement on the interfaces between the systems, which enabled teams to work mostly independently. These interfaces were enforced by CI/CD and automated testing to expose differences in assumptions early as well.

The benefit we got out of long term service ownership in teams was in ongoing maintenance and operations. After this change, teams at Yext owned the reliability of their parts of the system, so there was a clear place to go if something wasn’t working as expected. It also incentivized teams to handle technical debt to reduce issues and make the system more maintainable.

InfoQ: What have you done to mature practices within Yext?

MacIsaac: Adopting an external standard for compliance was a major step that helped us mature the processes both within engineering and company-wide. We are formally audited by EY as part of getting a SOC 2 attestation for security, confidentiality, and availability. As part of adopting the standard, we examined all of our practices and improved or formalized many of them. In addition to ensuring we met the criteria needed to pass the audit, we got engineering feedback on the development process and tools used for development and made improvements to them that not only increased security and availability, but also enhanced developer productivity.

A key example was the implementation of our current CI/CD system. As part of adopting the standard, we formalized the configuration of the CI/CD system, including defining automated unit, integration, and acceptance tests that need to pass before a service was promoted between our development, QA, sandbox, and production environments. While many individual steps and procedures were already automated or scripted, the end to end process of deployment was not. This left responsibility on individual team members to ensure the right thing happened for each deploy. After this work, we are able to confidently say that the appropriate tests have run and passed before any deployment, and engineers can move more quickly without having to memorize complex procedures or refer to a wiki for instructions.

The other way process has matured has been much more bottom-up. As part of regular retrospectives, individual teams examine and tweak their processes. Techniques that work well end up being shared between teams, both formally and informally, and are adopted widely. A great example of this is the adoption of the bug bash program at Yext. The first one was the result of a team refining their QA practices. It was so successful that engineering, product, UX, and others have decided to adopt it as a standard practice.

InfoQ: What’s your advice to startups that are growing and want to keep their agility?

MacIsaac: On the functional side, focus on communication. As part of the process of going from idea creation to having a product that customers can buy and use, consider what documents and status reports you create to what meetings you hold. However, this is always a balancing act. On one hand, it takes a significant amount of time to create and consume documentation and conduct meetings. On the other hand, it’s impossible to train a large sales team on product functionality without the appropriate artifacts, training sessions, etc. Continuing to co-locate people who need high bandwidth and continuous communication, such a UX, product, and engineering, is very helpful. As I mentioned, we go so far as to dedicate and include strategy, sales, and marketing when we undertake making new product lines to maintain a startup-like agility.

On the technical side, agility tomorrow is almost always a result of investments today. This includes activities like handling technical debt within the services a team owns to make the service more maintainable. We also make investments in developer tools and infrastructure. These investments typically increase the safety of making changes or reduce feedback time from an engineer writing code to being confident that it is working. At Yext, we aim to spend about 20% of our engineering time in aggregate on these types of activities. Within teams, it ebbs and flows depending on product demands of the team and the services they own, but we also have some engineers fully dedicated to tools and infrastructure.

Rate this Article