Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Netflix Announces SafeTest, Its Custom Approach to Front-End Testing

Netflix Announces SafeTest, Its Custom Approach to Front-End Testing

This item in japanese

Moshe Kolodny recently introduced SafeTest, described as a novel approach to front-end web testing. SafeTest orchestrates a test runner, a browser automation library, a UI framework, and dependency injection capabilities to alleviate the pain points of traditional UI testing methods. SafeTest is currently used at Netflix.

Kolodny explained in the release note that the two main approaches to UI testing, i.e. unit testing and end-to-end testing, both present tradeoffs.

As user interfaces become commonly written with a component-based framework (e.g., React, Vue, Angular), UI unit testing often devolves into component testing. Components being tied to a framework require that framework to provide ad-hoc testing capabilities (e.g., React’s act method in react-dom/test-utils, Angular’s TestBed API). As components became larger and more complex, components’ dependencies (e.g., the UI framework, the component’s children and props, the DOM, the network, miscellaneous side-effecting functions, configuration settings, and more) inflated. Unit testing (i.e., testing units in isolation of their dependencies) thus turned out to be an increasingly arduous task. Such unit testing often forced developers to reference internal implementation details in their tests, resulting in tests unduly failing as a component’s implementation changes.

As a result, front-end developers have moved in recent years away from shallow rendering (e.g., React’s Enzyme library) components to full DOM rendering (e.g., the dom-testing-library suite). As components are responsible for a smaller part of a full application’s specifications, the idea is to test those specifications by instrumenting the DOM as a real user would (i.e., mock the user’s actions on the DOM).

While high-quality mocks of the DOM exist (e.g., js-dom), primarily for use in non-browser runtimes such as Node.js, they still fail to implement the full list of web standards and thus do not fully replicate the browser behavior in all cases.

Kolodny explained:

While it’s easy to write the setup for the test, it’s hard to write the events needed to cause things on the page to happen. For example, displaying a fancy <Dropdown /> isn’t as simple as just calling'select') since the js-dom doesn’t perfectly match the real browser, so you end up needing to mouseover the label and then click the select. Along the same lines figuring out how to enter text on a smart <Input /> has a similar battle. Figuring out the exact incantation to make this happen is hard and brittle. Debugging why something stopped working is also hard since you can’t just open the browser and see what’s going on.

This led to testing components without mocking the browser, instead using headed or headless versions of browsers (e.g., Microsoft’s Playwright, Google’s Puppeteer).

At the extreme, removing all mocks leads to the second test method mentioned by Kodolny: end-to-end testing. Kodolny warned of the associated tradeoff:

E2E tests like Cypress and Playwright are great for testing the actual application. They use a real browser and run against the actual application. They’re able to test things like z-index issues, etc.

However, they lack the ability to test components in isolation, which is why some teams will end up having a Storybook adjacent build to point the E2E test at.

Another issue is that it’s hard to set up the different test fixtures. For example, if you want to test that an admin user has an edit button on the page while a regular user doesn’t, you’ll find some ways to override the auth service to return different results.

Similarly, component testing isn’t possible when we have external service dependencies like OAuth since Cypress and Playwright component testing do not run against an actual instance of the app, so any auth gating can make rendering components impossible.

SafeTest presents itself as a third way that combines the best of the two previous methods (unit testing and end-to-end testing). SafeTest allows leveraging a real browser (with screenshotting, video recording and replay, and trace viewer) while still allowing the mocking of non-browser dependencies.

SafeTest may leverage other libraries for the mocking or stubbing of external dependencies such as the network or web services. Playwright allows, for instance, stubbing the network layer. For internal dependencies (e.g., static functions, values, API responses), SafeTest provides a mock/spy API and a dependency injection mechanism appropriately called override.

One user on Hacker News emphasized that SafeTest could bring a proper dependency injection mechanism to React:

Should it be a general framework focused on DI [Dependency Injection] for React? That can make overrides more organic and have more benefits than just testing.

Another Hacker News user opined that dependency injection needs can be eased by making implicit dependencies explicit in the form of parameters:

I’m afraid the answer to this doesn’t actually lie in tooling. It lies in software design. If something needs to be controlled, it needs to be controllable. Typically this means push. In a React component, this means props. It could be an optional prop, but once that prop was there, this component could be controlled. Once the component could be controlled via push, the page rendering the component could also be controlled via push. How do you push to a page? Query string params is the most straightforward.

Storybook kind of enables [that].

Yet another user agreed that SafeTest addresses real pain points, albeit through an unfamiliar method:

I think the solutions outlined really interesting and valuable! There are some idiosyncrasies around unit testing UI that are just hard to work around. And the issues with e2e tests in the context of testing individual UI pieces are totally valid too.

The one thing that at least initially rubs me the wrong way is how the overrides work. Like I get why that’s a solution to how we can inject some data, but I don’t like the idea of writing test-specific code in a component just to enable tests with this tool. That being said, I’ve done similar things in the past when I’ve run out of options and this looks pretty clean, I just wonder if there’s another way.

SafeTest is open-source software under the MIT license. Developers can review the release note for more technical details and code samples.

About the Author

Rate this Article