Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Microsoft Introduces New MSTest Runner: Portability, Reliability, Extensibility and More

Microsoft Introduces New MSTest Runner: Portability, Reliability, Extensibility and More

Microsoft has introduced the MSTest Runner, a lightweight test runner designed specifically for MSTest tests. The primary goal of MSTest Runner is to enhance test portability, reliability, and speed while providing an extensible testing experience for users.

The new MSTest Runner was introduced as an independent portable executable, with a focus on eliminating the need for external tools like vstest.console, dotnet test, or Visual Studio during test execution. The development team describes this in a way that this approach makes it well-suited for authoring tests on devices with limited power or storage, offering a simplified testing solution.

As stated, developers of all skill levels and project sizes can benefit from the MSTest Runner, which comes bundled with the MSTest.TestAdapter NuGet package, version 3.2.0 onwards. It is integrated with common testing environments, including dotnet test, vstest.console, Visual Studio Test Explorer, and Visual Studio Code Test Explorer.

Concerning the comparisons to VSTest, as reported the MSTest Runner offers several advantages like portability, performance, reliability and extensibility.

In terms of portability, it simplifies test execution by running tests directly from an executable, eliminating complexities associated with traditional test infrastructure. With MSTest Runner projects are treated as regular entities, enabling developers to leverage existing dotnet tooling and run tests on multiple computers without additional setup.

It is announced that there is a plan for NativeAOT support. Also, on GitHub developers can explore the sample for running tests against a dotnet application hosted in a docker container that has no dotnet SDK available.

Furthermore, Performance-wise, the MSTest Runner will optimise resource usage on build servers, utilizing one less process and reducing inter-process serialized communication. As stated by the development team behind it, the internal Microsoft projects adopting this runner observed substantial savings in CPU and memory usage, with some projects completing tests three times faster while using four times less memory compared to the dotnet test.

Continuing with performance, there is an interesting question by user Michael Dietrich related to performance and its improvements raised attention. The question received a very detailed answer from Jakub Jareš, and it is highly recommended for readers to read through it.

On the other hand, the MSTest Runner introduces new defaults prioritizing safety and reducing the likelihood of accidentally missing test execution. Also, because of its architecture, without folder scanning, dynamic loading, or reflection for extension detection, guarantees consistent behaviour in both local and CI environments. Following, the runner's asynchronous and parallelizable design minimizes hangs or deadlocks, addressing common issues observed when using VSTest.

Furthermore, regarding the extensibility the MSTest Runner offers a flexible model, allowing users to extend or override various aspects of test execution. This model supports custom report generators, test orchestration, loggers, and additional command-line options. Microsoft provides a list of optional extensions, with a statement of ongoing efforts to enhance the library with more features.

The original blog post announcement carries a very active comments section, between community members and the development team. The community feedback was overall very positive and active with questions and reactions.

A user called, Michael Taylor asked about the future of Test Explorer in Visual Studio and pipelines utilizing VS Test and the dotnet CLI for running tests. Jakub Jareš responded comprehensively, offering detailed insights and outlining plans for these tools, stating that the recommendation is to use the MSTest runner for new MSTest projects. There are plans to update MSTest templates to automatically utilize the MSTest runner as the default option. Additionally, a parameter, --disable-runner, will be provided for those who wish to opt out of using the MSTest runner.

Additionally, user Michael Dietrich raised another question about the compatibility of the runner with other test frameworks. The response indicated that, while the runner currently supports only MSTest, it was intentionally built on framework-agnostic building blocks.

The decision was made to focus on MSTest, which provides greater flexibility, ease of backward compatibility, and quicker, more confident implementation of changes due to the team's familiarity with the MSTest codebase. Developers are also encouraged to vote and give their feedback regarding the other test frameworks.

Lastly, readers who are interested in more project details and updates can explore the official GitHub repository.

About the Author

Rate this Article