Introduction to Acceptance Testing, Gherkin and Spec Flow
Acceptance or functional testing is a type of testing where a system is tested to see if the required specifications are met. These tests are a type of black-box testing where the internal implementation is irrelevant. The acceptance tests only check to see if the system abides by the specification.
Look at the specification below for a login feature for a website:
Feature: Login
In order to access my account
As a user of the website
I want to log into the website
Scenario: Logging in with valid credentials
Given I am at the login page
When I fill in the following form
| field | value |
| Username | xtrumanx |
| Password | P@55w0Rd |
And I click the login button
Then I should be at the home page
Pretty readable, isn't it? The above specification is written using the Gherkin language. Gherkin is a domain specific language that lets us describe how our applications should behave without having to explain implementation details. Most of the above specification is free-text; there are only a few Gherkin specific keywords: Feature, Scenario, Given, When, And and Then. Everything else is free-text and is practically our documentation of how the feature is used.
Gherkin is a line-oriented language with each line in the scenario called a step. The first step in the "Logging in with valid credentials" scenario is "Given I am at the login page". The step needs a step definition so our test runner will know how to accomplish this step. A step definition in Spec Flow is simply a method with an attribute that contains the text of the step. All step definition methods need to be within a class that has a Binding attribute.
[Binding]
class LoginStepDefinitions
{
[Given("I am at the login page")]
public void GivenIAmAtTheLoginPage()
{
// TODO
}
}
The above class and method names are arbritrary. What's important are the attributes applied to the class and method. Without them, Spec Flow can't figure out which step definition methods should be bound to which steps.
All that's left is the step definition implementation. That's where WatiN and NUnit come into the picture.
WatiN is a browser automation tool. We're going to be using it to open an instance of IE, navigate to URLs, fill forms, click buttons or links, etc. We'll be using NUnit to assert our expectations. WatiN and NUnit are not compulsory; you could use Selenium for browser automation and virtually any unit testing framework for your assertion. You could even use Windows application automation library, like White, and write automated acceptance tests for your Windows Forms or WPF applications.
Let's try building an actual acceptance test for an actual application. For this article, we'll be using this sample application. Grab a copy of the application repository here. The repository also contains the completed acceptance test project, though I'd suggest creating your own acceptance test code by following along with the rest of this article.
Prerequisites
Spec Flow delegates the heavy lifting of actually running the acceptance tests to any of the supported 3rd party test runners. As mentioned earlier, we'll be using NUnit to run our tests and WatiN to automate our browser. Here's an example (from the WatiN website) on how you can use WatiN to automate the browser to perform a google search for WatiN.
[Test]
public void SearchForWatiNOnGoogle()
{
using (var browser = new IE("http://www.google.com"))
{
browser.TextField(Find.ByName("q")).TypeText("WatiN");
browser.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(browser.ContainsText("WatiN"));
}
}
The test above creates a new instance of IE and passes Google's URL to the constructor which will make the browser go to Google. Then a text field with the name of 'q' is searched for. That text field happens to be the one you type your search query into. The text field that is found then has the word 'WatiN' typed into it. Next, a button with the name of 'btnG' is searched for and is clicked. Finally, an assert is made to confirm that the text 'WatiN' exists (anywhere) on the page.
The above code gives a glimpse on how easy it is to automate WatiN to do regular tasks you'd want to perform on the browser window like filling in textboxes and pressing buttons.
Next, go ahead and create a new class library project in Visual Studio for your acceptance tests. You'll need to add the necessary DLLs to your acceptance tests project once you've downloaded NUnit and WatiN. Add a reference to nunit.framework.dll from your NUnit download to your acceptance tests project. For WatiN, you'll need to add references to two DLLs into your acceptance tests project; Interop.SHDocVw.dll and WatiN.Core.dll.
I should mention you could get both projects via NuGet. They're both easily discoverable via NuGet which will automatically add them to your project so if you already use NuGet in your projects, you might as well use it to download NUnit and WatiN.
Grab a copy of Spec Flow and install it into your system. Unlike NUnit and WatiN, you'll need to install Spec Flow rather than simply copy DLLs onto your system. Spec Flow comes with some tooling and whenever you add a feature file to your project it creates a code-behind file for it. Plus there's some syntax highlighting and other tweaks available when editing your feature files.
After successfully installing Spec Flow, check out the installation directory (defaulting to the Program Files). There's a bunch of DLLs within but only one needs to be added to your project as a reference; TechTalk.SpecFlow.dll.
Setting Up Your Acceptance Tests Project
Before we dive into writing our acceptance tests, we're going to need to setup our acceptance tests project. Within the project we're going to add a few folders just to keep things organized.
- Features
- This is where we'll be keeping all our specifications.
- StepDefinitions
- The step definitions for our scenario steps will be placed here.
- StepHelpers
- All helper classes will be added here.
Creating a Spec Flow feature file
Add a new Spec Flow feature file named Login.feature into the Features folder. It's going to come with the specification for an Addition feature with one scenario. Remove all of it and add the following text into our Login feature.
Feature: Login
Feature is a keyword in Gherkin. It should appear once in every feature file. It is followed by a colon and the title of the feature. You can then use any number of lines of free-text to describe the feature. To keep things simple, Gherkin's creators recommended keeping your documentation short and to follow the following format:
In order to realize a named business value
As an explicit system actor
I want to gain some beneficial outcome which furthers the goal
Personally, I tend to skip the above description for self-descriptive features like the Login feature we're working on it can get confusing trying to write your description to fit the specified format; sometimes it isn’t clear what the difference is between the "named business value" in the first line and the "beneficial outcome" you’re supposed to write in the third line. Remember, it's all free-text and you can write it however you want to. Let's skip it for now and write our scenario.
Feature: Login
Scenario: Logging in with valid credentials
Like Feature, Scenario is a Gherkin keyword and is followed by a colon and the title of the scenario. Unlike a feature, a scenario isn't complete in just one line but must be followed up with the steps needed to complete the scenario. Let's think it through; what do we need to do to successfully login with valid credentials?
- Fill in the login form
- Click the login button
But hold on, before we can fill in the login form, we need to be at the page the login form is presented. Afterwards, we're going to have to check that we've successfully logged in. Let's assume that if we've been redirected to the home page after logging in, that means we've been successfully logged in.
Now we have a pre-condition for our scenario (i.e. we must be at the login page) and a post-condition (i.e. we're at the home page). In Gherkin, pre-conditions should start with the keyword Given and post-conditions should start with the keyword When.
Feature: Login
Scenario: Logging in with valid credentials
Given I am at the 'Login' page
When I fill in the following form
| field | value |
| Username | testuser |
| Password | testpass |
And I click the 'Login' button
Then I should be at the 'Home' page
Notice the table-like structure. Spec Flow will automatically italicize the first row in a pipe-delimited line that begins with a pipe. That first row is considered the header of the table. Each column in subsequent rows is referred to by whatever text was used in the header row. For instance, the text Password is in the field column of the second row and the text testuser is in the value column of the first row.
Also, notice the step that begins with the word And. The And keyword can be used after any step and will automatically be assumed to be of the same type of the previous step. An And step that follows a Given step is also considered a Given step. In the above example, our And step is considered a When step. When steps are not to be used for pre or post conditions but for the actual steps needed to run the scenario.
Create your first step definition
Right now we've got our specification for our Login feature complete but our test runner has no idea on how execute each step in our feature's scenario. We'll need to define the step definitions for each of our 4 steps in our Login feature's scenario. To do so, we're going to create a class within the Steps directory. We're call this class LoginSteps. To let Spec Flow know that this class contains step definitions, we're going to apply the Binding attribute to the class. The Binding attribute is part of the TechTalk.SpecFlow namespace.
using TechTalk.SpecFlow;
[Binding]
class LoginSteps
{
}
Next, we'll need to make a method for each of our steps. This method will let Spec Flow know how to execute each of our steps. For now, we'll just be providing a Step Definition for our first step, "Given I am at the 'Login' page"
[Given("I am at the 'Login' page")
public void GivenIAmAtTheLoginPage()
{
// TODO
}
Notice the method has an attribute applied to it. This attribute lets Spec Flow know this method maps to the step that matches the attribute. Any step in any feature file beginning with the Given keyword and is followed by the text "I am at the 'Login' page" is going to be mapped to this method.
Now we need to write the implementation for this step definition. What we need to do is to tell WatiN to startup the browser and visit our app's login page. Before we do so, we're going to need to create an instance of a browser. We're also going to need to make sure the same instance of the browser is used for all the remaining steps in our scenario. To make sure we use the one browser instance for all our scenario's steps, we're going to create an instance of a browser object and store it in the ScenarioContext dictionary. The ScenarioContext dictionary can be used to store data during the execution of a Scenario. We're going to create a helper class called WebBrowser which will hold the browser instance that will be used during the execution of the scenario.
using TechTalk.SpecFlow;
using WatiN.Core;
static class WebBrowser
{
public static IE Current
{
get
{
if(!ScenarioContext.Current.ContainsKey("browser"))
ScenarioContext.Current["browser"] = new IE();
return ScenarioContext.Current["browser"] as IE;
}
}
}
Our helper class above has a Current property which fetches the current browser for the currently executing scenario. If it doesn't find a browser instance contained in the ScenarioContext dictionary, it creates a new browser instance and adds it to the dictionary. Afterwards, the browser instance contained in the dictionary is returned.
Finally, we can go back and implement our step definition. In our example application, the login page is located at http://localhost:9876/authentication/login. We're going to implement our step definition by making our current scenario's browser instance navigate to that URL. We could implement our step definition by making our browser instance go to the home page and click the login link but this seems simplest for now. Later on, we're going to refactor our step definitions so we can have one step definition handle navigation to any page of our application.
[Given("I am at the 'Login' page")
public void GivenIAmAtTheLoginPage()
{
// Make sure to add the namespace the WebBrowser class is inside
WebBrowser.Current.GoTo("http://localhost:9876/authentication/login");
}
Loose Ends
We're now ready to attempt to run our acceptance test. We've yet to complete all the step definitions but we want to make sure we've got everything wired up correctly before carrying on. Before running the test, make sure you've got all the assemblies we've added to our project set to Copy Local. The Interop.SHDocVw DLL will have its Embed Interop property set to true if you've added WatiN via NuGet so make sure you set the Embed Interop property to false so you can set its Copy Local property to true.
We'll also need to make NUnit run in a single-threaded apartment state otherwise we won't be able to automate Internet Explorer. We're using IE instead of Firefox because Firefox's constant major version changes keeps breaking WatiN's Firefox hook.
Setting NUnit's apartment state will require the use of a configuration file. Add an app.config file to the project and add the following configuration to it.
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<sectionGroup name="NUnit">
<section name="TestRunner" type="System.Configuration.NameValueSectionHandler"/>
</sectionGroup>
</configSections>
<NUnit>
<TestRunner>
<!-- Valid values are STA,MTA. Others ignored. --> <add key="ApartmentState" value="STA" /> </TestRunner>
</NUnit>
</configuration>
Run Your Acceptance Test
Your Login.feature file has a code-behind file called Login.feature.cs (or .vb). That file contains the class with the TestFixture attribute required so NUnit can figure out which class to test. If you've got NUnit test runner plug-in installed in Visual Studio, like TestDriven.Net or ReSharper, Login.feature.cs is the file you're going to ask it to run the tests for. If you don't have a test runner plugin that can do so, open the test runner that comes with the NUnit installation located in your programs menu and point it to the assembly generated by our acceptance test project and it'll locate the test fixture located within Login.feature.cs
Make sure the application's server is up and running then run your tests. A new instance of IE should open up and navigate to the login page. If it does, everything is wired up correctly and we can move on to completing the rest of our step definitions. If IE didn’t open up and navigate to the login page, it's time to roll up your sleeves and debug.
Complete the rest of your step definitions
Writing the rest of the step definitions is going to be simple. You'll need to know a thing or two about the WatiN API so you can figure out how to tell WatiN to fill a form or how to find a button or link and click it.
First, let's try doing the second step in our scenario, "When I fill in the following Form". You'll need to create a method in your LoginSteps class and given it an attribute to bind it to the step we're currently working on.
[When("I fill in the following form")]
public void WhenIFillInTheFollowingForm(TechTalk.SpecFlow.Table table)
{
// TODO
}
Notice the parameter we passed into this step definition. This Table object will contain the values we've writtern in our scenario file. WatiN also has a Table class in its namespace so to avoid conflict I've used the fully qualified name for this Table. The Table object will be made up of rows. Each column in the row can be accessed by index or column name. Our column names are defined by the first row of the table within the scenario file.
[When("I fill in the following form")]
public void WhenIFillInTheFollowingForm(TechTalk.SpecFlow.Table table)
{
foreach(var row in table.Rows)
{
var textField = WebBrowser.Current.TextField(Find.ByName(row["field"]));
if(!textField.Exists)
Assert.Fail("Expected to find a text field with the name of '{0}'.", row["field"]);
textField.TypeText(row["value"]);
}
}
That's it. In 5 lines of code, we've written our step definition for filling out a web form. We first loop through the rows within the table; then we attempt to find text fields with a name attribute matching the current row's field column value. If we don't find a matching text field, we fail the test explicitly (and leave a helpful message for ourselves). If we do find a matching text field, we instruct WatiN to type in the text we find in the current row's value column.
The rest of our step definitions will be written similarly. Here's the code for our next step, "And I click the 'Login' button".
[When("I click the 'Login' button")]
public void AndIClickTheLoginButton()
{
var loginButton = WebBrowser.Current.Button(Find.ByValue("Login"));
if(!loginButton.Exists)
Assert.Fail("Expected to find a button with the value of 'Login'.");
loginButton.Click();
}
For our last step, we need to figure out how to test if we are at the home page. We could inspect the document's title and see if it matches our expected title. Or perhaps we can inspect the URL and see if it matches the expected URL for the homepage. We'll inspect the URL, but keep in mind the implementation of our step definitions can depend entirely on our application. If we're making heavy use of Ajax to navigate around pages, the URL may not be updated and thus may not be useful to confirm if we are in fact on the correct page.
[Then("I should be at the 'Home' page")]
public void ThenIShouldBeAtTheHomePage()
{
var expectedURL = "http://localhost:9876/";
var actualURL = WebBrowser.Current.Url;
Assert.AreEqual(expectedURL, actualURL);
}
Run your acceptance test again and this time it should run to completion. If it did, congratulations; you've written and successfully passed your first acceptance test using Spec Flow.
Refactor Your Test Code
Now that we have our step definitions working, let's take a moment and review them. We've got a step definition for clicking a login button. We're probably going to have a ton of other buttons in our applications. Are we going to have step definitions for every button involved in our acceptance test suite? The only thing that will be changing for each of those step definitions will be the text that WatiN will search for when finding the expected button on the page. Here's an example:
[When("I click the 'Login' button")]
public void AndIClickTheLoginButton()
{
var loginButton = WebBrowser.Current.Button(Find.ByValue("Login"));
if(!loginButton.Exists)
Assert.Fail("Expected to find a button with the value of 'Login'.");
loginButton.Click();
}
[When("I click the 'Register' button")]
public void AndIClickTheRegisterButton()
{
var registerButton = WebBrowser.Current.Button(Find.ByValue("Register"));
if(!registerButton.Exists)
Assert.Fail("Expected to find a button with the value of 'Register'.");
registerButton.Click();
}
Luckily, Spec Flow has a way to avoid this problem. You can use regular expressions within the string passed to the binding attribute so you can have different steps bound to the same step definition. Any text captured by the regular expression can then be passed to the step definition method as a parameter. Here's an example:
[When("I click the '(.*)' button")]
public void AndIClickAButton(string buttonText)
{
var button = WebBrowser.Current.Button(Find.ByValue(buttonText));
if(!button.Exists)
Assert.Fail("Expected to find a button with the value of '{0}'.", buttonText);
button.Click();
}
The above step definition will match any of the following steps:
- And I click the 'Login' button
- And I click the 'Register' button
- And I click the 'Some Arbitrary Text' button
You can use this technique to make all your step definitions reusable. It is a good idea to make all your step definitions reusable and you should try to avoid writing step definitions tied only to one specific scenario. This will allow you to write scenarios that work immediately as you would have a reusable step definition already available. Don't try getting ahead of yourself and writing reusable step definitions before you need them though. Just write them when you need them. For the most part, your tests will be doing pretty much the same thing; navigating to a page, filling out a form, clicking something, etc. You'll rarely need to write a specific step definition for a scenario unless you testing something out of the ordinary like checking if the jQuery UI calendar popped up after click a textbox that needed a date value.
I'll leave it as an exercise to you to refactor your step definitions so each of them is reusable. After doing so, you could write a new feature file navigate to a link and submits a form that would just work immediately since the step definitions needed are already prepared. Of course, you're probably going to need more step definitions to automate different actions (like clicking a link) and you may need to refine your current step definitions to make them work in different scenarios. You'll notice our form filling step definition only handles filling a form with textboxes. If you want to tick checkboxes, find buttons by their value or by ID, or do other form related actions, you'll need to add the relevant logic to the form filling step definition.
Conclusion
Well, that's it for this tour of writing acceptance tests using Spec Flow. There's more to discover on your own, however. For instance, Spec Flow also has a tagging mechanism so you can tag a specific feature to run some code before or after a specific feature, scenario or step. This is useful if you want to logout after completing a scenario involving login, priming the database before the tests or simply want to close the browser window after each test completes.
As you create more tests, you'll notice the amount of time it takes to complete your acceptance test suite increases. The quicker your tests run, the faster you can get feedback and find bugs. You want your tests to run as fast as possible. One approach you can take is to parallelize your tests so instead of running sequentially, they run simultaneously and finish sooner. Spec Flow doesn't come with parallelization functionality so you'll have to look elsewhere. If you're using NUnit, you may want to look into PNUnit for your parallelization needs.
Gherkin, the DSL used to write our specifications, was designed to bridge the gap between your technical and non-technical stakeholders so they are able to come to an agreement on how a particular feature should work. Some have gone as far as to get their non-technical stakeholders to write the actual specification using the Given-When-Then syntax. As you can imagine, it wouldn't be that hard to train folks to do so, but your mileage may vary.
Others have taken their use of Spec Flow further. They start all new feature development by writing the specification first, then move on to their unit tests, and finally the actual code. They then work through the usual TDD red-green-refactor cycle until their unit tests pass and finally their acceptance tests pass. If you're already practicing TDD, you should give Acceptance Test Driven Development a go if you are planning to write acceptance tests for your project.
Whichever path you take, make sure it works for you and your project. If you're prototyping and constantly changing how your application works, keeping your acceptance test updated may hold you back. It's up to you to decide when the right time to write your acceptance tests; just don't let it be never.
About the Author
Mustafa Saeed Haji Ali lives in Hargeisa, Somaliland. He is a software developer that usually works with ASP.NET MVC. Mustafa enjoys testing and using JavaScript frameworks like KnockoutJS, AngularJS and SignalR and has a passion for evangelizing best practices.