Coding, Testing

ApprovalTests and MVC Views: Tackling the Build Server

In part one of ApprovalTests and MVC Views, Getting Started, I covered the basic mechanics of getting an MVC View under test.  In part two, Working with Data, I showed how you can create seams in your MVC project that allow you to avoid your repository and inject test data into the View.  These tests were more interesting that the tests in part one, and once you know where and how to create your seams for testing MVC views, you can really get into the groove covering your view with tests.  In this article we’ll look at running our tests on a build server.  Trying to run the tests on the build server will reveal one of the drawbacks of our dependency on the webserver.  But, we’ll see how to overcome it.

To review, we started this example with File->New Project and our goal is to create a set of tests that run consistently under Visual Studio, NCrunch, and CC.NET.  All of the source code is available as a mercurial repository on bitbucket.org.  If you haven’t already watched it, take 15 minutes to check out Llewellyn Falco’s video: Using ApprovalTests in .Net 20 Asp.Mvc Views.

Automated View Testing

You can build upon the techniques described in the previous two articles to build tests appropriate for your project. And you toddle along happily for a while writing your tests and code, until you hit the next speed bump when you try to send your code up to the build server.  After reading this article, you’ll know what to do to get your tests running on the server, and I hope it won’t seem like a speed bump to you at all. For now, lets just charge ahead like we don’t know any better and send our code to the build server.

I’m using CC.NET and other than the fact that I’m pretty sure its raising my electric bill my only complaint is that the configuration system is a little clunky. Maybe there’s better stuff out there but for now CC.NET is meeting my needs.  I configured the build server to keep an eye on MVCTestSite’s repository on bitbucket.org and fetched the test results once it was done. All the tests passed except for the view tests. Here’s our error:

Test method MVCTestSite.Tests.Views.HomeViewsTest.TestAboutView threw exception:

System.Exception: The following error occurred while connecting to:

http://localhost:61586/Home/About

Error:

Unable to connect to the remote server —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:61586

Looks familiar. It’s the second gotcha all over again, the web server is not running.  On the build server we don’t have the option of clicking “Start Without Debugging” in Visual Studio.  Even if we could, we want our build server to be completely automated.  If the build process requires us to do something before we can test, it’s broken.

Lucky for us, CassiniDev is a fantastic open source project well suited for solving this problem (emphasis original):

The goal of the CassiniDev project is to provide an open platform for developing a robust ASP.Net web server implementation that addresses many of the limitations and difficulties encountered when using Cassini and/or Visual Studio Development Server.
CassiniDev is packaged as a standalone WinForms GUI application, a console application and library assembly suitable for self hosting and in automated testing scenarios including continuous integration and as a 100% compatible drop-in replacement for the Visual Studio 2008/2010 development server.

Sounds like just the thing we need. CassiniDev can take the place of IIS Express or Cassini, but since we want to run this server from a continuous integration system, we want to use the self-hosting option. We’ll do this by taking a reference to CassiniDev in our test project, starting the server during test setup, and stopping the server during teardown. With CassiniDev to serve our pages, we should be able to get our tests working in all three build environments.

Install CassiniDev using NuGet.

mvcviewtest7

Once we have a reference to CassiniDev, test classes that contain web requests need to extend the CassiniDevServer class. All of our view tests (some real and some imagined) already extend our own MvcTest base class to perform the PortFactory setup, so by extending CassiniDev server with MvcTest, we should be able to add CassiniDev support to all of our View tests at once.

Import the CassiniDev namespace and extend CassiniDevServer:

using CassiniDev;

namespace MVCTestSite.Tests.Views
{
  public class MvcTest : CassiniDevServer
  {
    public MvcTest()
    {
      PortFactory.MvcPort = 61586;
    }
  }
}

Next, add setup and teardown to MvcTest. In MSTest these methods are identified by the TestInitalizeAttribute and the TestCleanupAttribute. If we try to use these attributes in a class not marked with the the TestClassAttribute, we’ll get a warning, so we’ll add that attribute as well. You can make your life a little easier by writing the teardown method first. Remember, NCrunch is watching you, and if you write the setup method first, NCrunch will execute it and start the server. I found that this ended up locking some files in the bin directory and I had to restart Visual Studio. So, write the cleanup first.

[TestCleanup]
public void Cleanup()
{
  StopServer();
}

Now write the setup:

[TestInitialize]
public void Setup()
{
  StartServer(@"..\..\..\MVCTestSite", PortFactory.MvcPort, "/", "localhost");
}

The minimum information needed by CassiniDev is the path to the web application folder. With that info in hand, it can find an open port to host the site on and start. Since we are using ApprovalTests, we can’t allow CassiniDev to pick its own port, since changing the expected port would cause the tests to fail in a variety of ways. First, VerifyMvcPage wont be hitting the right port, so we’ll get a 500 error.  CassiniDev provides a method to generate paths with the correct port and we could use this method along with VerifyUrl to reach our MVC actions. However, this wont work for us because the port number is actually in the HTML we receive with ApprovalTests, so we need that port to remain consistent for comparison with the approved file.

CassiniDev allows us to choose the port using an overload to StartServer, so that’s what we do, and we pass in the port that we earlier assigned to the PortFactory. While we’re here, we should also change the port. This breaks our tests momentarily, but we don’t want to use the same port that IIS Express or Cassini are configured to use, we may have started one of those servers for debugging and CassiniDev cannot start if the port is already occupied.

Lets just pick the next port up:

public MvcTest()
{
  PortFactory.MvcPort = 61587;
}

Now CassiniDev can start and is able to serve our pages, our ApprovalTest fails as expected because we changed our port. File launcher reporter shows us some dramatic differences from what we have seen until now:

mvcviewtest8

All of our styles appear to be missing. By the time that the browser loads the received file, CassiniDev has already shut down and is not available to serve the request of the CSS data.  All that we see is raw HTML without any styles.

Although this isn’t the reason the test failed, it’s an important caveat when using CassiniDev and ApprovalTests together.  If you are using ApprovalTests to get feedback while making changes, you should probably use a local server like IIS Express, Cassini, or even the WinForms variation of CassiniDev.  This way you can see your changes with the styles applied and feel confident that your design is correct.  Once you have approved your changes, and you’re using ApprovalTests to detect regression, you can switch back to the self-hosted variant of CassiniDev.  You certainly need to switch back to self-hosted CassiniDev before sending your tests to the build server.

With that discussion in mind, lets switch to a DiffReporter and see what’s really changed in this file.

mvcviewtest9

First, we notice that the port is different. That change was expected so we can just use the diff utility to move the new line over to the approved file. But we also see that the welcome message changed because I switched computers after approving this file the first time. The greeting is part of the default MVC template that isn’t really important to me, so I simply delete this block from the approved file:

<div id="logindisplay">
  Welcome <strong>Starbuck\Jim Counts</strong>!
</div>

I also need to visit the master page and delete the same block. Now, if you require a personalized greeting then it is actually possible to make a strongly typed master page with a model, and you could introduce seams to control that model. I’ve also read that its possible to pass this data in using the ViewData dictionary, but I haven’t tried it. Check out SO2821488 if you’re interested in going that route.

After updating the master page to match the new expectations in my approved file, I run the tests again, expecting them to pass. Index works now, but About is impacted by the change I made to the master page, so I need to reapprove its output. I run the tests again in Visual Studio and they all pass, but NCrunch still seems unhappy. Looking at the output from NCrunch I can see that it’s getting 500 errors when it tries to run the View tests. It turns out that this line is the culprit:

[TestInitialize]
public void Setup()
{
  StartServer(@"..\..\..\MVCTestSite", PortFactory.MvcPort, "/", "localhost");
}

This relative path to the web application folder is not valid from NCrunch’s workspace. Nasty annoyance, or valuable feedback? In my opinion, this is valuable feedback, NCrunch has uncovered an invalid assumption. Its possible that this assumption would also be invalid on the build server. Having it break in NCrunch gives me the opportunity to make changes now, while it’s a little more convenient because I don’t have to inspect the failure in the build server logs. The first time I encountered this problem I solved it using a brute force approach to locate the MVCTestSite folder, but I knew this had a bad code smell and when I showed it to Llewellyn, he immediately proposed a more elegant solution.

So let’s skip the brute force and look at the elegant solution. Here is what we would like our test setup to look like:

[TestInitialize]
public void Setup()
{
  StartServer(MvcApplication.Directory, PortFactory.MvcPort, "/", "localhost");
}

It would be nice if MvcApplication would just tell us where it is on the disk. We can make it do so by leveraging ApprovalUtilities once more. MvcApplication lives in Global.asax.cs, the “code behind’ for Global.asax. Go there import this namespace:

using ApprovalUtilities.Utilities;

Then declare a read-only property called Directory with this implementation:

public static string Directory
{
  get
  {
    return PathUtilities.GetDirectoryForCaller();
  }
}

As soon as I finish implementing this property, NCrunch turns green. Looks like my tests are working again. I run them in Visual Studio and confirm.

Another Note for NCrunch Users

When using NCrunch and CassiniDev with a statically defined port, your tests may fail with an error like this:

Initialization method MVCTestSite.Tests.Views.HomeViewsTest.Setup threw exception. System.Exception: System.Exception: Port 61587 is in use..

This is caused by a race condition and will only happen if NCrunch and the Visual Studio test runner are executing the tests concurrently. Sometimes NCrunch wins the race and occupies the port, and other times Visual Studio wins the race. Once the winner finishes testing, you can run the loser again and the tests should pass.

For the time being this is just a minor inconvenience if you know what’s going on. Because our port number is part of the approved file, we can’t just use a compiler condition to give a different port to NCrunch, we have to let NCrunch try to use the same port. On the bright side, this is really only a problem on the developer’s workstation. NCrunch does not run on the build server, so the test agent running in that environment should be able to get exclusive access to the port whenever it runs.

Does It Work Now?

Yes, it works now on the build server without any further changes. When I commit the changes, the build server fetches the new code, and turns green.  So now we have three green lights: Visual Studio, NCrunch, and CC.NET.  We have a mix of tests on the Models, the Views, and the Controllers, some go through the web server and some are Plain-Old-Tests.  When we want to test Views with fake data, we need to use seams inside the MVC project.  When we want fully automated regression tests, we can use CassiniDev, and when we want to design with feedback, we can use a persistent server, but we have to start it ourselves.

If you’ve read this far, I congratulate you. This was a long read, but hopefully it gets you up to a plateau where you’re no longer confused by MvcApprovals and it all seems simple. If you do have any more questions leave a comment, or tweet with the hash-tag #ApprovalTests. Llewellyn watches that feed closely, and I’m usually lurking on it as well.  Better yet, if you get it working, please consider helping other developers by writing about your experiences and sharing online.

Thanks for reading and good luck.

One thought on “ApprovalTests and MVC Views: Tackling the Build Server

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s