Coding, Testing

ApprovalTests and MVC Views: Tackling the Build Server

In part one of ApprovalTests and MVC Views, Getting Started, I covered the basic mechanics of getting an MVC View under test.  In part two, Working with Data, I showed how you can create seams in your MVC project that allow you to avoid your repository and inject test data into the View.  These tests were more interesting that the tests in part one, and once you know where and how to create your seams for testing MVC views, you can really get into the groove covering your view with tests.  In this article we’ll look at running our tests on a build server.  Trying to run the tests on the build server will reveal one of the drawbacks of our dependency on the webserver.  But, we’ll see how to overcome it.

To review, we started this example with File->New Project and our goal is to create a set of tests that run consistently under Visual Studio, NCrunch, and CC.NET.  All of the source code is available as a mercurial repository on bitbucket.org.  If you haven’t already watched it, take 15 minutes to check out Llewellyn Falco’s video: Using ApprovalTests in .Net 20 Asp.Mvc Views.

Automated View Testing

You can build upon the techniques described in the previous two articles to build tests appropriate for your project. And you toddle along happily for a while writing your tests and code, until you hit the next speed bump when you try to send your code up to the build server.  After reading this article, you’ll know what to do to get your tests running on the server, and I hope it won’t seem like a speed bump to you at all. For now, lets just charge ahead like we don’t know any better and send our code to the build server.

I’m using CC.NET and other than the fact that I’m pretty sure its raising my electric bill my only complaint is that the configuration system is a little clunky. Maybe there’s better stuff out there but for now CC.NET is meeting my needs.  I configured the build server to keep an eye on MVCTestSite’s repository on bitbucket.org and fetched the test results once it was done. All the tests passed except for the view tests. Here’s our error:

Test method MVCTestSite.Tests.Views.HomeViewsTest.TestAboutView threw exception:

System.Exception: The following error occurred while connecting to:

http://localhost:61586/Home/About

Error:

Unable to connect to the remote server —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:61586

Looks familiar. It’s the second gotcha all over again, the web server is not running.  On the build server we don’t have the option of clicking “Start Without Debugging” in Visual Studio.  Even if we could, we want our build server to be completely automated.  If the build process requires us to do something before we can test, it’s broken.

Lucky for us, CassiniDev is a fantastic open source project well suited for solving this problem (emphasis original):

The goal of the CassiniDev project is to provide an open platform for developing a robust ASP.Net web server implementation that addresses many of the limitations and difficulties encountered when using Cassini and/or Visual Studio Development Server.
CassiniDev is packaged as a standalone WinForms GUI application, a console application and library assembly suitable for self hosting and in automated testing scenarios including continuous integration and as a 100% compatible drop-in replacement for the Visual Studio 2008/2010 development server.

Sounds like just the thing we need. CassiniDev can take the place of IIS Express or Cassini, but since we want to run this server from a continuous integration system, we want to use the self-hosting option. We’ll do this by taking a reference to CassiniDev in our test project, starting the server during test setup, and stopping the server during teardown. With CassiniDev to serve our pages, we should be able to get our tests working in all three build environments.

Install CassiniDev using NuGet.

mvcviewtest7

Once we have a reference to CassiniDev, test classes that contain web requests need to extend the CassiniDevServer class. All of our view tests (some real and some imagined) already extend our own MvcTest base class to perform the PortFactory setup, so by extending CassiniDev server with MvcTest, we should be able to add CassiniDev support to all of our View tests at once.

Import the CassiniDev namespace and extend CassiniDevServer:

using CassiniDev;

namespace MVCTestSite.Tests.Views
{
  public class MvcTest : CassiniDevServer
  {
    public MvcTest()
    {
      PortFactory.MvcPort = 61586;
    }
  }
}

Next, add setup and teardown to MvcTest. In MSTest these methods are identified by the TestInitalizeAttribute and the TestCleanupAttribute. If we try to use these attributes in a class not marked with the the TestClassAttribute, we’ll get a warning, so we’ll add that attribute as well. You can make your life a little easier by writing the teardown method first. Remember, NCrunch is watching you, and if you write the setup method first, NCrunch will execute it and start the server. I found that this ended up locking some files in the bin directory and I had to restart Visual Studio. So, write the cleanup first.

[TestCleanup]
public void Cleanup()
{
  StopServer();
}

Now write the setup:

[TestInitialize]
public void Setup()
{
  StartServer(@"..\..\..\MVCTestSite", PortFactory.MvcPort, "/", "localhost");
}

The minimum information needed by CassiniDev is the path to the web application folder. With that info in hand, it can find an open port to host the site on and start. Since we are using ApprovalTests, we can’t allow CassiniDev to pick its own port, since changing the expected port would cause the tests to fail in a variety of ways. First, VerifyMvcPage wont be hitting the right port, so we’ll get a 500 error.  CassiniDev provides a method to generate paths with the correct port and we could use this method along with VerifyUrl to reach our MVC actions. However, this wont work for us because the port number is actually in the HTML we receive with ApprovalTests, so we need that port to remain consistent for comparison with the approved file.

CassiniDev allows us to choose the port using an overload to StartServer, so that’s what we do, and we pass in the port that we earlier assigned to the PortFactory. While we’re here, we should also change the port. This breaks our tests momentarily, but we don’t want to use the same port that IIS Express or Cassini are configured to use, we may have started one of those servers for debugging and CassiniDev cannot start if the port is already occupied.

Lets just pick the next port up:

public MvcTest()
{
  PortFactory.MvcPort = 61587;
}

Now CassiniDev can start and is able to serve our pages, our ApprovalTest fails as expected because we changed our port. File launcher reporter shows us some dramatic differences from what we have seen until now:

mvcviewtest8

All of our styles appear to be missing. By the time that the browser loads the received file, CassiniDev has already shut down and is not available to serve the request of the CSS data.  All that we see is raw HTML without any styles.

Although this isn’t the reason the test failed, it’s an important caveat when using CassiniDev and ApprovalTests together.  If you are using ApprovalTests to get feedback while making changes, you should probably use a local server like IIS Express, Cassini, or even the WinForms variation of CassiniDev.  This way you can see your changes with the styles applied and feel confident that your design is correct.  Once you have approved your changes, and you’re using ApprovalTests to detect regression, you can switch back to the self-hosted variant of CassiniDev.  You certainly need to switch back to self-hosted CassiniDev before sending your tests to the build server.

With that discussion in mind, lets switch to a DiffReporter and see what’s really changed in this file.

mvcviewtest9

First, we notice that the port is different. That change was expected so we can just use the diff utility to move the new line over to the approved file. But we also see that the welcome message changed because I switched computers after approving this file the first time. The greeting is part of the default MVC template that isn’t really important to me, so I simply delete this block from the approved file:

<div id="logindisplay">
  Welcome <strong>Starbuck\Jim Counts</strong>!
</div>

I also need to visit the master page and delete the same block. Now, if you require a personalized greeting then it is actually possible to make a strongly typed master page with a model, and you could introduce seams to control that model. I’ve also read that its possible to pass this data in using the ViewData dictionary, but I haven’t tried it. Check out SO2821488 if you’re interested in going that route.

After updating the master page to match the new expectations in my approved file, I run the tests again, expecting them to pass. Index works now, but About is impacted by the change I made to the master page, so I need to reapprove its output. I run the tests again in Visual Studio and they all pass, but NCrunch still seems unhappy. Looking at the output from NCrunch I can see that it’s getting 500 errors when it tries to run the View tests. It turns out that this line is the culprit:

[TestInitialize]
public void Setup()
{
  StartServer(@"..\..\..\MVCTestSite", PortFactory.MvcPort, "/", "localhost");
}

This relative path to the web application folder is not valid from NCrunch’s workspace. Nasty annoyance, or valuable feedback? In my opinion, this is valuable feedback, NCrunch has uncovered an invalid assumption. Its possible that this assumption would also be invalid on the build server. Having it break in NCrunch gives me the opportunity to make changes now, while it’s a little more convenient because I don’t have to inspect the failure in the build server logs. The first time I encountered this problem I solved it using a brute force approach to locate the MVCTestSite folder, but I knew this had a bad code smell and when I showed it to Llewellyn, he immediately proposed a more elegant solution.

So let’s skip the brute force and look at the elegant solution. Here is what we would like our test setup to look like:

[TestInitialize]
public void Setup()
{
  StartServer(MvcApplication.Directory, PortFactory.MvcPort, "/", "localhost");
}

It would be nice if MvcApplication would just tell us where it is on the disk. We can make it do so by leveraging ApprovalUtilities once more. MvcApplication lives in Global.asax.cs, the “code behind’ for Global.asax. Go there import this namespace:

using ApprovalUtilities.Utilities;

Then declare a read-only property called Directory with this implementation:

public static string Directory
{
  get
  {
    return PathUtilities.GetDirectoryForCaller();
  }
}

As soon as I finish implementing this property, NCrunch turns green. Looks like my tests are working again. I run them in Visual Studio and confirm.

Another Note for NCrunch Users

When using NCrunch and CassiniDev with a statically defined port, your tests may fail with an error like this:

Initialization method MVCTestSite.Tests.Views.HomeViewsTest.Setup threw exception. System.Exception: System.Exception: Port 61587 is in use..

This is caused by a race condition and will only happen if NCrunch and the Visual Studio test runner are executing the tests concurrently. Sometimes NCrunch wins the race and occupies the port, and other times Visual Studio wins the race. Once the winner finishes testing, you can run the loser again and the tests should pass.

For the time being this is just a minor inconvenience if you know what’s going on. Because our port number is part of the approved file, we can’t just use a compiler condition to give a different port to NCrunch, we have to let NCrunch try to use the same port. On the bright side, this is really only a problem on the developer’s workstation. NCrunch does not run on the build server, so the test agent running in that environment should be able to get exclusive access to the port whenever it runs.

Does It Work Now?

Yes, it works now on the build server without any further changes. When I commit the changes, the build server fetches the new code, and turns green.  So now we have three green lights: Visual Studio, NCrunch, and CC.NET.  We have a mix of tests on the Models, the Views, and the Controllers, some go through the web server and some are Plain-Old-Tests.  When we want to test Views with fake data, we need to use seams inside the MVC project.  When we want fully automated regression tests, we can use CassiniDev, and when we want to design with feedback, we can use a persistent server, but we have to start it ourselves.

If you’ve read this far, I congratulate you. This was a long read, but hopefully it gets you up to a plateau where you’re no longer confused by MvcApprovals and it all seems simple. If you do have any more questions leave a comment, or tweet with the hash-tag #ApprovalTests. Llewellyn watches that feed closely, and I’m usually lurking on it as well.  Better yet, if you get it working, please consider helping other developers by writing about your experiences and sharing online.

Thanks for reading and good luck.

Advertisements
Coding, Testing

ApprovalTests and MVC Views: Working With Data

In part one of ApprovalTests and MVC Views, Getting Started, I covered the basic mechanics of getting an MVC View under test, but the pages I tested weren’t very interesting.  Those pages were dynamic only in the slimmest sense (they displayed the user identity).  In this article we will introduce some data models into our testing scenario so that we can see what testing more realistic pages look like.

Just a reminder, we started this example at File->New Project and our goal is to create a set of tests that run consistently under Visual Studio, NCrunch, and CC.NET.  All of the source code is available as a mercurial repository on bitbucket.org.  Finally, if you are interested in testing MVC Views, a great place to get started (other than this blog series) is by watching Llewellyn Falco’s video on the subject: Using ApprovalTests in .Net 20 Asp.Mvc Views.

Introducing Models

Designing and testing Models are not that different than testing any other .NET class, and smarter people than me have already covered the subject. I won’t go into details about the models for this example site, but if you are interested you can see them in the code repository that accompanies the article. The only feature of interest in this example’s model is the repository implementation. Our repository is never available, any attempt to read data from the default repository implementation will throw a NotImplementedException. This behavior is meant to simulate an unavailable SQL server, but NotImplementedExceptions are easier to generate than SQLExceptions.

After creating the models, I deleted the default Index view and replaced it with a strongly typed list view using the default tools in Visual Studio. The test immediately fails in NCrunch with a NullReferenceException because current controller implementation does not pass a Model to the new View. We have broken our Index View test, and also broken one of the original tests provided by Visual Studio.

Lets replace the broken Controller test with something that works, then move on to our View test. In the interest of readability, this test has been formatted to fit your screen:

[TestMethod]
public void Index()
{
  // Arrange
  Mock<ICarRepository> repositoryMock 
  repositoryMock = new Mock<ICarRepository>();
  repositoryMock.DefaultValue = DefaultValue.Mock;
  HomeController controller;
  controller = new HomeController(repositoryMock.Object);

  // Act
  controller.Index();

  // Assert
  repositoryMock.Verify(repo => repo.GetCars());
}

Since this is a Controller test we have the option to create an instance and manipulate its state. So I create a mock repository and pass it to the HomeController, call Index, and use the mock to verify that Index method is trying to get its data from the repository. Whether or not that data access call should be locked down by a test, or simply kept hidden as an implementation detail is up to you. The point of showing this example is to remind you that this is exactly the kind of thing you can’t do with a View test. Any changes to the Controller instance created in the test method will not be reflected in the View when it’s compiled on the web server and delivered to MvcApprovals.

In this example, I do want the data access to remain locked down so that this test will ensure me that I don’t break the default behavior as I create seams to get my View test to work. In “production” we want the call to Index to hit the repository and bring down the cars. Naturally, I had to make some changes to the HomeController after writing this test, and after overloading the constructor, adding a field for the repository, and putting the necessary method call into the Index method, the test passes, but our View test still fails.

We know that we have an uncooperative repository that always throws exceptions, and that might be a reason its failing. Sure enough, it is.

mvcviewtest4_thumb2

We need to resolve that issue before we can approve the new version of the index page. We can’t resolve the issue by creating Mocks in our test method. Remember that the controller instance we create in the test is just a convenient way to pass metadata to MvcApprovals. The real code that we’re testing is the code on the web server. The web server has its own instance of the HomeController, which it instantiated using the default constructor.  In our case, the default constructor happens to create an instance of the unusable repository. The key to testing this new Index view is to create a seam that lets us avoid the repository entirely.

Once we create seams, we can add MVC actions that pass fake data into the seams for us. These new actions have to be inside the production site, not the test project.  The web server needs to see and execute the test actions for us.  After we create the test actions, we update our View tests to use the test actions instead of the default actions.  Maybe now you’re screaming, “Dear God, first you try to convince me that taking a dependency on the web server isn’t an integration test, and now you want to put test code in production!” Well, you shouldn’t scream like that, you’re scaring the neighbors. Its just code, and we’ll take steps later to isolate this code so it can be easily excluded during Release builds.

Now that you’re done screaming, lets work our way inward until we’ve implemented the scenario described above. First, we’ll update our test so that it points to our unwritten test action in the MVC project.

[TestMethod]
public void TestIndexView()
{
  MvcApprovals.VerifyMvcPage(new HomeController().TestIndex);
}

Now we can create this method on our HomeController using our favorite refactoring tool. Before we can implement this action, we need to create a seam inside HomeController. Lets think about what the default Index method does:

public ActionResult Index()
{
  var cars = repository.GetCars();
  return View(cars);
}

Looks to me like TestIndex needs to do two things:

public ActionResult TestIndex()
{
  // Get the data
  // Pass data to the view.
}

Of these two parts, one part is the same between test and production, and one part varies. We need to separate the part that varies from the part that stays the same. In our simple scenario, that means moving the part that stays the same “Pass data to the view” into its own method so that both the production method and the test method can do what they want with the part that changes “Get the data”.

Here is the new Index method and the seam:

public ActionResult Index()
{
  var cars = repository.GetCars();
  return Index(cars);
} 
private ActionResult Index(IEnumerable<ICar> cars)
{
  return View(cars);
}

For reasons that will become apparent, the private seam must have the same name as the MVC action. We make the seam private to avoid ambiguity when MVC tries to resolve the route to the Index method.

We can create any model that we want in TestIndex, and pass it to the view using the private seam. The repository is not required inside the seam so we don’t care if it throws exceptions. In the test action we could use fakes or mocks, but we don’t even need to. We have access to our concrete models, and we can just insatiate them with any data we want. In this example, our Car implementation is just an ordinary class, but you can just as easily create EntityFramework or LINQ to SQL objects.

Before getting into that, lets isolate our test code by turning HomeController into a partial class and putting TestIndex into another file. Create HomeController.Tests.cs in your Controllers folder and put TestIndex in there:

public partial class HomeController
{
  public ActionResult TestIndex()
  {
    // Create data
    // Pass to view
    return null;
  }
}

Now lets create some test data and pass it to the seam.

public ActionResult TestIndex()
{
    // Create data
    IEnumerable<ICar> cars = new Car[] {
        new Car() { Make = "VW", Model = "Passat", Year = 2002 },
        new Car() { Make = "Honda", Model = "Fit", Year = 2007 },
    };

    // Pass to view
    return Index(cars);
}

That should be it. Our test should pass. But now its failing with a new error:

mvcviewtest5_thumb8

MVC is looking for a View called “TestIndex” even though our seam is called Index. The view name inference depends on the view engine implementation.  In the WebFormsViewEngine, the view name seems to be inferred from the virtual path as near as I can tell.  After reading the source code for awhile my brain started to hurt.  Anyway, whether MVC is inferring the data from the path, or getting it from the route data, we can override the behavior.  We’ll give MVC the view name using an overload that allows us to specify the name explicitly:

private ActionResult Index(IEnumerable<ICar> cars)
{
  return View("Index", cars);
}

This introduces a magic string into our code and we’d like to avoid that.  We can do so by taking advantage of a convenience method provided by the ApprovalTests companion library, ApprovalUtilities. We need to add a reference to ApprovalUtilities to our MVC project.  The easiest way to do that is to use NuGet.  If we use NuGet we will also get a reference to ApprovalTests.  If having a reference to ApprovalTests in your MVC project bothers you, don’t use NuGet.  You can always add the reference by hand if you like.   I’m going to use NuGet.

After adding the reference, import this namespace:

using ApprovalUtilities.Asp.Mvc;

Then update the private Index method again:

private ActionResult Index(IEnumerable<ICar> cars)
{
  return View(cars).Explicit();
}

This extension method is functionally equivalent to the magic-string version.

Note for NCrunch Users

Update 2012/06/18: You can skip this!  NCrunch v1.40b resolved this issue.  Cheers!

There is an issue using this extension method in the current version of NCrunch. On two of my machines, NCrunch will fail the Controller test (not the View test) when run with the Explicit method. I get this exception:

System.MissingMethodException: Method not found: ‘System.Web.Mvc.ViewResult ApprovalUtilities.Asp.Mvc.MvcUtilites.Explicit(System.Web.Mvc.ViewResult)’.

The documentation for MissingMethodException indicates that the exception is thrown when you try to dynamically access a method that does not exist.  For example, you use reflection to access a type, but then call InvokeMember with the name of a method that doesn’t exist.  I’ve looked at the assemblies generated by NCrunch, both with and without instrumentation, and its not trying to do this.  So I don’t know why we get this exception.  This occurs whether called as an extension method or as a regular static method call. This occurs in both Debug and Release configurations. This occurs when compiled for Any CPU and when compiled for x86. This occurs under MSTest and under xUnit.  It happens in 1.38b and 1.39b.

Researching this, I could only find one other person who looked like they had an issue that was remotely similar, but the similarity was very remote indeed and he hadn’t found a resolution yet. So for the time being I’m working around this issue by checking for NCrunch at compile time:

  private ActionResult Index(IEnumerable<ICar> cars)
  {
#if NCRUNCH
    return View("Index", cars);
#else
    return View(cars).Explicit();
#endif
  }

There are two other alternatives. First we can give up on using the extension method, and second we can give up on NCrunch. I don’t think we need to give up either. NCrunch is a great tool, but its still in beta. This is the first real issue I’ve had with it since the March 2012 release. I’m hopeful that this issue can be resolved with further research, or in a future version of the tool . This compile time condition will remind me to test this with NCrunch the next time an update is released, and to keep looking for an answer online.  When this issue is resolved I can get rid of the branch that uses the magic string.

Update 2012-05-18: 

I decided it might be a good idea to report this issue to the NCrunch developer, Remco Mulder, over on the NCrunch Forums.  We we were able to isolate the cause of the exception.  Our example site is built against MVC 3 and ApprovalUtilities is built against MVC 2.  The current version of NCrunch seems to be incorrectly loading both MVC assemblies into its test environment, creating confusion over what version of ViewResult is valid for use with Explicit().  The good news is that the problem only occurs if you have MVC 2 in your GAC.  I still need it because I have some sites that are built against MVC 2, but if you don’t have MVC 2 on your machine, then you probably wont see this problem at all, and you won’t need to use this workaround.

Back to Our Story…

Now that we’ve explicitly specified the view’s name (one way or another) we can run the tests again. This time we get an approval failure and ApprovalTests launches the browser to show us the received file. Great Success!

mvcviewtest6_thumb2

This page may not be the epitome of good design or usability, and maybe we want to customize it further, but at the moment we can see our test data in the page, and that’s awesome. If we do want to customize, we can do so with feedback. When we finally get the page looking the way we would like, we can approve the output and achieve protection from regressions.

At this point we have a few tests. We have an original test generated by Visual Studio, and we replaced one generated test with one that now exercises the default Index method with a mock. We have our two View tests which use MvcApprovals, and one of these uses seams to test the Index with fake data. We also have a couple of tests on the Models which I haven’t mentioned until now, because they are not very interesting. The only failing test is the Index view test, and it passes after I approve the output.

The tests are green in two build environments, Visual Studio and NCrunch. Seems like a good time for a commit, followed by some housekeeping.

Housekeeping

This housekeeping round is relatively minor. We want to make sure that our test actions are not available when we compile in Release mode. This is optional, but probably a good idea.

We can just eliminate half of our partial class using a compiler condition:

#if DEBUG
namespace MVCTestSite.Controllers
{
  public partial class HomeController
  {
    public ActionResult TestIndex()
    {
      // Create data
      IEnumerable<ICar> cars = new Car[] {
        new Car() { Make = "VW", Model = "Passat", Year = 2002 },
        new Car() { Make = "Honda", Model = "Fit", Year = 2007 },
      };

      // Pass to view
      return Index(cars);
    }
  }
}
#endif

Since our test action will no longer be available in release mode, we should also eliminate our “test pointer” when compiling in release mode:

namespace MVCTestSite.Tests.Views
{
  [TestClass]
  [UseReporter(typeof(FileLauncherReporter), typeof(ClipboardReporter))]
  public class HomeViewsTest : MvcTest
  {
#if DEBUG
    [TestMethod]
    public void TestIndexView()
    {
      MvcApprovals.VerifyMvcPage(new HomeController().TestIndex);
    }
#endif

    [TestMethod]
    public void TestAboutView()
    {
      MvcApprovals.VerifyMvcPage(new HomeController().About);
    }
  }
}

We can leave the test on the About view outside the conditional, since it does not require any non-production actions.

Since we are still in Debug mode, these conditions remain true and our tests still run and pass. There is another factor to consider when our project is compiled in Release mode and that is optimization. Our friend the Explicit() method infers the view name by examining the name of the method that calls it. This is why we needed to make sure that our private Index method had the same name as the public method. However, the default configuration for Release mode includes optimizations which could cause all of our action methods to be in-lined out of existence. This changes the method that calls Explicit. In my case, an anonymous delegate ended up calling Explicit and MVC complained that there was no view called “lambda_method”. Oops.

Llewellyn’s video shows how to work around this issue by disabling optimization in the Release configuration. If that idea bothers you, there is another way to do it on a method-by-method basis using an attribute:

[MethodImpl(MethodImplOptions.NoInlining)]
private ActionResult Index(IEnumerable<ICar> cars)
{
  return View(cars).Explicit();
}

Both work, but you need to pick one of these or you risk the “View Not Found” error.

Now that we have seen how to work with data, we can use these techniques continue to develop our site.  We can use ApprovalTests or traditional Asserts to work with our Models and Controllers, and we now have the ability to use MvcApprovals with our views.

Up until now, we’ve been working in two build environments, Visual Studio and NCrunch.  Although these environments are different, they share a lot of environmental factors because they both run on the same host.  To help ensure that our tests are robust, we should get them running on a different host with a different set of rules.  In the last part of this series, ApprovalTests and MVC Views: Tackling the Build Server, I’ll try to get these tests running under CC.NET.

Coding, Testing

ApprovalTests and MVC Views: Getting Started

Llewellyn Falco recently posted Using ApprovalTests in .Net 20 Asp.Mvc Views which has been one of his more popular videos in the Using ApprovalTests in .Net series. The level of interest in this video is justified. Unit testing MVC Views is not easy at first. The learning curve is steep, but not long. Once you get started and know the basics, its pretty easy to repeat the process. I was lucky enough to get some pointers from Llewellyn on testing a project of mine, and I’d like to share what I’ve learned about testing MVC. My plan is to take you from the default tests provided by File->New Project all the way up to a project that tests consistently in Visual Studio, under NCrunch and on a build server. In this process, I will try to make every mistake I can possibly think of and show the resolution to each problem. That will make this article significantly longer, but I hope it will also make reading this more useful.

Many bloggers, faced with a topic that can’t be summed up in a few hundred words, will start a blog series. More accurately, many bloggers will declare that they’re starting a blog series, write one post, and then never mention the subject again. Not me. I already wrote all the parts. But I am a merciful blogger, and interest of avoiding being slapped with a TL;DR, I will post each part separately over the next couple of days.

I’ve created an example project at https://bitbucket.org/magnifico/mvctestsite which includes spoilers, but if you want to dive right in you can. By the time I post all three parts of this article, everything in there should be fully explained.

Getting Started

We’ll start with a virgin MVC 3 intranet site. The Razor ViewEngine is all the rage, and if this were anything other than demo code, I’d recommend choosing it. But just to prove that the view engine doesn’t matter to this testing process, I created the project using the old ASPX ViewEngine. Also, using this ViewEngine makes the demo code more closely resemble the project that I learned these techniques on, since that project is a site that’s been slowly evolving since MVC 1, back when a razor was something you shaved with.

I checked the check box requesting “HTML 5 Semantic Markup”. I don’t even know what that means, but clicking checkboxes is fun. Of course, I also requested a unit test project. I selected MSTest as my framework, because its easy and does what I need. After a few moments my new MVC site appeared in Visual Studio, which I have named “MVCTestSite” in a stroke of creative genius.

A note on web servers

I feel the need to digress from the strictly linear telling of the story of the MVCTestSite at this point. If all had gone according to plan then about now I would be explaining how I set up the project to use IIS Express. Indeed, I did set it up to use IIS Express and followed the instructions in the readme to configure my site to use Windows Authentication. Namely, I disabled Anonymous Authentication and enabled Windows Authentication. This worked fine for awhile, right up until I tried to use ApprovalTests with this configuration. With IIS Express configured in this manner, ApprovalTests (as of v1.9) wont be able to retrieve pages for verification, IIS Express will reject the requests with a 401.2 error.

I spent a long time trying to figure out what I had done wrong, since I had used IIS Express and approval tests in projects at work. But like I said, that project has been evolving since MVC 1, and no doubt its web.config is full of all kinds of crazy kruft. Its probably not a good base case for comparison. Anyway, the issue comes down to this, ApprovalTests uses WebClient to make its requests, and by default WebClient does not send the user name to IIS Express. No username means Anonymous Authentication, which we previously disabled when we followed the instructions in the readme file.

So if you want to use IIS Express, you can, but you have to relax the restriction on Anonymous Authentication. This is really just an IIS Express setting, changing those properties does nothing to your project, so if you need to deploy with Anonymous Authentication disabled, that’s separate from what you do to IIS Express. Your other option is to use the Visual Studio Development Server (aka Cassini).

For the rest of the article I will just assume Cassini.

The Default Tests

mvcviewtest0_thumb[1]

As we can see in the Solution Navigator, the MVC project template gives us two tests by default, both are controller tests, one for the Index method and one for the About method. Since I chose the Intranet template, MVC did not provide a default AccountController or any tests to go with it.

Lets take a closer look at these two tests:

[TestMethod]
public void Index()
{
  // Arrange
  HomeController controller = new HomeController();

  // Act
  ViewResult result = controller.Index() as ViewResult;

  // Assert
  Assert.AreEqual("Welcome to ASP.NET MVC!", result.ViewBag.Message);
}

[TestMethod]
public void About()
{
  // Arrange
  HomeController controller = new HomeController();

  // Act
  ViewResult result = controller.About() as ViewResult;

  // Assert
  Assert.IsNotNull(result);
}

These two tests are Controller tests, they test the behavior of the Controllers. That’s nice, but testing Controllers isn’t really that much harder than testing ordinary C# classes. That’s one of the great things about MVC. Even without ApprovalTests to help us, Models have also been easy to test using traditional asserts. It’s the View that’s been difficult. There is no built in method to test the Views, and the closest we usually get to them is the ViewResult. That’s what the default tests use, but the ViewResult is not the View, its just another model that MVC uses to create the View. There is nothing wrong with testing the ViewResult, and when you do you are ensuring that your Controller is providing the correct building blocks for the given input, but you should also verify that the View conforms to your expectations. We can trust MVC to find the View, that’s framework code that we shouldn’t need to worry about. But once MVC does find the View, it’s executing code we wrote.

Some argue that the code in Views should be so trivial as to obviate testing. Certainly that is a good strategy for minimizing bugs, but without automated tests you have no safety net against regressions, and no executable specification. Your Views matter… a lot. To your user, the View is the application. I get more positive feedback from changing the background color on a site that I ever get from improving the Model. ApprovalTests can make testing Views easy enough to be practical. Like I said, the learning curve is steep, but it plateaus early. Scramble up the hill with me, Views are worth testing.

NCrunch

I’d like to digress once more to talk a little bit about NCrunch.

NCrunch is an automated parallel continuous testing tool for Visual Studio .NET. It intelligently takes responsibility for running automated tests so that you don’t have to, and it gives you a huge amount of useful information about your tests (such as code coverage and performance metrics) inline in your IDE while you work.

I like NCrunch. The first time I tried it, it didn’t really work with my project. Something to do with assembly signing. When v1.38b arrived on March 7th 2012, I tried it again, and the problem with my assemblies was gone. Its worked more or less flawlessly ever since. I strongly encourage you to try it. I’m going to assume that its active in this project as I build it, although if you download the source and play with the site, everything should work fine without NCrunch.

If you want some detailed discussion about NCrunch, listen to this episode of Herding Code:

HERDING CODE 135: REMCO MULDER AND JEFF SCHUMACHER ON CONTINUOUS TESTING

For me, there are two big advantages to running NCrunch:

  1. First, I get a tighter feedback loop. Often tests run when I stop to think for a moment. I can see right away if any tests are failing, even if I wasn’t planning on running the tests just then.
  2. Second, I get a second build environment. NCrunch makes its own workspace to build your project in. Compiling in this workspace reveals hidden dependencies and assumptions in your project and tests. In other words, things break in NCrunch that don’t break in Visual Studio, and correcting these breakages can make our code more robust.

So, if you want to give NCrunch a try, its free while in beta. Download it from www.ncrunch.net and follow the easy wizard to enable it in our project.

I enabled it right before pausing to write down some notes about NCrunch, and it executed the two default tests by the time I was done jotting things down. These two tests passed in NCrunch, and then I ran them in Visual Studio and they passed too. So for the time being, we have tests that run consistently in two environments. Later we will setup the project on a build server, and that will provide yet another environment.

I Thought This Post Was About ApprovalTests?

Yeah, yeah, I’m getting there. All right, lets implement a View test using MvcApprovals. First we need to add ApprovalTests to our test project using NuGet. As of this writing, the most recent version is v1.19:

mvcviewtest2_thumb[2]

In my mind, a View test is distinct from a Controller test, so I’m going to create a folder in my test project called “Views” and put a Basic Unit Test in the folder called “HomeViewsTest”. Maybe you are not a fan of putting files into arbitrary buckets, that’s cool. You don’t have to have the folder if you don’t want to, just put the test anywhere that makes you happy.

Import these namespaces into your HomeViewsTest:

using ApprovalTests.Asp;
using ApprovalTests.Asp.Mvc;
using ApprovalTests.Reporters;

Add a UseReporterAttribute (pro tip: you can also create a default reporter by adding an assembly level attribute to AssemblyInfo.cs).

[TestClass]
[UseReporter(typeof(FileLauncherReporter), typeof(ClipboardReporter))]
public class HomeViewsTest
{
  [TestMethod]
  public void TestIndexView()
  {
  }
}

When working with Views, it’s convenient to start with the file launcher reporter so that you can see the rendered results. The clipboard reporter puts a move command into your clipboard that will approve your received file if you paste it into a command window. Later on you can switch to a diff reporter to look for regressions. You’ll also notice that I’ve added the first test and called it TestIndexView.

Now add a line to the test:

MvcApprovals.VerifyMvcPage(new HomeController().Index);

At first, its easy to get confused by an MvcApproval. I freely admit that when I read a line of code like this in ApprovalTests own tests I made some bad assumptions about what was going on with that HomeController instance. So let me decompose this line into something a little less elegant. But first, lets take a look at the method signature for VerifyMvcPage:

public static void VerifyMvcPage(Func<ActionResult> func)

We can see that the argument if a Func<ActionResult>. Well that’s fine, it just so happens that the Index action method is a function without parameters that returns an action result. Lets make that a little clearer:

[TestMethod]
public void TestIndexView()
{
  Func<ActionResult> func = new HomeController().Index;
  MvcApprovals.VerifyMvcPage(func);
}

Now its pretty clear that we are passing a delegate to VerifyMvcPage. But if its still unclear, lets break it down again:

[TestMethod]
public void TestIndexView()
{
  HomeController controllerInstance = new HomeController();
  Func<ActionResult> func = controllerInstance.Index;
  MvcApprovals.VerifyMvcPage(func);
}

So now its clear that we create a controller instance, then we create a delegate pointing at that instance’s Index method, then we pass that delegate as an argument to VerifyMvcPage. But what’s not so clear is that “controllerInstance” has no impact on the test result. It’s only purpose in life to generate the func delegate. Furthermore, the func delegate is never invoked. It’s only purpose in life is to deliver metadata about the method (such as the method name and the name of the type it is declared on) to MvcApprovals. The same metadata could be delivered using a MethodInfo instance from the System.Reflection namespace, but there would actually be more ceremony involved in getting a MethodInfo instance than there is in creating a Func<T>. If you have ever used Moq, they use a very similar technique.

Under the hood, the MvcApprovals uses the metadata to create a URL. The controller name is inferred from the type that func points at, in this case “HomeController” becomes “Home”. The action is inferred from the method name, in this case “Index”. Once MvcApprovals creates a URL, it passes the string to another method called VerifyUrl. You can call VerifyUrl yourself, but then you would need to maintain the URL strings as well. When you use VerifyMvcPage, ApprovalTests maintains the URLs for you.

The important thing to remember is that the controller instance has no impact on your tests. If you fill up that instance with mocks, set properties, or make any other calls that manipulate the Controller state, it wont change anything in the test results. This is the first “gotcha” when testing Views with ApprovalTests. The reason that none of this has any effect should become clear as we continue this example.

While I’ve refactored this test from one line to three, NCrunch has been quietly running the tests in the background and glaring at me with it’s red “N”, which means I have a failing test. So, lets run the tests in Visual Studio and see what the failure looks like:

Test method MVCTestSite.Tests.Views.HomeViewsTest.TestIndexView threw exception:
System.MissingFieldException: ApprovalTests.Asp.PortFactory.MvcPort is uninitialized.
You are using a method that is using ApprovalTests.Asp.PortFactory.MvcPort,
but you have not set a value for this port first

ApprovalTests knows that the typical setup for ASP.NET development is to run on a port other than 80, but it has no way of knowing which port the development server is listening on. So, lets give that information to MvcApprovals:

[TestMethod]
public void TestIndexView()
{
  // Let ApprovalTests know which port to use.
  PortFactory.MvcPort = 61586;

  // Only used to generate delegate.
  HomeController controllerInstance = new HomeController();
  // Never invoked.
  Func<ActionResult> func = controllerInstance.Index;
  // Test view.
  MvcApprovals.VerifyMvcPage(func);
}

Now we should be golden, but NCrunch is still notifying me that I have broken tests. Of course, I’m working with ApprovalTests, so I should expect the test to keep failing until I approve some output for the test to Verify. To do that I need to run the test again, but it still errors out before giving me any output to approve. Here is the new error message:

Test method MVCTestSite.Tests.Views.HomeViewsTest.TestIndexView threw exception:

System.Exception: The following error occurred while connecting to:

http://localhost:61586/Home/Index

Error:

Unable to connect to the remote server —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:61586

Ok, now that we’ve given MvcApprovals enough metadata, it was able to generate the correct URL and now we can see that. But we can also see that our local machine refused to serve the request. The reason is simple… we haven’t started the web server yet! This is the second gotcha, you have to start the server before you try to run your test. There is no possible way for the test to pass if the server is not running.

You may be about to object that a unit test shouldn’t rely on the web server.  Ok, your objection is noted and overruled. Step back and look at what you do when you test a business object in .NET. You never actually test your code. That’s not even possible. What you actually happens when you hit “go” in the test runner is this:

  1. Your project builds.
  2. Your test project builds
  3. Your tests run.

Look at steps 1 & 2. Your code is consumed by the compiler, and the compiler emits MSIL code. In step three, the MSIL code is tested, not your C#/VB.Net code. Yet no one points at the compiler and calls this process an integration test. For MVC, there’s more than one compilation step. Some of the code is pre-compiled, and some of the code is compiled on demand by the web server. For our purposes, the web server is just another compiler host. For the time being it’s the only one that emits the output we want to test.

Maybe you can live with that line of reasoning, maybe not. If you do accept the idea, then you might wonder what kind of wrinkles are introduced by taking a dependency on a webserver. Let’s keep working through our example and find out. Fire up your second compiler by hitting “Start Without Debugging” on your toolbar (or under the Debug menu in Visual Studio). Your configured development server (IIS Express or Cassini) will start in the system tray, and the browser should launch and display the site’s home page.

Run the tests again. They fail because nothing is approved, but this time MvcApprovals was able to connect to the web server and read the page! The file launcher reporter takes the output and shows it to us in the browser.

mvcviewtest3_thumb[5]

I’ve circled part of the URL to emphasize that we’re looking at a local file.

I haven’t made any changes to the home page, so this looks like what I expected. I can use the data provided by the clipboard reporter to approve this file using a command window. After re-running the test in the Visual Studio test runner, it passes. NCrunch still shows a failure though because it hasn’t noticed that I approved the file. There are different ways to wake up NCrunch. We can make trivial edits to the file, or we can request a test run by right clicking on a red coverage marker, or by left clicking on the NCrunch notifier (which brings up the NCrunch Test window). All of these work ok, but we can help NCrunch notice the approved file by adding the file to the project. Once the the file is part of the project, NCrunch will notice any time we update the approval and run again. Adding a file to the project also causes NCrunch to wake up, so after adding the approved file, NCrunch turns green.

Happy, happy green.

N + 1

All right, we have one MvcApproval, and we have it working in two build environments, NCrunch and Visual Studio. Our goal is to get it running under CC.NET for a total of three build environments. But before we move on to that challenge, lets create a second MvcApproval and see if we have any opportunities for de-duplication once we have two tests.

We’ll create a test for the About view, since its the only other view our project has at the moment. First, let’s squash our call to VerifyMvcPage back down to one line in the Index test:

[TestMethod]
public void TestIndexView()
{
  PortFactory.MvcPort = 61586;
  MvcApprovals.VerifyMvcPage(new HomeController().Index);
}

The About test should look the same:

[TestMethod]
public void TestAboutView()
{
  PortFactory.MvcPort = 61586;
  MvcApprovals.VerifyMvcPage(new HomeController().About);
}

We can see immediately that the MVC port assignment is duplicated. Before we try to remove that duplication, lets run the tests. The Index test still passes, but the About test fails because its not yet approved. It looks fine so I approve it and after that both tests pass. (Incidentally, I left the two tests that came with the MVC template in place. And they are still working too.)

Lets try moving the port assignment to the constructor.

public HomeViewsTest()
{
  PortFactory.MvcPort = 61586;
}

After this change, the tests still pass. If this site will only ever have one controller, then moving this assignment to the constructor might be enough. I’m a fan of YAGNI but for the purposes of this example lets pretend that we eventually plan on adding more controllers, which would mean more View test classes, and this code would then be duplicated across the separate constructors. So lets move this assignment one more time, this time we’ll put it in a base class that HomeViewsTest can extend. (Since I can see the future, I also know that we will have some additional setup code in the base class before we are done, so bear with me.)

Here’s our new base class:

public class MvcTest
{
  public MvcTest()
  {
    PortFactory.MvcPort = 61586;
  }
}

And the current state of the test class:

[TestClass]
[UseReporter(typeof(FileLauncherReporter), typeof(ClipboardReporter))]
public class HomeViewsTest : MvcTest
{
  [TestMethod]
  public void TestIndexView()
  {
    MvcApprovals.VerifyMvcPage(new HomeController().Index);
  }

  [TestMethod]
  public void TestAboutView()
  {
    MvcApprovals.VerifyMvcPage(new HomeController().About);
  }
}

After this refactoring, our tests still pass. So that didn’t seem too hard right? There were a couple gotchas but we overcame them. But so far we’ve just been doing beginner stuff because our two pages only display static content (other than the “Welcome user!” message inherited from the master page). In a real website we’re probably going to need to access some data. So to take things to the intermediate level, we’ll need to introduce some models.

We’ll look at that in the next part of this series, ApprovalTests and MVC Views: Working With Data