Coding, Testing

MEF Composition Tests, Redux

Photo Credit: FutUndBeidl

A while back I gave a talk at SoCalCodeCamp about testing MEF composition.  Because there are a few steps involved in setting up the test, I wrote Stop Guessing About MEF Composition And Start Testing.  I had hoped that the article, along with the source code from my talk would help more people cross the gap from having once heard that it was possible to test composition, to actually implementing tests.  Since there were only about five people at my talk, it didn’t take long for the number of blog readers to exceed the number of attendees, and yet I still wonder how many people decide setting up the test is too complicated and just decide to use MEFX or Visual MEFX “as needed”.

While MEFX tools are valuable (and these composition tests are just a more powerful way to automate MEFX), running tests on an ad-hoc basis is never ideal.  Bugs creep in when you don’t expect them to, and automatic tests are your defense against them.  I also find composition tests provide valuable design-time feedback.  While taking questions at the end of the talk, I was asked about my process for designing MEF parts.  I answered that I more or less disregarded MEF until the end of the process, and then just added the needed attributes.  That was a truthful answer, but over the intervening months, I’ve realized it was the wrong answer.  What finally put the nail in the coffin was this great article: How test-driven development works (and more!), by J. B. Rainsberger.  After reading, it hit me that that I had setup a little waterfall process.  First, design and build a unit.  Then, glue MEF onto it.  The problem is that even when “keeping MEF in mind” I would sometimes do foolish things like pass a bunch of primitives into a constructor, and this would lead me to rework at the end of the development cycle when I tried to shove my class into a container.

So, to avoid rework and break some bad habits, I decided to start testing composition to get feedback during the design phase as soon as I know I’m going to use MEF (which these days is most of the time).  Its already paying dividends on my current project, as soon as I see the need to pass in a simple piece of data like a string or an integer, my composition test will blow up if I pass it through the constructor.  A lot of times I still prefer to pass the data in through the constructor, but since my test is blowing up, I’m forced design and test the system for getting that data to the constructor immediately, rather than at the end of the cycle.

I started testing composition out of necessity, I had a broken composition that needed fixing.  Now, I do it to save time and effort, and it seems more valuable to me every day.  More people should do it, but I’m afraid some will be turned off by the number of steps and concepts involved.

Here are a few of them:

  1. Synchronizing catalogs between test and production
  2. For some, this could also be their first exposure to ApprovalTests
  3. Finding a copy of MEFX.
  4. Setting up the test.

So, does it have to be so complicated?

Reducing the Surface Area

Lets look at a sample project and implement the integration test I described in my last article about composition tests.  Some people have a one-track mind.  Mine has two tracks: cars and pizza.  This time we will go with cars and our sample project will be a simple program called CarDealership with some parts related to cars in it.  It doesn’t have to be very complex since our primary interest is the test.

Lets look at that test:

[TestClass]
public class IntegrationTest
{
  [TestMethod]
  public void DiscoverParts()
  {
    try
    {
      var catalog = new DirectoryCatalog(".");
      var host = new CompositionContainer(catalog);
      var compositionInfo = new CompositionInfo(catalog, host);
      using (var stringWriter = new StringWriter())
      {
        CompositionInfoTextFormatter.Write(
            compositionInfo,
            stringWriter);
        Approvals.Verify(stringWriter.ToString());
      }
    }
    catch (ReflectionTypeLoadException ex)
    {
      Array.ForEach(
          ex.LoaderExceptions,
          lex => Console.WriteLine(lex.ToString()));
      throw;
    }
  }
}

A few things are already different about the test.  There’s no UseReporterAttribute on the class anymore, because I usually define this in AssemblyInfo.cs now using an assembly scoped attribute:

// AssemblyInfo.cs
using ApprovalTests.Reporters;
[assembly: UseReporter(typeof(DiffReporter))]

I also moved a few more lines inside the test body because I don’t usually have a reason to run more than one test with the same CompositionInfo.  Also, since my intention is to break this test down into something cleaner, I want to see it all in one place.

Right now, our test wont compile because we don’t have references to ApprovalTests or MEFX.  ApprovalTests has been available on NuGet for some time, and lately the NuGet package has been kept up to date with releases on SourceForge.  So, go ahead and use NuGet to install ApprovalTests:

In my previous post I explained where to find MEFX, which includes the CompositionDiagnostics library that we need for our test.  But you had to download it or compile it, and then add the reference by hand… icky.  But, we had no choice because this library wasn’t available on NuGet.  Well, it is available now, because I put it there a few months ago.  Now its much easier to get your hands on a copy of the library:

image

Now our test compiles and runs.  While you weren’t looking I added some parts to the CarDealership program.  But it doesn’t matter because I haven’t setup a pre-build event to copy assemblies from production to test.

I never liked doing that.  I knew it would be brittle and it was.  Ideally, I’d just like to use the exact catalog from production in my test.  It turns out this is a lot easier to arrange if you don’t use a DirectoryCatalog.  AssemblyCatalogs or TypeCatalogs are easy to share programmatically.  So, ask yourself if you really need to use a DirectoryCatalog.  Lets see what the test looks like when we use an AssemblyCatalog:

[TestMethod]
public void DiscoverParts()
{
  try
  {      
    var compositionInfo = new CompositionInfo(
      Program.Catalog, 
      Program.Host);
    using (var stringWriter = new StringWriter())
    {
      CompositionInfoTextFormatter.Write(
          compositionInfo,
          stringWriter);
      Approvals.Verify(stringWriter.ToString());
    }
  }
  catch (ReflectionTypeLoadException ex)
  {
    Array.ForEach(
        ex.LoaderExceptions,
        lex => Console.WriteLine(lex.ToString()));
    throw;
  }
}

To facilitate sharing, I’ve added static properties on the Program class.  They are simple properties, initialized in Program’s class constructor:

using System.ComponentModel.Composition.Hosting;
using System.ComponentModel.Composition.Primitives;
using System.Reflection;

namespace CarDealership
{
  public class Program
  {
    static Program()
    {
      Catalog = new AssemblyCatalog(Assembly.GetAssembly(typeof(Program)));
      Host = new CompositionContainer(Catalog);
    }

    public static ComposablePartCatalog Catalog { get; private set; }

    public static ExportProvider Host { get; private set; }

    private static void Main(string[] args)
    {
    }
  }
}

Looking back at the test, we can see the effect of sharing the catalog and host from Program.  We can now get the CompositionInfo with just one statement:

var compositionInfo = new CompositionInfo(Program.Catalog, Program.Host);

Another interesting development, the catalog and host are only referenced on this line.  Could we create a method that takes “compositionInfo” as a parameter without changing this test’s behavior?  The answer is no, but lets try it and see why.

[TestMethod]
public void DiscoverParts()
{
  var compositionInfo = new CompositionInfo(
    Program.Catalog,
    Program.Host);
  DiscoverParts(compositionInfo);
}

private static void DiscoverParts(CompositionInfo compositionInfo)
{
  try
  {
    using (var stringWriter = new StringWriter())
    {
      CompositionInfoTextFormatter.Write(
          compositionInfo,
          stringWriter);
      Approvals.Verify(stringWriter.ToString());
    }
  }
  catch (ReflectionTypeLoadException ex)
  {
    Array.ForEach(
        ex.LoaderExceptions,
        lex => Console.WriteLine(lex.ToString()));
    throw;
  }
}

To make the CompositionInfo a parameter, we pulled it out of the try-catch block.  Since the CompositionInfo constructor can throw the ReflectionTypeLoadException we’re trying to catch, moving it out of the try-catch block defeats the purpose of the exception handler.  Moving the exception handler back to the test method requires us to implement the handler (which never changes) every time we write a test.  We need to control when “new” is called on CompositionInfo.  We can do so by changing our parameter into a Func<CompositionInfo>:

[TestMethod]
public void DiscoverParts()
{
  DiscoverParts(() => new CompositionInfo(Program.Catalog, Program.Host));
}

private static void DiscoverParts(Func<CompositionInfo> getCompositionInfo)
{
  try
  {
    using (var stringWriter = new StringWriter())
    {
      CompositionInfoTextFormatter.Write(
          getCompositionInfo(),
          stringWriter);
      Approvals.Verify(stringWriter.ToString());
    }
  }
  catch (ReflectionTypeLoadException ex)
  {
    Array.ForEach(
        ex.LoaderExceptions,
        lex => Console.WriteLine(lex.ToString()));
    throw;
  }
}

Now, I could still create a CompositionInfo instance in the test method, then use the delegate to pass back that instance.  Doing so, I would not be protected by the exception handler, but that would be a minor problem.  The exception handler only exists as a time saver, when a ReflectionTypeLoadException occurs, it will print additional information not included in the default dump.  Without the handler, you would need to rerun the test in the debugger in order to see the loader exceptions.  So, its not the end of the world, but at least now its possible to reuse the exception handler as long as we remember how to build the delegate correctly.

Can we make DiscoverParts easier to use by removing the caveat that the constructor must be part of the delegate?  We could provide an overload that creates the delegate for the caller:

[TestMethod]
public void DiscoverParts()
{
  DiscoverParts(Program.Catalog, Program.Host);
}

private static void DiscoverParts(ComposablePartCatalog catalog, ExportProvider host)
{
  DiscoverParts(() => new CompositionInfo(catalog, host));
}

private static void DiscoverParts(Func<CompositionInfo> getCompositionInfo)
{
   // ...
}

That’s better, and maybe we don’t need that delegate after all.  For the time being, I’m going to leave it there because I think the more interesting question is: Do we need the host?  If you look again at the construction of the host in Program, you’ll see that it’s the simplest possible CompositionContainer, and the only interesting thing about it is the catalog it consumes.  So why not focus our test on the catalog?

[TestMethod]
public void DiscoverParts()
{
  DiscoverParts(Program.Catalog);
}

private static void DiscoverParts(ComposablePartCatalog catalog)
{
  DiscoverParts(catalog, new CompositionContainer(catalog));
}

private static void DiscoverParts(ComposablePartCatalog catalog, ExportProvider host)
{
  DiscoverParts(() => new CompositionInfo(catalog, host));
}

In a great many cases, the container itself is very boring, but there certainly are plenty of cases where some other, more interesting, ExportProvider comes into play. So, both overloads are useful.  In the case where only the catalog is interesting, the overload without the host should be used.  This sends the signal that the host is not important.  On the other hand, if you choose to use the overload with the host specified, then you’re sending the signal that the host is interesting and important.  Who are you sending the signal too?  Probably your future self.

So we’ve accomplished a lot.  Our test method has been reduced to a simple method call that can take as little as one parameter.  The best thing about it is that this parameter is part of the “normal’ set of Composition types, it does not require the caller to know anything about CompositionInfo, CompositionInfoTextFormatter, or ReflectionTypeLoadException.  All that code is still there though, and its not conveniently packaged for distribution.  Can we make this test even easier by completely hiding all this code?

Introducing CompositionTests

One could take everything we’ve looked at here, bundle it up into a library, and put it on NuGet and GitHub.  I’ve done just that:

image

It’s just a little library but it should make composition tests dead simple:

using CompositionTests;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace CarDealership.Tests
{
  [TestClass]
  public class IntegrationTest
  {
    [TestMethod]
    public void DiscoverParts()
    {
      Composition.DiscoverParts(Program.Catalog);
    }
  }
}

That’s the whole test.

You can also scrub the formatted text before sending it to ApprovalTests.

using ApprovalUtilities.Utilities;
using CompositionTests;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace CarDealership.Tests
{
  [TestClass]
  public class IntegrationTest
  {
    [TestMethod]
    public void DiscoverParts()
    {
      Composition.DiscoverParts(
        Program.Catalog,
        StringExtensions.ScrubVersionNumber,
        s => s.ScrubPath(@"C:\Test"));
    }
  }
}

You can use any number of scrubbers to transform your text into a consistent state.  For example, scrubbing version numbers is useful when you use an AssemblyCatalog with an assembly that has an auto incrementing build number.  I’ve also included an extension to scrub the public key token, which is useful when using NCrunch (which disables assembly signing in its builds) alongside other test runners (which do sign the assemblies).  Or, you can use any Func<string, string> that you can think of.

The library also includes (and uses by default) an text formatter that orders the parts by name.  The default text formatter does not guarantee that the parts come out in any particular order.  I have seen them printed in different orders in different build environments, so enforcing order is a must if we want our tests to play nice with VS, NCrunch and a build server.  I’m still not sure that I’ve covered all the scenarios that can lead to things appearing out of order, because the behavior is not easy to trigger.  Luckily I had an example in production which I was able to recreate in the library’s unit tests.  If I encounter any more ordering problems, they will be addressed in future versions of the library.

Check It Out

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s