Coding, Tips, Troubleshooting

Stop Guessing About MEF Composition And Start Testing

Diagnosing MEF Failures | SoCalCodeCamp @ CSU Fullerton | 1/29/2012

I gave my first talk at a Code Camp last weekend. The topic was “Diagnosing MEF Failures” at it was well received by the 5 attendees.  A recording of the talk is embedded at the top of this post, and the direct link is just below, along with the slides and demo code.  This blog post compliments the talk by providing a step by step guide about how to test your MEF composition using ApprovalTests and Microsoft’s CompositionDiagnostics library.

Finally, I revisted this subject in June 2012.  If you only want to know the easiest way to setup composition tests, read MEF Composition Tests, Redux instead.  If you’re interested in knowing how everything works under the hood, read this article first, then read the June article.

Why?

Diagnosing MEF composition failure is tricky, especially when parts have two or more nested dependencies.  In this scenario, MEF’s error messages aren’t very helpful.  MEF was unable to satisfy one or more parts in the chain, but the error message points you to the part at the top of the chain, to really fix the problem you need to know about the bottom of the chain.   If you start by looking where the error message sends you, you can end up feeling like you’re on a wild goose chase because nothing appears to be wrong with that part.

Recognizing this problem, the MEF team created the a static diagnostics library as a “Sample”, but when MEF was integrated into the BCL with .NET 4, this useful library never made it in.  You can still get the code from the MEF CodePlex site, along with a shell that allows you to define directory or assembly catalogs from the command line and analyze the parts within.  This shell is useful and flexible (for example, you could use it to examine deployed parts in production without changing any code) but sometimes tedious to use.  An alternate shell (Visual MEFx) brings some nice GUI touches to the concept.  The GUI makes it much easier to play with what if scenarios while examining the collection of parts.

Either of these tools can be used to analyze real, deployed compositions, even when dealing with parts authored by a third party.  That’s a really powerful concept.  But for some MEF users (myself included), third parties are not a big concern.  I’m using MEF to compose a collection of known parts, so I can say with confidence what the composition should look like.  Sometimes I might want to play out a what-if scenario, but most of the time I just want some assurance that all my parts are there, and therefore all my application’s expected behaviors will be there.

Ideally, I’d like to automate this assurance without relying on a GUI, or trying to spin up a separate process during integration testing.  Finally, it would be great if I could be assured that my composition looked correct without having to write one test for every part that should be in the container.  How I achieve these goals is the subject of this post.

Background

If you don’t know about ApprovalTests you really should take a moment to learn about them.  Read about this library at http://approvaltests.sourceforge.net/, and be sure to listen to the Herding Code podcast.  You don’t need to understand much about ApprovalTests to follow the recipe I plan to lay out below, but even if you aren’t interested in MEF at all, you should still check out ApprovalTests.

Presumably if you are reading this you are already familiar with MEF, but if not, get started here: “What is MEF?”.  You don’t have to read the whole thing to start getting the point of what MEF is for, maybe just the first few sections.  Follow that with the MEF Programming Guide on CodePlex.

If you would rather read than watch my video, I primarily drew from two references while putting together my talk.  Part 13 of the MEF programming guide is about Debugging and Diagnostics.  Finally, this entry on Daniel Plaisted’s Blog is a great reference which includes background on why MEF fails in general, quite a few corner cases, and an overview of live and static debugging tools.  My talk focuses on using the tools, so read Daniel’s blog if you’re really interested in studying MEF failure.

Since Daniel covers MEFx and Visual MEFx, I wont discuss them here, instead I’ll focus on presenting the recipe for testing MEF composition with ApprovalTests.

What You Will Need

  • For starters, you’ll need a project that uses MEF, but since you are here I’ll assume you already have that.
  • You’ll need to download ApprovalTests and reference it in your test project.  There is a version on NuGet, but its not always up to date with the latest release from the SourceForge site.  The recipe should work with 1.0.11 or greater, so its up to you if you want the latest and greatest.
  • You’ll need a version of the CompositionDiagnostics library.  A quick way to get your hands on a binary copy is to download MEFx.  The MEFx zip contains two files, one is a command line shell, and the other is the library.  You can obtain the source by downloading MEF Preview 9.  Note that there are MEF 2 previews also in the release list, you want the preview from MEF 1.

The CompositionDiagnostics source may be useful if you run into any issues with imports showing up out of order on different machines.  If you have this problem, add this method to the CompositionInfoTextFormatter:

private static string GetPartDefinitionKey(object definition)
{
    if (definition == null)
        return null;
    var compositionElement = definition as ICompositionElement;
    return compositionElement != null ?
        compositionElement.DisplayName :
        definition.ToString();
}

Then update the Write() method to order the PartDefinitionInfos by this “key”.

Test Setup

I’m using MSTest, it works fine for me.  If you have a favorite test framework, this shouldn’t be too hard to adapt.

If you don’t have a test project for your solution, set one up.  It will be important to keep your test and production catalogs in sync.  You don’t want to fix a problem in test and still have your composition fail in production.  I use directory catalogs, so I keep my catalogs in sync using a simple pre-build event in the test project properties:

copy /y “$(SolutionDir)$(SolutionName)\bin\x86\Debug\*.dll” “$(TargetDir)”

Add references to ApprovalTests.dll and ApprovalUtilities.dll.

Unzip the MEFx zip file, or compile the code yourself.

Grab the Microsoft.ComponentModel.Composition.Diagnostics.dll file, and place it near your solution (a lib folder would be a good option) then take a reference to the library.

Add a test class to the solution and call it whatever makes sense to you, I’ll use the name “IntegrationTest” in my example.

Import these namespaces into your IntegrationTest file.

using System;
using System.ComponentModel.Composition;
using System.ComponentModel.Composition.Hosting;
using System.IO;
using System.Reflection;
using ApprovalTests;
using ApprovalTests.Reporters;
using Microsoft.ComponentModel.Composition.Diagnostics;

Add a UseReporterAttribute to your test class.   DiffReporter is my favorite:

[TestClass]
[UseReporter(typeof(DiffReporter))]
public class IntegrationTest

The CompositionDiagnostics library provides us with a few new types.  The most interesting to me is the CompositionInfo class.  This is where the magic happens.  We create a catalog and a container as usual, then instantiate an instance of CompositionInfo, passing the catalog and container into the constructor.  I find it useful to put this into its own method, since I usually have at least two integration tests.

private static CompositionInfo GetCompositionInfo()
{
    var catalog = new DirectoryCatalog(".");
    var host = new CompositionContainer(catalog);
    var compositionInfo = new CompositionInfo(catalog, host);
    return compositionInfo;
}

Now you can add a test method.  Here’s the code, followed by an explanation:

[TestMethod]
public void DiscoverParts()
{
    try
    {
        using (var stringWriter = new StringWriter())
        {
            CompositionInfoTextFormatter.Write(
                GetCompositionInfo(),
                stringWriter);
            Approvals.Verify(stringWriter.ToString());
        }
    }
    catch (ReflectionTypeLoadException ex)
    {
        Array.ForEach(
            ex.LoaderExceptions,
            lex => Console.WriteLine(lex.ToString()));
        throw;
    }
}

The CompositionDiagnostics library provides another useful class, the CompositionInfoTextFormatter, which we can use to create a text representation of the data in the CompositionInfo.  The CompositionInfo class does all the heavy lifting.  We use the method Approvals.Approve to verify the text representation, more on this later, for now its enough to know that this line takes the place of an “Assert” call in an ordinary unit test.

This test also catches the ReflectionTypeLoadException. Type loading problems can cause your test to fail when the CompositionInfo is under construction. Therefore, you would never reach the ApprovalTest. MSTest will catch this and dump the exception in the test result, but the really interesting information on that exception is in the LoaderExceptions property which isn’t shown by default. So, in the catch block each loader exception is dumped to the console before allowing the exception to go on it’s merry way. This saves the extra step of rerunning the test under the debugger just to drill into the LoaderExceptions.

That’s it for the first test, if you’ve been following along you should be able to run it.

The Results

When you run the test for the first time, it will fail.  This is not unusual for ApprovalTests.  The ApprovalTest library needs “approved” output to verify the test output against.  Since we used the DiffReporter, you are notified of the failure when a diff tool launches showing output on one side and a blank slate on the other.  (If nothing happens, you probably need TortiseSVN, which provides the default diff tool.  You can switch to NotepadLauncher if you decide you don’t like the idea of installing TortiseSVN just for this purpose.  Other reporters are supported, but that’s outside the scope of this entry.)

MEF1

You can flip the left and right sides to make it easier to read.

So what is this?  Briefly, the diagnosis begins by looking at the Parts property on the catalog, examining the metadata for each, and eventually tries to compose each part defined in the catalog.  This will uncover any problems with the composition by throwing exceptions.  The analyzer catches the exceptions, and stores them in a data structure along with the other metadata.  Any parts that fail for reasons not related to importing other parts have the potential to be the root cause of the composition failure.  Each will be marked as a [Primary Rejection] by the text formatter.  Exceptions and the other metadata will also be printed with each part.

That’s the sad path.  On the happy path, your composition succeeds and each part is still listed, along with some metadata and information about which parts satisfied nested imports, if any.  So, in our test, we get the CompositionInfo, and format the data using the CompositionInfoTextFormatter.  The output is captured by a StringWriter, and passed to Approvals.Approve() for comparison.  If the output is correct, we should see no exceptions listed, no rejections, and each part [SatisfiedBy] the exports we expected to be imported.  If so, we can right click on the “received” file and choose “Use this whole file”.  Save.  It is now approved.  Rerunning the test should result in a pass.

If the composition has problems, we need to work on those problems until the output looks right, just as we would in any other TDD scenario.  Once it is right, approve it so it can be checked in the future.  Whenever any change is made to the composition (expected or not) this test will fail and let you know that you either need to approve the new composition, or resolve the unexpected failure.

Since the ReflectionTypeLoadException causes the test to fail before the approval stage, you should look at the trace output or test results when these occur.  Your loader exceptions will likely be MissingMethodExceptions with messages explaining that certain methods have no implementation.  Whichever part is meant to provide those implementations cannot load, and you may have to do some detective work to figure out why.  Often I find that these problems are related to parts expecting different versions of the same assemblies, or that the parts refer to assemblies that are unavailable.

Another Test

I usually have this next test in my IntegrationTest as well.

[TestMethod]
public void InstantiateService()
{
  try
  {
    Approvals.Verify(
      GetCompositionInfo().Host.GetExportedValues<IPizzaMaker>(),
      s => s.GetType().FullName);
  }
  catch (ReflectionTypeLoadException ex)
  {
    Array.ForEach(
      ex.LoaderExceptions,
      lex => Console.WriteLine(lex.ToString()));
    throw;
  }
}

I look at it now and I wonder why.  This test will grab all instances of a certain type of part, compose them, and return them.  ApprovalTests will approve the list.  This doesn’t really do anything new compared to the full dump of the CompositionInfo in DiscoverParts.

I think that over time I must have become over zealous with the Don’t Repeat Yourself principle and used the GetCompositionInfo() method when I shouldn’t have.  The original form of this test was probably this:

[TestMethod]
public void InstantiateService()
{
  try
  {
    var catalog = new DirectoryCatalog(".");
    var host = new CompositionContainer(catalog);
    Approvals.Verify(
      host.GetExportedValues<IPizzaMaker>(),
      s => s.GetType().FullName);
  }
  catch (ReflectionTypeLoadException ex)
  {
    Array.ForEach(
      ex.LoaderExceptions,
      lex => Console.WriteLine(lex.ToString()));
    throw;
  }
}

So, what’s the difference/use of this second version?  In the version that uses GetCompositionInfo, the container and catalog construction happen inside the method, then are passed to a CompositionInfo.  Later, the test just used the container (accessed through the Host property) to ask for the IPizzaMaker instances.  So the construction of the CompositionInfo was a waste, but you might live with that for the sake of staying DRY.  However, since we know that the CompositionInfo can die during construction, its more than just a waste, it’s a potential source of test failure that has nothing to do with the ability to instantiate IPizzaMakers.

In the second version we construct the host without bothering with the CompositionInfo, and we avoid ReflectionTypeLoadExceptions unless they have something to do with IPizzaMaker.  So this test might tell you something interesting if making IPizzaMakers was your primary concern, but it wont give you the type of root cause analysis provided in the CompositionDiagnostics output.  So using both can potentially give you a smidge more insight into your composition.

If you reuse GetCompositionInfo to access the host, it tends to fail in concert with the DiscoverParts test, and so tells you nothing new.

MEF 2

In MEF 2, the InstantiateService test has the potential to take center stage, while the DiscoverParts test will be more of a niche player.  This is because the MEF team has made big improvements to the baked in diagnostics and exception messages.  First of all, MEF always knew what the root cause was, it just failed to tell you about it in the error message.  In MEF 2 Preview 5, the composition exception indicates the primary rejection at the top of a chain of failures leading all the way down to the part you asked for.

By default, the container will still wait for you to ask for a part that requires the rejected part (or the rejected part itself) before any exception is thrown.  This is part of MEFs “Stable Composition” feature.  You can disable this feature by passing the CompositionOptions.DisableSilentRejection option to the container’s constructor.  The result will be that MEF will throw an exception as soon as it rejects the part.  You will definitely want to use this option on the container used for integration tests, and if you aren’t working with third parties, disabling silent rejection is recommended on your production container as well.

When you run InstantiateService with silent rejection disabled, you should get an exception if any part in the container would be rejected, not just the IPizzaMakers.  This allows InstantiateService to detect a larger class of failures in MEF 2 compared to MEF 1.

The DiscoverParts test will still have its niche in MEF 2.  This test will remain useful in detecting missing optional parts.  A part that is optional and missing will not throw an exception even when silent rejection is disabled.  Although other parts want to import the optional parts, they don’t “depend” on them to operate, since classes that import optional parts should all be able handle the case where the parts aren’t available.  Since its missing (meaning—not in the catalog) there’s no way it could have a problem itself and throw an exception.

So if the part is optional should your test pass or fail?  Its up to you to decide, but remember that the whole premise of this test strategy is that you already know what the composition should look like.  You could have a collection of three services you want hosted that work together in an assembly line fashion. You might want to reserve the right to increase the number of steps in the assembly line in the future so you use [ImportMany] to get the whole collection.  Now you’ve just made your imports optional, but do you really want to deploy your application if service 2 of 3 is missing?  The DiscoverParts test can detect this for you, but “noisy” rejection can’t.  It’s a niche, but it’s a niche I find myself in, so I think the DiscoverParts test will remain useful even after MEF 2 arrives.

Conclusion

Really, I just hope that someone will find this idea useful.  Feel free to run with it and let me know if you can think of any other strategies for testing composition.  In particular I’d be interested to hear ways that people have found for testing scenarios with third parties or in production.  If you are interested in additional thoughts from me on the subject, please read: MEF Composition Tests, Redux.

Thanks for reading.

4 thoughts on “Stop Guessing About MEF Composition And Start Testing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s