Damn you VSUnit!

I ran into quite an unusual quirk with VSUnit today. (I’ve taken to calling Visual Studio Unit Testing VSUnit for brevity.) I’ve been doing some Sitecore dev using Test Driven Development (TDD) which I have had great success with in the past when working in a purely NUnit space. But with VSUnit I like the ability to run my tests straight inside the IDE (please don’t point me at TestDriven.net :) ). And also the integrated code coverage (please don’t point me at NCover). Yeah, I know there are things I can use with NUnit to give me these capabilities, but the VSUnit things requires no additional setup. It’s there out of the box.

So enough about the good points of VSUnit…onto the source of this rant. So, there are 2 buttons in the toolbar which allow me to either run the tests in the current context, or all tests in the project.

The tests in the current context depend where your cursor is. If your cursor is in the body of a test method, then only that single test will be run. If it’s in a test class file outside a method, then all the tests for that class will be run. But when I have several tests and test classes, I don’t want to execute each test class separately, I want to make use of that “run all tests in project” button.

So when I click to execute all tests in the project, what order did VS run them in? Almost random apparently. Hmm. I was expecting it to behave like NUnit, where if I select to run all tests they will run in some kind of predictable order. At least all the tests for a test class are executed one after another in NUnit. VSUnit runs them completely out of order. It might execute Class1.Test1 then Class1.Test2 then Class2.Test1 then back to Class1.Test3…Completely out of order. OK, I can see the point that unit tests should be isolated and atomic and should be able to be run in any order. And I agree. Test 3 should not rely on Test 1 being executed before it. But what about when I’m doing class initialisation and cleanup? I’ll get back to that in a second as it ties into my next rant.

So what happens if I need to perform class initialisation and cleanup. This is the perfect place to create the Sitecore items which are required by your tests, and then destroy them to return the environment to a known good state when our tests are complete. As anyone who has done this before, you’ll know that item creation in Sitecore is not cheap…it takes time. So we don’t really want to be creating and destroying items for each test. Now in NUnit, because everything runs in an order as I expect, I know that the class initialise will execute, then the classes tests, then the cleanup, then onto the next test class. Not so with VSUnit. It will call the classes initialise, then some of the tests, then it will branch off to another test. And if that test requires initialisation it will call that, then the test, etc. Note, we haven’t called any cleanup yet. VSUnit waits until all tests have completed before calling any of the cleanup methods. So in the case above where I create Sitecore items during initialisation and clean them up in cleanup, my items don’t get cleaned up before the next test class runs, so when it initialises it’s initialising into a dirty environment.

In the end I had to move my Class Init and Cleanup to Test Init and Cleanup. Looks like I might have to reassess the use of those other NUnit extensions for VS…