Automated Testing and Sitecore - Part 1
Well, my last post seems to have generated some interest. Automated testing with Sitecore is a hot topic. It is much more difficult and complex than those 2 second demos you see for general unit testing in your code. The purpose of this post is to describe how I currently perform my automated testing against Sitecore. I say currently because I am constantly updating how I do this as I discover better ways to test, and as people question me on my current processes. I encourage this questioning as it causes me to re-evaluate what I have established to make sure I am actually doing the best I can. And to this end, I encourage your comments if you think I’m totally off the mark.
So let me set a perspective for you. When you write code for Sitecore, you’re not writing standalone code. You’re writing a customisation for an existing system. How can I unit test customisation code for an external system? Well I just need to extract my code from that environment and test it in isolation. To do that I will probably need to mock a few things that my customisations are relying on. Like the Sitecore context. And therein lies the problem.
Sitecore is a complex system. I’m pretty sure I’ve got a good idea about how it works, and what I can expect when I call certain methods on certain classes and objects, but how can I be absolutely 100% sure? I started thinking about trying to mock Sitecore and it’s context so I could unit test one day, and after a lot of making my head hurt with the sheer amount I would have to mock, I pretty much gave up. I hold the opinion that you shouldn’t mock anything that you can’t ultimately change. Mocking is great for removing dependencies in a custom dev project for code that I can control and change. I should have absolute intimate knowledge of this code and how it MUST work. I can set the expectations quite easily. Not so with a 3rd party system. I can have a pretty good idea about how I think it should work, but ultimately I don’t think I ever have enough knowledge of it to properly mock it.
There might be some classes in Sitecore I might be able to mock easily. Can I mock an item? Can I mock a database? When I call GetItem on the Database class, I know I should expect these certain Items back as long as the items exist in the content tree, and are in a final workflow step, and don’t have any publishing restrictions, and a publish has occurred, and, and, and. Can you see where I’m headed here? Well perhaps I don’t care about all those things and I just want to test the case where my items defiantly are returned? Well maybe I could mock that. But when I started looking into it last time it would have taken a hell of a lot of effort to do so.
Instead, I made the realisation and admittence that you can’t unit test code written to customise a 3rd party system. I put the effort I would have spent trying to mock Sitecore into writing integration tests. When I’m testing code for Sitecore, I’m not unit testing. All my tests are integration tests, hence the title of this post. That’s why I call it automated testing, cause that’s what I’m more concerned with. Being able to repeat my tests at the click of a button rather than completely isolating my code under test. It’s not ideal, and as I said earlier, I’m constantly reassessing my testing techniques. I might spend the time one day to attempt to mock Sitecore to simplify my tests, but for now I’m happy to do integration rather than unit tests.
I wrote a custom NUnit test runner which is an ASP.NET page which runs my NUnit tests. The benefit of my custom test runner is that the test runner itself runs in the ASP.NET context and can be dropped into a Sitecore site, so now my test runner is running in the Sitecore context and I can call into the Sitecore API. There are many problems trying to use the Sitecore API from a console or windows app; I’m sure you’ve all tried :) .
Depending on what we’re testing will determine what kind of test I write. If I’m testing low level methods that use the Sitecore API, then I’ll write a standard NUnit test which runs in my custom test runner. If I’m testing a Sitecore web control (class which inherits from Sitecore.Web.UI.WebControl) then I’ll write a NUnit test to run in the custom test runner and call the RenderAsText method from my TestMethod. If I’m testing other page elements which generate markup and have no dynamic page behaviors I’ll use a test which doesn’t run in the custom test runner, and I’ll make an HTTP request to a page hosting the control under test, then use HTML Agility Pack to navigate the response and make my assertions on the markup. HTML Agility Pack is also used by Sitecore internally (check the bin folder). If I’m testing a control which does exhibit dynamic behaviors I’ll use WatiN to make the request as this will use IE and will run javascript. I’ll also use WatiN if I’m testing a form or anything else which requires more than just a request (like posting back).
In all cases I need to setup the content tree before my tests run, and pull it down after they run. I normally have a separate sub content subtree for each Test Fixture / Class. If my test is running in the Sitecore context, then I can call directly into the API to setup the test items, and also to delete them when done. If I’m calling from a separate test framework which doesn’t use my custom NUnit test runner, then I’ll do my creation and cleanup in a layout which is attached to a permanent item in the content tree. The query string I pass this item determines what action to take such as create test content 1, test content 2, etc or cleanup. That way, in my test setup all I have to do is make an HTTP request to the appropriate URL and the items required for test get created. Likewise, it’s just another HTTP request to cleanup when I’m done.
That explains my testing processes at a high level. Over the next few posts I’ll delve into how I actually do each of the different kinds of tests I mention above, and I’ll also include how to create the custom test runner.
> There are many problems trying to use the Sitecore API from a > console or windows app; I’m sure you’ve all tried :) .
Is it possible? Any tips?