codeflood logo

Automated Testing and Sitecore - Part 1

Well, my last post seems to have generated some interest. Automated testing with Sitecore is a hot topic. It is much more difficult and complex than those 2 second demos you see for general unit testing in your code. The purpose of this post is to describe how I currently perform my automated testing against Sitecore. I say currently because I am constantly updating how I do this as I discover better ways to test, and as people question me on my current processes. I encourage this questioning as it causes me to re-evaluate what I have established to make sure I am actually doing the best I can. And to this end, I encourage your comments if you think I'm totally off the mark.

So let me set a perspective for you. When you write code for Sitecore, you're not writing standalone code. You're writing a customisation for an existing system. How can I unit test customisation code for an external system? Well I just need to extract my code from that environment and test it in isolation. To do that I will probably need to mock a few things that my customisations are relying on. Like the Sitecore context. And therein lies the problem.

Sitecore is a complex system. I'm pretty sure I've got a good idea about how it works, and what I can expect when I call certain methods on certain classes and objects, but how can I be absolutely 100% sure? I started thinking about trying to mock Sitecore and it's context so I could unit test one day, and after a lot of making my head hurt with the sheer amount I would have to mock, I pretty much gave up. I hold the opinion that you shouldn't mock anything that you can't ultimately change. Mocking is great for removing dependencies in a custom dev project for code that I can control and change. I should have absolute intimate knowledge of this code and how it MUST work. I can set the expectations quite easily. Not so with a 3rd party system. I can have a pretty good idea about how I think it should work, but ultimately I don't think I ever have enough knowledge of it to properly mock it.

There might be some classes in Sitecore I might be able to mock easily. Can I mock an item? Can I mock a database? When I call GetItem on the Database class, I know I should expect these certain Items back as long as the items exist in the content tree, and are in a final workflow step, and don't have any publishing restrictions, and a publish has occurred, and, and, and. Can you see where I'm headed here? Well perhaps I don't care about all those things and I just want to test the case where my items defiantly are returned? Well maybe I could mock that. But when I started looking into it last time it would have taken a hell of a lot of effort to do so.

Instead, I made the realisation and admittence that you can't unit test code written to customise a 3rd party system. I put the effort I would have spent trying to mock Sitecore into writing integration tests. When I'm testing code for Sitecore, I'm not unit testing. All my tests are integration tests, hence the title of this post. That's why I call it automated testing, cause that's what I'm more concerned with. Being able to repeat my tests at the click of a button rather than completely isolating my code under test. It's not ideal, and as I said earlier, I'm constantly reassessing my testing techniques. I might spend the time one day to attempt to mock Sitecore to simplify my tests, but for now I'm happy to do integration rather than unit tests.

I wrote a custom NUnit test runner which is an ASP.NET page which runs my NUnit tests. The benefit of my custom test runner is that the test runner itself runs in the ASP.NET context and can be dropped into a Sitecore site, so now my test runner is running in the Sitecore context and I can call into the Sitecore API. There are many problems trying to use the Sitecore API from a console or windows app; I'm sure you've all tried :) .

Depending on what we're testing will determine what kind of test I write. If I'm testing low level methods that use the Sitecore API, then I'll write a standard NUnit test which runs in my custom test runner. If I'm testing a Sitecore web control (class which inherits from Sitecore.Web.UI.WebControl) then I'll write a NUnit test to run in the custom test runner and call the RenderAsText method from my TestMethod. If I'm testing other page elements which generate markup and have no dynamic page behaviors I'll use a test which doesn't run in the custom test runner, and I'll make an HTTP request to a page hosting the control under test, then use HTML Agility Pack to navigate the response and make my assertions on the markup. HTML Agility Pack is also used by Sitecore internally (check the bin folder). If I'm testing a control which does exhibit dynamic behaviors I'll use WatiN to make the request as this will use IE and will run javascript. I'll also use WatiN if I'm testing a form or anything else which requires more than just a request (like posting back).

In all cases I need to setup the content tree before my tests run, and pull it down after they run. I normally have a separate sub content subtree for each Test Fixture / Class. If my test is running in the Sitecore context, then I can call directly into the API to setup the test items, and also to delete them when done. If I'm calling from a separate test framework which doesn't use my custom NUnit test runner, then I'll do my creation and cleanup in a layout which is attached to a permanent item in the content tree. The query string I pass this item determines what action to take such as create test content 1, test content 2, etc or cleanup. That way, in my test setup all I have to do is make an HTTP request to the appropriate URL and the items required for test get created. Likewise, it's just another HTTP request to cleanup when I'm done.

That explains my testing processes at a high level. Over the next few posts I'll delve into how I actually do each of the different kinds of tests I mention above, and I'll also include how to create the custom test runner.

Comments

> There are many problems trying to use the Sitecore API from a console or windows app; I’m sure you’ve all tried :) .
Is it possible? Any tips?

Alistair Deneys

Hi Bob, The reason we have issues calling the Sitecore API from a console app is that Sitecore requires an HTTP context to work. The best way to call into Sitecore from a console or windows app would be to host a service (web, .net remoting or WCF) in your Sitecore application and call that from your console app.

Hmm, that's unfortunate. So if I want to, say, import 70,000 documents from another repository, I can't really do it in a single process, because - since I have to build my program to talk to the Sitecore API as an ASP.Net program - I'll be at the mercy of the web server's patience, which I don't really expect to last through that many documents (my preliminary experiments show that it takes about 9 seconds to import one of the smaller documents). That means all of the optimizations I might otherwise achieve (by doing template lookups once per import, for example, rather than once per document) are out the window. What other restrictions does Sitecore impose on efficiency and flexibility (we're in the process of evaluating the available products, Sitecore among them)?
Cheers & thanks, Bob

Alistair Deneys

Hi Bob, Have you thought about segmenting your imports and batching the processing? And running multiple batches through seperate service calls at once? And there's no reason to throw away the idea of caching template lookups. Being that the above scenario is a migration we can somewhat forego better application design and for this case make use of HttpApplication or HttpSession, so you only need to do the lookup once for the batch / import process rather than for each doucment. Or what about turning the import around and have Sitecore pull the data from the other repository? Do you really need to run it from a console app? And I would still import the documents in batches. Processing 70,000 documents in a single hit for any system scares me :) .

Batching might have been a productive approach in other circumstances, but the length of time it takes Sitecore to import a single document varies so widely that sometimes a request to import just one of the more complex documents will time out (so you needn't worry any further about the prospect of Sitecore trying to process 70,000 documents in a single job, though that's something we do every night without breaking a sweat in another system of ours). As a result I think I'll stick with one document per request so I can repeat any requests that don't make it, without the additional complexity of re-batching. Your idea for using the session cache for the ID lookups is a good suggestion.
> Do you really need to run it from a console app?
At this point, we're looking at this limitation as one of the many factors which get folded into our assessment of which CMS product would best meet the customer's needs. I would turn the question around: Does Sitecore really need to limit access to the data using their API to web applications/services? I am reminded of the earlier years of Microsoft's entry into the server market, when they were so enmeshed in the world of the Windows desktop that their engineers saw no problem with making graphical utilities the primary (and in some cases the only) tools available for performing essential server management.
I appreciate your suggestions. I suspect that the inability to talk to the Sitecore API from a console application, as arbitrary and annoying as that limitation would be, will turn out to be a less significant factor than other issues (such as performance and quality of documentation).
Cheers, Bob

Anonymous

Hey Bob, "be at the mercy of the web server’s patience" - this is not how things work when hosting .NET code in an ASP.NET application in IIS. You are not writing your code in Page_Load and hoping the script won't time out. You'll not be at the mercy of anything.

Bob Kline

Hey Anonymous:
I'm not sure what you're trying to say. If you're saying it's possible to increase the web server's timeout settings: that's true for any web server + CGI-based tools combination, but it would be a silly way to solve the problem. If you're implying that it's good practice to restrict HTTP processing to lightweight tasks which will complete quickly, queuing up the heavy lifting to be done outside of the constraints of the web server's timeout limitations: sure, that's my point; in this case the heavy lifting was importing documents into Sitecore, and (as Alistair confirms above) it's not possible to use the Sitecore APIs without an HTTP context. If you're saying Alistair is wrong, it would be helpful if you would elaborate, with links to the evidence. If you're saying Alistair meant to say that what is required is an HTTPContext object, and that an HTTPContext object doesn't necessarily need to exist within any HTTP context: I'm prepared to believe you (it wouldn't be the most bizarre concept Microsoft has promulgated), but I'd be grateful if you could point to documentation explaining how that would work. If there's some other possible interpretation of your comment, please enlighten me.
Cheers, Bob

Let me know if I can help out. http://www.linkedin.com/pub/jason-evans/19/a19/462

Thanks for the offer. The customer decided not to go with Sitecore.

[...] Automated Testing and Sitecore part 1 – Introduction [...]

Adam

I'm trying to build a WCF service hosted from a sitecore site and still having context issues. Once I call Sitecore.Context.Database it errors out. I even followed this suggestion: http://sitecoreblog.alexshyba.com/2009/03/attach-wcf-services-to-sitecore-context.html and it made no difference. Isn't there some way to get into the API that's not using the website?

Jason Horne

It is possible to create a console app to update Sitecore, I'm working on one right now which updates media and content items name to a common naming convention.

Alistair Deneys

Hi Jason, Officially, no. The Sitecore API cannot be called without an HttpContext. I would suggest you create yourself a WCF services hosted by your Sitecore application and call into that from your console app. This is by far the easiest way of doing what you want.
I have however been successful in getting the Sitecore API (or at least parts of it) to work without an HttpContext. Check out the great article from Michael Edwards on the subject. http://blog.experimentsincode.com/index.php/2010/08/30/232

Have you had any luck using Visual Studio 2010 and the web performance test tools? I tried doing so, and performance is terrible.

Alistair Deneys

Hi Doug, Sorry, I can't say I've tried using the VS performance tools. Normally when performance tuning my Sitecore solutions I use the Sitecore debugger to help pinpoint components that have performance issues. I've also seen others use Selenium grid and jMeter to performance test their Sitecore solutions.

Leave a comment

All fields are required.