Extending the Sitecore image processor

You've no doubt used the dynamic image manipulation capabilities of Sitecore before. This feature allows you to dynamically manipulate images from the media library using the query string. For example, to request an image with width 150 I would append w=150 into the query string:

http://domain/~/media/images/myimage.ashx?w=150

Sitecore will then dynamically scale this image to 150 pixels wide (whist preserving aspect ratio) and send the 150 width image down the wire to me. This means I no longer have to manage a set of thumbnails for each image I upload into Sitecore. What's more, Sitecore only processes this image once. After the first request the image is cached to disk, so we don't double up on processing for subsequent requests.

Dynamic image manipulation is achieved in Sitecore by using a pipeline, the getMediaStream pipeline, so I can extend, tweak and alter the pipeline for my purposes.

So let's add some capabilities to this great feature. I'm going to add a processor to the pipeline to allow flipping of the image requested. First thing to do is create a processor class to be hooked into the pipeline. We'll add the standard "Process" method. We also need to make sure this method accepts the correct pipeline arguments class.

namespace ImageProc
{
  public class Transform
  {
    public void Process(GetMediaStreamPipelineArgs args)
    {
    }
  }
}

Next we need to pick a query string key to invoke the custom processor on. For this example I'll use "flip" and invoke the processor if it's set to "1" in the query string.

if ((string)args.Options.CustomOptions["flip"] == "1")
{
}

Now for the bit that does the work. The following code uses the standard .net Bitmap class to flip the image upside down. You'll notice most of the code below is used to convert the image data out of the stream and then back into the stream at the end.

var bm = (Bitmap)Bitmap.FromStream(args.OutputStream.Stream);
bm.RotateFlip(RotateFlipType.RotateNoneFlipXY);
var stream = new MemoryStream();
bm.Save(stream, ImageFormat.Png);
args.OutputStream = new MediaStream(stream, "png", args.MediaData.MediaItem);

Now to insert our custom processor into the getMediaStream pipeline so we can invoke it. Open web.config and find the pipeline definition. Add the processor at the end of the existing processors.

<processor type="ImageProc.Transform,ImageProc"/>

Now request some media from Sitecore and append flip=1 in the query string.

http://domain/~/media/image/myimage.ashx?flip=1

Here's the original image:

And this is what we get:

So you can see how easy it is to extend the image processor. But that was a pretty simple example. How about something cooler?

Have you heard of the image resizing technique called seam carving? A collegue of mine at Next Digital, Joel Wang, recently sent me a video of seam carving in action. When I watched it, it was truly one of those "Wow" moments. Seam carving is a way in which to resize an image, without affecting the aspect ratio like stretching and scaling does. Seam carving removes lines from the image to preserve the aspect ratio of the important things in the image. This technique was created by Shai Avidan and Ariel Shamir. The best way to get a feel for this technique is to find a video of it in action (there's a heap out there).

There are already a few implementations of this technique out there. I haven't found a .net library, but I did manage to find a Windows implementation that could be invoked using the command line. So I can now create a dynamic image processor that uses seam carving.

For this example I'll be using a release from the CAIR project. CAIR is an implementation of the seam carving technique above. The release includes source code (so I can create a .net implementation at some point :) ) and also a Windows command line executable. Luckily we can invoke executables from inside .net code.

So onto the processor.

using System.Diagnostics;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;
using System.Web;
using Sitecore.Data;
using Sitecore.Resources.Media;

namespace ImageProc
{
  public class Carver
  {
    const string CAIR_PATH = "c:\\temp\\cair\\cair.exe";

    public void Process(GetMediaStreamPipelineArgs args)
    {
      if (args.Options.CustomOptions["cw"] != null ||
        args.Options.CustomOptions["ch"] != null)
      {
        HttpContext context = HttpContext.Current;
        string folder = context.Server.MapPath
          (Sitecore.Configuration.Settings.TempFolderPath +
            "\\ImageProc\\");
        string temp = folder + ID.NewID.ToString();
        string procfile = temp + "_proc";
        if (!Directory.Exists(folder))
          Directory.CreateDirectory(folder);

        var bm = (Bitmap)Bitmap.FromStream(
          args.OutputStream.Stream);
        bm.Save(temp, ImageFormat.Bmp);

        int targetWidth = bm.Width;
        int targetHeight = bm.Height;

        if (args.Options.CustomOptions["cw"] != null)
          int.TryParse(args.Options.CustomOptions["cw"],
            out targetWidth);

        if(args.Options.CustomOptions["ch"] != null)
          int.TryParse(args.Options.CustomOptions["ch"],
            out targetHeight);

        var proc = new Process();
        proc.StartInfo.FileName = CAIR_PATH;
        proc.StartInfo.Arguments = string.Format(
          "-I {0} -O {1} -X {2} -Y {3}", temp, procfile,
            targetWidth, targetHeight);
        proc.Start();
        proc.WaitForExit();

        if (File.Exists(temp))
          File.Delete(temp);

        if (proc.ExitCode == 0)
        {
          var fstream = File.OpenRead(procfile);
          var pbm = (Bitmap)Bitmap.FromStream(fstream);
          var stream = new MemoryStream();
          pbm.Save(stream, ImageFormat.Png);
          args.OutputStream = new MediaStream(stream, "png",
            args.MediaData.MediaItem);
          fstream.Close();

          if (File.Exists(procfile))
            File.Delete(procfile);
        }
        else
          Sitecore.Diagnostics.Log.Error(
            "Error during CAIR execution: "
            + proc.ExitCode.ToString(), this);
      }
    }
  }
}

You'll see that most of the code above is to do with saving the image to disk to allow CAIR to manipulate it, then shooting the result back down the stream. I've used "cw" and "ch" for "carve width" and "carve height" so I can choose the target size of the image. I of course only have to supply 1 of these, or I can supply both. To invoke the executable I'm using the Process class. Note that I need to wait for the execution of this process to complete before I continue, and I can do this by calling the WaitForExit method on the Process class. The arguments passed to the executable are pretty straight forward. I supply the input, the output file I want generated, the target width and target height. For a full description or for other options, just execute CAIR without any arguments in a command prompt and the help will be displayed.

Don't forget to add the processor to the pipeline as we did above.

Now to try this out. The above image is 300 pixels wide. Let's see what it looks like, carved down to 150 pixels.

http://domain/~/media/image/myimage.ashx?cw=150

Note how the antelope in the image still looks OK. It hasn't been skewed or compressed and it's aspect ratio is still intact. The background in the image is what got removed, but the antelope was left alone.

At this point I hope you're going "WOW!" just like I was when I first saw this technique.

But does this new way of scaling images affect performance? Well, yes. But only the first request will take that performance hit. After that, Sitecore will cache the processed image on disk and subsequent requests don't take the performance hit of reprocessing the image.

Now, it wouldn't be too hard to tweak the code example above to use a different external application to do some image processing for us. In fact, GIMP (the GNU image manipulation program) can be invoked through a command line. This application is an open source image processing and manipulation tool similar to Photoshop. So anything I can do in GIMP, I could automate through the batch interface and link that into a custom image processor.

OMG, I've just opened a huge world of possibilities for dynamic image processing with Sitecore and external image processing applications.