Implementing a Differentiated robots.txt on a Sitecore Multisite

I recently answered a question on how to implement a differentiated robots.txt in Sitecore when you have multiple websites running in the same instance. I know that to some this may seem like a trivial task but I can understand that this may not be as obvious to someone less experienced with Sitecore and asp.net in general, so I thought that I would make a quick post with one possible solution.

This can also be used for other scenarios, like icon.ico or sitemap.xml files where you usually have a single static file on the server but need to handle multiple sites.

First we need to implement a processor that inherits from Sitecore.Pipelines.HttpRequest.HttpRequestProcessor like the following.

using System;
using System.IO;
using System.Runtime.Caching;
using System.Web;
using Sitecore;
using Sitecore.Data.Items;
using Sitecore.IO;
using Sitecore.Pipelines.HttpRequest;

namespace SEO
{
 public class RobotsHandler : HttpRequestProcessor
 {
  private const string RobotsFileName = "/robots.txt";
  private const string CacheKey = "SEO-ROBOTS";

  public int CacheExpirationInMinutes { get; set; }

  public override void Process(HttpRequestArgs args)
  {
   Uri url = HttpContext.Current.Request.Url;
   if (!url.PathAndQuery.EndsWith(RobotsFileName))
    return;

   string robotsText = GetDefaultRobotsText();

   if (Context.Site != null && Context.Database != null)
   {
    Item startItem = Context.Database.GetItem(Context.Site.StartPath);
    if (!string.IsNullOrEmpty(startItem?["RobotsContent"]))
    {
     robotsText = startItem["RobotsContent"];
    }
   }

   args.AbortPipeline();
   args.Context.Response.ContentType = "text/plain";
   args.Context.Response.Write(robotsText);
   args.Context.Response.End();
  }

  private string GetDefaultRobotsText()
  {
   string value = MemoryCache.Default.Get(CacheKey) as string;
   if (value != null)
    return value;

   if (!FileUtil.Exists(RobotsFileName))
    return string.Empty;

   using (StreamReader streamReader = new StreamReader(FileUtil.OpenRead(RobotsFileName)))
   {
    value = streamReader.ReadToEnd();
   }
   MemoryCache.Default.Set(CacheKey, value, DateTimeOffset.UtcNow.AddMinutes(CacheExpirationInMinutes));
   return value;
  }
 }
}

In the processor, we are simply detecting if the request is for a robots.txt file and then we fetch the contents of the field RobotsContent on the start item of the current site. Of course, this field could be placed somewhere else within the context of the current site but for the sake of brevity let us keep it simple. If the context site or database is empty or the start item does not contain a robots text then we have fall back that will retrieve the contents of a static robots.txt file if placed in the root of the website.

With the processor in place, we now need to hook it up to the httpRequestBegin pipeline in Sitecore like the following.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
 <sitecore>
  <pipelines>
   <httpRequestBegin>
    <processor patch:after="*[@type='Sitecore.Pipelines.HttpRequest.DatabaseResolver, Sitecore.Kernel']" type="WDH.Feature.SEO.Pipelines.RobotsHandler, WDH.Feature.SEO" >
     <CacheExpirationInMinutes>720</CacheExpirationInMinutes >
    </processor>
   </httpRequestBegin>
  </pipelines>
 </sitecore>
</configuration>

With the pipeline step in place the processor that we built will now handle all incoming requests for the robots.txt file and server a site-specific response.

I hope that someone will find this useful either for serving robots.txt files or any other kind of static files.