NullReferenceException in SessionStateItemCollection.Remove() in Sitecore 8.2 update 3

After upgrading to Sitecore 8.2 update 3 I received the following error at random in the content editor and experience editor.

[NullReferenceException: Object reference not set to an instance of an object.]
  System.Collections.Specialized.NameObjectCollectionBase.BaseGetKey(Int32 index) +35
  System.Collections.Specialized.NameObjectCollectionBase.BaseRemove(String name) +127
  System.Web.SessionState.SessionStateItemCollection.Remove(String name) +106
  Sitecore.Shell.Applications.Dialogs.ItemLister.SelectItemOptions.Parse(List`1& list, String handle, String postfix, Database database) +229
  Sitecore.Shell.Applications.Dialogs.ItemLister.SelectItemOptions.Parse() +874
  Sitecore.Shell.Applications.Dialogs.SelectItem.SelectItemForm.OnLoad(EventArgs e) +90
  Sitecore.Shell.Applications.Dialogs.SelectRendering.SelectRenderingForm.OnLoad(EventArgs e) +47

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
  System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
  System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +76
  System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +211
  System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) +35
  Sitecore.Web.UI.Sheer.ClientPage.OnLoad(EventArgs e) +337
  System.Web.UI.Control.LoadRecursive() +71
  System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +3178

After a lot of dialog with Sitecore support, we were able to determine that the error is caused by a concurrency issue with the System.Web.SessionState.SessionStateItemCollection.

The SessionStateItemCollection is responsible for storing key value pairs in the HttpContext.Current.Session object and the error was due to the SessionStateItemCollection’s private member _entriesArray containing an unexpected null value in the middle of the collection.

When the value is removed from the SessionStateItemCollection by key the code iterates the _entriesArray. The Key property of each item is compared to the key passed as a parameter, but there is no null check when the items in the _entriesArray are accessed which means that a NullReferenceException is thrown when the iteration reaches the null value in the middle of the array.

The null value can occur in the middle of the _entriesArray because even though the methods that remove items from the array are locked and thread safe, none of the methods or indexers that adds new values to the SessionStateItemCollection are. This means that a race condition can occur where an item is being removed from the array at the exact same time as a new value is added to the array where the array is resized by the current size + 1. This means that the array effectively ends up with a null somewhere in the array.

This error has been reported to Microsoft. However, it took a lot of digging and help from Sitecore support to figure out why it was throwing errors in Sitecore.

In the end it turned out that the culprit was Sitecore.ContentTesting.Pipelines.HttpRequest.SetReadOnlySessionStateForScreenshot where the following call caused the error.

HttpContext.Current.SetSessionStateBehavior(SessionStateBehavior.ReadOnly);

It is not obvious why this should cause an error as the code is only calling standard .NET functionality but for some reason it creates a race condition.

The only solution at the moment of writing as recommended by Sitecore support is to remove the SetReadOnlySessionStateForScreenshot pipeline processor.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <pipelines>
      <httpRequestBegin>
        <processor type="Sitecore.ContentTesting.Pipelines.HttpRequest.SetReadOnlySessionStateForScreenshot, Sitecore.ContentTesting">
          <patch:delete />
        </processor>
      </httpRequestBegin>
    </pipelines>
  </sitecore>
</configuration>