Scaling EXM

You Want To Send Massive Amounts of Email, Eh?

When Sitecore 9.0.1 or higher is installed, Email Experience Manager is automatically installed regardless of the topology that is set up. However, the performance of EXM is not optimized. In fact, it’s my opinion, that Email Experience Manager, out of the box, is purposefully scaled down to prevent accidental slowness, especially in a combined XP Single implementation. As such, there are a number of configurations and architecture that need to be considered when wanting to scale up performance.

When scaling EXM, here are the areas that needs to be considered. These areas will be expanded on in this post.

• Architecture Topology – Adding Dedicated Dispatch Servers
  • On-Prem
  • Azure PaaS
• The configuration of the Dispatch Process – Per Server
  • Optimizing Dispatch
  • Disabling Dispatch on Primary Content Management

Architecture Topology

In order to effectively scale out EXM, the rest of Sitecore needs to be scaled out sufficiently. Therefore, while EXM works in an XP Single topology, if the objective is to send more than a million emails in a short period of time, Sitecore should be implemented in an XP Scaled configuration. Which looks something like the diagram below.

Notice that the Sitecore Processing and Reporting services are on their own servers/instances and that Content Delivery is scaled out and distributed. In this configuration, the Content Management server is considered the Primary Content Management server with regard to Email Experience Manager.

Introducing the Dedicated Dispatch Server

For many versions of EXM, there has always been the concept of a dispatch server that can be configured to assist in dispatching of emails. With Sitecore 9, configuration has been simplified across the board by the introduction of the role:define variable in the web.config.

    <!-- SUPPORTED SERVER ROLES     
         Specify the roles that you want this server to perform. A server can perform one or more roles. Enter the roles in a comma separated list. The supported roles are:

         ContentDelivery
         ContentManagement
         Processing
         Reporting
         Standalone
         DedicatedDispatch
          
    Default value: Standalone
    -->
    <add key="role:define" value="ContentManagement,DedicatedDispatch"/>

Dispatch Servers are defined as both a ContentManagement and DedicatedDispatch in the role define. This is because Dedicated Dispatch servers are still content management servers. However, the Primary Content Management server initiates dispatches, not Dispatch Servers.

A Sitecore deployment can have multiple Dedicated Dispatch servers. Check with your Sitecore Technical Account Manager to determine if your license allows multiple instances.

On-Prem

In an On-Prem environment, adding Dedicated Dispatch Servers is the same process as deploying your Content Management server. The only difference is that role:define includes the additional role of DedicatedDispatch as shown above. In this configuration, the topology changes as shown below.

Primary Content Management Server Config

The most important configuration on the Primary Content Management server is the inclusion of the list of Dedicated Server URL’s that are needed. For every Dedicated Dispatch Server set up, there needs to be an FQDN listed in the <DedicatedServers> node.

<!-- Patch Config for Primary Content Management Server DDS listing -->  
<sitecore exmEnabled:require="yes" role:require="(Standalone or ContentManagement) and !DedicatedDispatch">
    <DedicatedServers>
      <!--<address value="http://dedicated.server" />-->
      <address value="https://dds1-server-domain"/>
      <address value="https://dds2-server-domain"/>
    </DedicatedServers>
  </sitecore>

Dedicated Dispatch Server Config

All that is needed to define a Dedicated Dispatch Server is to ensure the role:define is correct on the DDS.

<!-- Web.Config Application Setting -->
    <!-- SUPPORTED SERVER ROLES     
         Specify the roles that you want this server to perform. A server can perform one or more roles. Enter the roles in a comma separated list. The supported roles are:

         ContentDelivery
         ContentManagement
         Processing
         Reporting
         Standalone
         DedicatedDispatch
          
    Default value: Standalone
    -->
    <add key="role:define" value="ContentManagement,DedicatedDispatch"/>

Azure PaaS

Actual Topology on Azure PaaS, is no different from On-Prem. Install the Dedicated Dispatch Server following the Sitecore Azure Toolbox parameter settings. You can view my early blog on 9.0.1 to get an idea.

Installing Multiple DDS in Azure PaaS

DO NOT use the Azure Scale-Out function on the App Service to scale out the DDS server. This will not work, as the Sitecore Primary CM server needs a FQDN name for each instance of EXM. This means each Dedicated Dispatch Server in Azure PaaS, needs to be it’s own Web App Service.

Dispatch Process Configuration

The Dispatch process occurs on both the Content Management Server as well as each Dedicated Dispatch Servers. The note here is that, out of the box settings are not configured for optimal dispatch. Said differently, Sitecore is configured to perform low and not utilize a server’s resources to it’s full potential. This is because the dispatch process can slow down the authoring experience on a Content Management Server. Often times, it is more desirable that dispatch is completely disabled on the Content Management server.

Testing Dispatch Scale is EXTREMELY difficult to do in a non-production environment. This is because testing the dispatch of a batch of a million emails is not actually that simple. As such, the following configuration variables may need to be adjusted in production in order to optimize the dispatch. There is a level of trial and error involved in order to find the right settings.

Optimizing Dispatch

Improving and optimizing the dispatch experience (on CM and/or DDS, but mostly DDS) is controlled through the increasing of a handful of configuration variables. These variables cause Sitecore to utilize more CPU during a dispatch process. The following are out of the box settings.

    <settings>        
      <!-- The number of threads that you can use for sending messages. -->
      <setting name="NumberThreads" value="10" />

      <!-- Specifies how many sending threads can generate messages at the same time.
           Value should be no less than 1.
           Default value: Environment.ProcessorCount * 2 -->
      <setting name="MaxGenerationThreads" value="" />

      <!-- The number of recipients in each batch enqueued in the dispatch queue. -->
      <setting name="DispatchEnqueueBatchSize" value="300" />

      <!-- The number of threads that adds recipient batches to dispatch queue. -->
      <setting name="DispatchEnqueueThreadsNumber" value="4" />
        
      <!-- The number of contacts that each dispatch thread will attempt to process at a time. -->
      <setting name="EXM.DispatchBatchSize" value="100" />        
    </settings>

These values effect the two phases of dispatch: Contact Queuing and Message Dispatch.

• Contact Queuing Variables
  • DispatchEnqueueBatchSize
    • Increasing this will consume more CPU and RAM, but can increase the speed for adding large contact lists to the queue.
  • DispatchEnqueueThreadsNumber
    • Number of threads to create to batch users. Increasing will consume more CPU.
• Message Dispatch
  • NumberThreads
    • Increases CPU. I have set this significantly higher on DDS servers (to the magnitude of 100+ threads.
    • This value must be equal to the exm/eds/connectionPoolSettings/maxPoolSize setting. So if you significantly increase NumberThreads, you will want to increase the maxPoolSize.
  • MaxGenerationThreads
    • Increases CPU. I have set this equally high on DDS Servers.

Sitecore has a very good page detailing ALL of the variables that can be modified. However, the variables listed above are definitely the most important and have the largest effect.

Performance tuning of EXM is best when monitored during actual dispatches. Sitecore provides a Dispatch Summary page that displays a large amount of data about dispatches, as dispatches are going. You can view this page at /sitecore/admin/dispatchsummary.aspx. See Sitecore documentation for more information.

https://doc.sitecore.com/developers/exm/91/email-experience-manager/en/performance-tuning-and-the-exm-performance-measurement-tool.html

Disabling Dispatch on CM

A common change in out-of-the-box behavior is disabling the dispatch on the Content Management server. While not straight forward, disabling dispatch is done via configuration changes. Specifically, the <DispatchNewsletter> pipeline is where changes occur. Let’s take a look at the default pipeline.

      <DispatchNewsletter>
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.CheckPreconditions, Sitecore.EmailCampaign.Cm" resolve="true" />
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.MoveToQueuing, Sitecore.EmailCampaign.Cm" resolve="true" />
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.DeployAnalytics, Sitecore.EmailCampaign.Cm" resolve="true" />
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.PublishDispatchItems, Sitecore.EmailCampaign.Cm" resolve="true" />
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.QueueMessage, Sitecore.EmailCampaign.Cm" resolve="true"/>
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.MoveToProcessing, Sitecore.EmailCampaign.Cm" resolve="true" />
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.LaunchDedicatedServers, Sitecore.EmailCampaign.Cm" />
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.SendTestMessage, Sitecore.EmailCampaign.Cm" resolve="true" />
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.SendMessage, Sitecore.EmailCampaign.Cm" resolve="true">
            <!-- The number of milliseconds to wait after dispatch is completed/aborted/paused in order to ensure logging statistics are updated across all dispatch servers. Can be set to 0 if there are no dedicated dispatch servers configured. -->
            <Sleep>2000</Sleep>
        </processor>
        <!-- The WaitForDispatchToFinish pipeline processor should only be enabled if you have at least one dedicated dispatch server enabled.If you enable this processor you should disable the SendMessage processor. -->
        <!--<processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.WaitForDispatchToFinish, Sitecore.EmailCampaign.Cm" resolve="true">
          <TimeToWaitBetweenChecks>1000</TimeToWaitBetweenChecks>
        </processor>-->
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.MoveToSent, Sitecore.EmailCampaign.Cm" resolve="true"/>
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.NotifyDispatchFinished, Sitecore.EmailCampaign.Cm" resolve="true"/>
        <processor type="Sitecore.EmailCampaign.Cm.Pipelines.DispatchNewsletter.FinalizeDispatch, Sitecore.EmailCampaign.Cm" resolve="true" />
      </DispatchNewsletter>

1. Include the WaitForDispatchToFinish processor on Primary CM server.
   1. Increase TimeToWaitBetweenChecks to 5000.
2. Remove or disable SendMessage processor on Primary CM Server.
   1. NOTE: On DDS servers, reduce the <Sleep> parameter to something very small… like 5. This improves DDS speed.

1 Day Left in 25 Days of EXM

Last but not least, on Christmas Day, December 25th, the last day of our 25 Days of EXM, I will be posting a pretty intense Developer Guide to EXM, showing off a number of API’s and Pipelines that help make EXM run.

Did You Miss EXM Live!?

You’re in luck!  You can now view EXM Live! On-Demand on YouTube!

Tagged as: 25days, EXM

Categorised in: EXM, Sitecore