Planning for sending and receiving emails at scale

From PegaWiki
This is the approved revision of this page, as well as being the most recent.
Jump to navigation Jump to search



Description Considerations when planning to send and receive email on a large scale
Version as of 8.5
Application Platform
Capability/Industry Area Data Integration



Sending and receiving email at low to moderate volume and scale is often straightforward. However, at high volume and scale there are additional aspects to consider, which require a bit of planning to determine the best configuration to meet your specific needs. This article will outline some of the considerations and configuration options that together will help you achieve these goals.

Before you begin[edit]

Please familiarize yourself with the About Email article on Pega Community.

Inbound Email[edit]

When planning inbound email requirements, the key components to understand are:

  • How many email inboxes will you need to receive inbound email from?
  • What is the peak, average and daily inbound volume that is seen on average in each inbox?
  • How quickly do emails needs to be processed and available to an agent?
  • How long does it take, on average, to process and create the corresponding case/cases for the email?
  • What is the sleep time between runs, and the maximum number of records to process for each run?

The following table of examples takes the factors listed above and shows the average and peak listener thread concurrency needed to process emails within 1 hour of being received by the mail provider.

  • Concurrent threads per hour uses 3,600 seconds in an hour, divided by average time per message.
  • The 3 main factors that will drive your throughput are the time it takes to process an individual message, the max messages per iteration, and the sleep period between cycles. Try playing with the numbers and see if your processing is bound by the maximum messages or by time by adjusting the columns.
  • When planning your capacity, always include additional buffer capacity. The examples below assume 20%. Numbers should be rounded up.
Email Account Name Peak Inbound Volume

per hour

Average Daily Volume Average processing

time in seconds for

an individual message

Maximum Number

of messages to process before

latency period

Latency/Sleep

period in seconds

Max Processing

Time in seconds

Seconds Available

Per Hour to Process

Processing Capacity

Per Hour

Concurrent Listener

threads required

With 20%

Capacity Buffer

sales@pegaSample.com 850 6,120 3.1 500 100 500 3000 968 1 1
offers@pegaSample.com 12,000 86,400 3.1 500 100 500 3000 968 12 15
marketing@pegaSample.com 2,800 20,160 3.1 500 100 500 3000 968 3 3
members@pegaSample.com 300 2,160 5 500 100 500 3000 600 1 1
support@pegaSample.com 6,500 46,800 3.1 500 100 500 3000 968 7 8

The screenshot below shows the email listener configuration properties that drive throughput.

EmailSettings2.jpg

Once you have determined the maximum number of threads required, with a buffer, there are a few further configuration options to keep in mind:

  • When setting up your listeners for a high volume inbound situation, ensure that you segment the background processing of inbound email from interactive user sessions to avoid performance contention. One option for email listeners is the ability to control which nodes or what class of node the listener threads will run on. In the screenshot below, the node classification-based startup has been selected, and the node type of background processing is selected to ensure that only nodes of this type will be able to run this listener.
  • You will also need enough nodes tagged with this node classification to handle the number of threads. Try to limit the threads per node to 6 or less. Thus, if your concurrent listener count total is 24 (as in the above example), you should have 4 nodes dedicated to email processing.
  • Try to limit the number of inboxes that you listen to. As the number of inboxes grows, the management of listeners becomes more difficult.

The screenshot below shows a graphical representation of the moving parts in an email listener:

Email Threads diagram.jpg

Outbound Email[edit]

With outbound email, there are fewer factors to consider within Pega platform. Some of the metrics that need to be understood for outbound email include:

  • How many unique email addresses do you want to send email out on?
  • Does your email provider have any per-account or global throttles in place that may limit your ability to reach the volume of emails you want to send out in a timely fashion?
  • The one item you may need to tune is the number of concurrent TCP connections set on your operating system, which limits the number of connections to your mail service provider. This is often port 25.

Results[edit]

With the above detail you should have a better understanding of the factors that go into planning and setting up large-scale email shots, both inbound and outbound.