Load testing scenarios
Load testing scenarios
Pega recommends starting load testing early in the development cycle - as soon as application services and data flows have been unit tested.
The initial import and export baseline testing and load testing service scripts should be available for testing in the QA environment at least 6 weeks prior to the anticipated Go Live production date. Full volume, end-to-end testing should be performed in the production environment at least 3 weeks prior to production Go Live.
Load testing does not require a special environment or a full set of data. The purpose of load testing is to confirm that the data import and export processes and the channel and data integration services can process without intervention, and to document baseline conditions as input to application and infrastructure performance tuning.
In addition, load testing using Monte Carlo and whatever customer-related data that may be available provides substantial feedback to application developers about the overall quality of the Pega Customer Decision HubTM application by testing against a substantially wider variety of data than is typically used in data transforms and unit tests. In addition, it helps discover potential operational issues early in development.
Batch data import and export load tests should typically be run once each sprint to ensure that development changes have not impacted overall processing.
Service processing (e.g. Make Decision, Capture Response, Real Time Data import and export, and event or tag processing) load tests should typically be run daily and their results should be published to both the technical development team and the business actions, strategy and model configuration teams to ensure that business changes are not impacting overall performance.
Batch data import and ingestions
For data imports, it is important to check end-to-end durations (in seconds) as soon as the import and ingestion process is configured and available for testing as follows:
- Once the process is configured, use Monte Carlo to generate a sample in a file dataset to use as source of import for a set of sample runs.
- Based on the sample runs, use the measured throughput to extrapolate the end-to-end duration and see it if fits in the timeline agreed with the client.
Once larger environments are available and the data import and ingestion process are established for those environments, it is important to run a more extensive set of multiple imports to fully test the load and capacity of the solution.
Typically, this will be done by first running the data import and ingestion with nothing else executing and taking screen shots of the data import data flows on a regular basis (e.g. every 20 minutes for ~100 million records.) That data from the data flow screenshots is then put in a spreadsheet to compute average throughput between each screenshot, and to also review timings for each phase of the import run (S3 to staging, and then staging to xCAR.) This should be done for each import file set separately, and for multiple imports running simultaneously to understand the processing impact. This provides the application baseline.
Once the baseline is understood and the Make Decision Container and Capture Response services are available, then an additional set of the Batch Data Extract and Export load runs can be executed to understand the impact between import data flows and services.
Batch data extract and export
Likewise, for data extracts and exports, it is important to check end-to-end durations (in seconds) as soon as the extract and export process is configured and available for testing as follows:
- Once the process is configured, use Monte Carlo to generate a sample in a file dataset to use as source of data to be extracted for a set of sample runs.
- Based on the sample runs, use the measured throughput to extrapolate the end-to-end duration and see it if fits in the timeline agreed with the client.
Once larger environments are available and the data extract and export process are established for those environments, it is important to run a more extensive set of multiple exports to fully test the load and capacity of the solution.
At this point, testing should use actual data from decisioning and will require the Make Decision Container and Capture Response services and some base next-best-action strategies to be configured and generating interaction history entries, ADM snapshots, and other data. The xCAR database and related customer data for this testing can be test or Monte Carlo generated data. The intent is to have substantial data in interaction history and other tables and files to use in the testing. For large enterprises, the typical testing would be performed with 1 million, 10 million, 50 million and 100 million transactions.
The recommendation is to initially run the extract and exports with nothing else executing and take screen shots of the data extract data flows on a regular basis (e.g. every 20 minutes for ~100 million records.) The data from the data flow screenshots are then put in a spreadsheet to compute average throughput between each screenshot and to also review timings for each phase of the export run (IH to staging and then staging to S3.) This should be done for each file set to be exported separately, and for multiple exports running simultaneously to understand the processing impact. This provides a baseline.
Once the baseline is understood and the Make Decision Container and Capture Response services are available, then an additional set of the Batch Data Extract and Export load runs are executed while processing Make Decision and Capture response services to understand the impact on processing and then further testing sets executed with simultaneous services and import processing.
Batch strategy execution
To provide a baseline for Next-Best-Action Designer strategy execution time, as soon as the initial taxonomy, action set and initial strategies are configured in NBA Designer, a sample Monte Carlo-generated set of customer data should be created and used to execute Next-Best-Action Designer in batch.
This will provide a baseline Next-Best-Action Designer strategy execution time latency, in milliseconds, that should be evaluated against what has been agreed with the client.
Interaction history summarization rate
To provide a baseline for interaction history summarization, Monte Carlo data generation should be used to generate interaction history records at a rate equal to expected peak processing. During execution, the summarization rate should be reviewed and evaluated as follows:
- On every read, is there a need to read from interaction history instead of relying on interaction history summaries only?
- For sustained processing (at least 1 hour and up to 24 hours) is the throughput (records per second) sufficient to ensure that interaction history summarization is keeping up with the incoming number of responses?
Event processing rate
To provide a baseline for event processing, Monte Carlo data generation should be used to generate events at a rate equal to expected peak processing. During execution, the summarization rate should be reviewed and evaluated as follows:
- For sustained processing (at least 1 hour and up to 24 hours) is the throughput (records per second) sufficient to ensure that events are being processed at a rate that is keeping up with the incoming number of events?
Make Decision / Capture Response Services
For the NBA Make Decision, Container, Capture Response and Real Time Event Processing services, we recommend creating a set of JMeter (or other load tool) load scripts, such as those identified in the Services testing scenarios section below. The approach is to set up test runs for each scenario that can be manually and automatically executed.
For the load scripts, the test customer IDs should be externalized in the load test tool, and should reflect a representative of .5-1% of the overall customer base. In addition, other parameters that should be externalized in the load test tool include page / placement requests and environment-specific factors, such as URLs that will allow the tests to be run against the different Pega development, test and simulation environments.
There are 2 areas that should be evaluated as part of the initial and end-to-end services load testing:
- Throughput (records per second)
- Latency profile (average milliseconds for processing all transactions, % of transactions processed under 150 milliseconds and % of transactions processed over 150 milliseconds)
Data update services
Pega Customer Decision Hub expects that real time data will be pushed to the NBA application as changes in the underlying systems of record occur. The data can be pushed by using a stream service such as Kafka, or by consuming an exposed service. Regardless of the transport, the data will be updated using a data flow. The processes for real time updates should be included in the load test scenarios as soon as the data flows are available for testing.
For the load scripts, we recommend externalizing the test data in the load test tool to provide a random set of data attributes representative of at least .1% of the overall records in the associated tables. In addition, other parameters that should be externalized in the load test tool include environment-specific factors, such as URLs that will allow the tests to be run against the different Pega development, test and simulation environments.
Load and performance testing: assumptions and objectives
Pre-Go Live performance testing scenarios should be executed using the production environment to confirm the production environment and establish the production environment performance baseline.
This will include end-to-end data import processing, services execution and end-to-end data export processing.
An agreed subset of the previously configured and executed Load Test scenarios is used as the basis for executing the performance tests. The tests should be executed using production data and the performance test parameters should be reflective of actual expected production execution.
The load tests for the data import and export processes should be executed at full expected volume both on a standalone basis and while services are executing. Load tests for the services should be executed at 50%, 100%, 150% and 200% of expected peak load to establish the production environment baseline.
Once all performance testing is completed, the various data tables, files and data sets should be truncated to avoid conflict with actual production.
At the time the production baseline performance testing is executed, an equivalent set of standardized tests should be executed in the Testing / Simulation environment that will be used for subsequent comparison as identified in the Post “Go Live” testing below.
Post-Go Live regular automated performance testing should be executed in one of the Testing / Simulation environments that will be sized differently than the production environment.
Typically, this environment will contain the code that will be promoted to production and will also represents the production as-is until the next production deployment in case something needs to be tested and reviewed.
These tests should be automatically run each night to identify and report relative performance from the day to day load runs in the Testing / Simulation environment against the baseline established as part of the pre-Go Live testing in the Testing / Simulation environment. The results should be reviewed each day to identify any direct issues (e.g. failures in logs) and indirect issues (e.g. changes in response time / time to execute a given set of tests).
Services testing scenarios
A typical set of testing scenarios for the services will include the following:
- Scenario1: Web channel makeDecision and captureActivity. The load tool executes 1 makeDecision service request and upon receiving the response, immediately executes up to 4 resulting captureActivity service request responses (1 for each action up to 4.) The captureActivity responses should have 4 views and 1 click for the last action. After the click, the load test tool will wait 30 seconds and executes the next makeDecision request. The ratio of views to makeDecision requests and the ratio of clicks to makeDecision requests and the wait time after the click until the next makeDecision is executed should be variables in control tables.
- Scenario2_MD: Web channel makeDecision only. The load tool executes 1 makeDecision services request and upon receiving the response waits 30 seconds and executes the next makeDecision request. The wait time between makeDecision requests should be a variable in control tables.
- Scenario3: Mobile channel makeDecision and captureActivity. The load tool executes 1 makeDecision services request and upon receiving the response, immediately executes 1 resulting captureActivity service. The captureActivity response should be 1 view followed by 1 click. After the click, the load test tool will wait 30 seconds and executes the next makeDecision request. The ratio of views to makeDecision requests and the ratio of clicks to makeDecision requests and the wait time after the click until the next makeDecision is executed should be variables in control tables.
- Scenario4_MD: Mobile channel makeDecision only. The load tool executes 1 makeDecision services request and upon receiving the response waits 30 seconds and executes the next makeDecision request. The wait time between makeDecision requests should be a variable in control tables.
- Scenario5: Agent channel makeDecision and captureActivity. The load tool executes 1 makeDecision services request and upon receiving the response, waits 20 seconds and executes up to 4 resulting captureActivity service request responses (1 for each action up to 4.) The captureActivity responses should have 1 accept for the first action and rejects for the remaining actions. The load test tool will wait 120 seconds and execute the next makeDecision request. The ratio of accepts to makeDecision requests and the ratio of rejects to makeDecision requests and the wait time after the last response until the next makeDecision is executed should be variables in control tables.
- Scenario6_MD: Agent channel makeDecision only. The load tool executes 1 makeDecision services request and upon receiving the response waits 120 seconds and executes the next makeDecision request. The wait time between makeDecision requests should be a variable in control tables.
- Scenario7: Tag processing (optional for deliveries that include a requirement to process digital tags and event from various sources.) The load test tools execute a tag transaction / event every 20 seconds. The wait time between tag requests should be a variable in control tables.
- Scenario8: Real-time data processing (optional for projects that include requirements to process data in real time from systems of records.) The load test tools execute a real time data update request transaction every 10 seconds. The wait time between real time update requests should be a variable in control tables.
Typical test cases
Following are typical test cases that have been used in previous delivery projects. These test cases are for scenarios 1, 2 and 7. Similar test cases can be articulated for scenarios 3 & 5 following those identified for scenario 1; for scenarios 4 & 6 following those identified for scenario 2; and for scenario 8 following those identified for scenario 7.
User = load test user
For stress testing, ramp-up of users in the high-volume scenarios should be extended until the data flows fail with statistics recorded at each step ramp-up.
| ID: | 1. |
| Scenario Name: | Baseline Test 1 |
| Description: | Run Scenario2 for 1/2 hour at a rate of 2 “makeDecision” transactions per second (TPS) for 1 Customer. |
| Steps: | 1. User executes “makeDecision” service request, receives responses.
2. User waits 30 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 12 users (should generate ~10 “makeDecision” responses per second.)
Duration: 1/2 Hour Response Time: 200 milliseconds for Step 1. |
| ID: | 2. |
| Scenario Name: | Baseline Test 2 |
| Description: | Run Scenario1 for 1/2 hour at a rate of 2 “makeDecision” transactions per second (TPS) for 1 Customer. |
| Steps: | 1. User executes “makeDecision” service request, receives responses.
a. Change Customer for each “makeDecision” request. 2. User executes 4 “captureActivity” views (1 each for the first 4 offers.) 3. User then executes 1 “captureActivity” click (click is for the last Action presented.) User waits 30 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 12 users (should generate ~10 “makeDecision” responses per second.)
Duration: 1/2 Hour Response Time: 200 milliseconds for Step 1. |
| ID: | 3. |
| Scenario Name: | Baseline Test 3 |
| Description: | Run Scenario7 for 1/2 hour at a rate of 50 “Tag” transactions per second (TPS) for 1 Customer. |
| Steps: | 1. User executes 1 “captureActivity” Tag “firing” (1 Tag only and 1 Customer only.)
2. User waits 20 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 150 users (should generate ~50 “captureActivity” responses per second.)
Duration: 1/2 Hour Response Time: N/A – test passes if volume is handled. |
| ID: | 4. |
| Scenario Name: | Expected Volume Test 1 |
| Description: | Run Scenario2 for 1 hour at a rate of 100 “makeDecision” transactions per second (TPS) rotating through Customers. |
| Steps: | 1. User executes “makeDecision” service request, receives responses.
a. Change Customer for each “makeDecision” request. 2. User waits 30 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 120 users (should generate ~100 “makeDecision” responses per second.)
Duration: 1 Hour Response Time: 200 milliseconds for Step 1. |
| ID: | 5. |
| Scenario Name: | Expected Volume Test 2 |
| Description: | Run Scenario1 for 1 hour at a rate of 100 “makeDecision” transactions per second (TPS) rotating through Customers. |
| Steps: | 1. User executes “makeDecision” service request, receives responses.
a. Change Customer for each “makeDecision” request. 2. User executes 4 “captureActivity” views (1 each for the first 4 offers.) 3. User then executes 1 “captureActivity” click (click is for the last Action presented.) User waits 30 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 120 users (should generate ~100 “makeDecision” responses per second.)
Duration: 1 Hour Response Time: 200 milliseconds for Step 1. |
| ID: | 6. |
| Scenario Name: | Expected Volume Test 3 |
| Description: | Run Scenario7 for 1 hour at a rate of 500 “Tag” transactions per second (TPS) rotating through Customers. |
| Steps: | 1. User executes 1 “captureActivity” Tag “firing” (1 Tag only and changing customer each transaction.)
2. User waits 20 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 1,500 users (should generate ~500 “captureActivity” responses per second.)
Duration: 1 Hour Response Time: N/A – test passes if volume is handled. |
| ID: | 7. |
| Scenario Name: | Double Expected Volume Test 1 |
| Description: | Run Scenario2 for 1 hour at a rate of 200 “makeDecision” transactions per second (TPS) rotating through Customers. |
| Steps: | 1. User executes “makeDecision” service request, receives responses.
a. Change Customer for each “makeDecision” request. 2. User waits 30 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 240 users (should generate ~200 “makeDecision” responses per second.)
Duration: 1 Hour Response Time: 200 milliseconds for Step 1. |
| ID: | 8. |
| Scenario Name: | Double Expected Volume Test 2 |
| Description: | Run Scenario1 for 1 hour at a rate of 200 “makeDecision” transactions per second (TPS) rotating through Customers. |
| Steps: | 1. User executes “makeDecision” service request, receives responses.
a. Change Customer for each “makeDecision” request. 2. User executes 4 “captureActivity” views (1 each for the first 4 offers.) 3. User then executes 1 “captureActivity” click (click is for the last Action presented.) User waits 30 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 240 users (should generate ~200 “makeDecision” responses per second.)
Duration: 1 Hour Response Time: 200 milliseconds for Step 1. |
| ID: | 9. |
| Scenario Name: | Double Expected Volume Test 3 |
| Description: | Run Scenario7 for 1 hour at a rate of 1000 “Tag” transactions per second (TPS) rotating through customers. |
| Steps: | 1. User executes 1 “captureActivity” Tag “firing” (1 Tag only and rotating through customers.)
2. User waits 20 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 3,000 users (should generate ~1,000 “captureActivity” responses per second.)
Duration: 1 Hour Response Time: N/A – test passes if volume is handled. |
| ID: | 10. |
| Scenario Name: | Ramped Volume Test |
| Description: | Run Scenario1 for 2 1/2 hours starting at a rate of 200 “makeDecision” requests per second (TPS) and incrementing to 1,000 “makeDecision” requests per second rotating through all Customers. |
| Steps: | 1. User executes “makeDecision” service request, receives responses.
a. Change Customer for each “makeDecision” request. 2. User executes 4 “captureActivity” views (1 each for the first 4 offers.) 3. User executes 1 “captureActivity” click (click is for the last action received.) 4. User waits 30 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users:
Start with 240 users (should generate ~200 “makeDecision” per second and 1,000 “captureActivity” transactions per second.) After ½ hour, increment to 480 users (should generate ~400 “makeDecision” requests per second and 2,000 “captureActivity” responses per second.) After ½ hour, increment to 720 users (should generate ~600 “makeDecision” requests per second and 3,000 “captureActivity” responses per second.) After ½ hour, increment to 960 users (should generate ~800 “makeDecision” requests per second and 4,000 “captureActivity” responses per second.) After ½ hour, increment to 1,200 users (should generate ~1,000 “makeDecision” requests per second and 5,000 “captureActivity” responses per second.) Duration: 2 1/2 Hours Response Time: 200 milliseconds for Step 1 and volume handled for Step 3. |
| ID: | 11. |
| Scenario Name: | Soak Test |
| Description: | Run Scenario1 for 10 hours starting at a rate of 100 “makeDecision” requests per second (TPS) incrementing through all Customers. |
| Steps: | 1. User executes “makeDecision” service request, receives responses.
a. Change Customer for each “makeDecision” request. 2. User executes 4 “captureActivity” views (1 each for the first 4 offers.) 3. User then waits 3 seconds and executes 1 “captureActivity” click (click is for the 1 offer received.) 4. User waits 3 seconds then goes back to step 1. |
| Acceptance Criteria : | Number of Concurrent Users: 120 users (should generate ~100 “makeDecision” responses per second and 600 captureActivity transactions per second.)
Duration: 24 Hours Response Time: 200 milliseconds for Step 1. |