Best practices for importing data in Pega Sales Automation
Best practices for importing data in Pega Sales Automation
|Description||Based on the volume of data getting imported, choose the best recommended approach to have an efficient data import.|
|Version as of||8.4|
|Application||Pega Sales Automation|
|Capability/Industry Area||Data import|
Moving data from one system to another can be a demanding task considering the challenges involved in importing huge volumes of data for the first time. Another challenge might be to regularly add and update records like leads, contacts, opportunities, and so on.
To simplify the data import process, Pega Sales Automation includes tools to help you to complete your data import faster and with less effort.
This article describes the recommended approaches for importing data into the Pega Sales Automation application.
Before importing data, identify your selling mode and data model, prepare files for import, and follow the best practices before starting the import. See the following sections:
- Identifying the selling mode and the data model
- Preparing files for import
- Best practices before starting the import
Identifying the selling mode and the data model
The Pega Sales Automation data model consists of a set of standard entities that support the sales process. Some of them are:
- Sales goals
For more information about Pega Sales Automation entities, see Pega Sales Automation entities.
Business objects depend on the selling mode that you select for your implementation. In Pega Sales Automation, you can use the following selling modes:
- Business-to-business (B2B) – if you are selling products and services to other businesses.
- Business-to-consumer (B2C) – if you are selling products and services directly to individual consumers.
- Mixed mode – if you are selling products to both individual consumers and to other businesses.
The following diagram shows the Pega Sales Automation business objects that are used in the B2B selling mode and the relationships between these objects:
The following diagram shows the Pega Sales Automation business objects that are used in the B2C selling mode and the relationships between these objects:
The mixed selling mode includes business objects from both B2B and B2C entity diagrams.
For more information, see Pega Sales Automation data model.
Preparing files for import
To import your data into Pega Sales Automation, use comma-separated values (CSV) files. The columns in the CSV files are mapped to fields in the Pega Sales Automation data model by using templates. Use the Pega Sales Automation sample data templates as examples when creating your CSV files.
Sample Contacts.csv file
Note: If you are importing data from other CRM systems like Salesforce or Siebel, Pega Sales Automation provides default templates for importing that data.
Best practices before starting the import
- Review the data types and how entities relate to other entities and their fields.
- If you find that the default templates do not fulfil your requirements, you can extend these templates. For more information, see Data import scenarios.
- Ensure, that the data types are correct and that date formats are unified across all the CSV files.
- Ensure, that the relationships between the records are correct to provide a proper associations between the entities. You can use VLOOKUP in your Excel files to match the parent record IDs with child records in your export CSV files. For example, check if contacts are linked to accounts that they are related to.
- The operator ID in the CSV file must be exactly the same as the created operator ID. Otherwise, the imported work objects associated with the operator ID, for example, organizations, accounts, leads, contacts, or opportunities will not appear in the application list views.
Choosing the best import method
With Pega Sales Automation you can import data in multiple ways. Considering the volume of the data to be imported, below are the recommended approaches:
- Import data by using the data import wizard for less that 50K records
- Import data by using file listeners for more than 50K records
The best way to import smaller chunks of data (< 50K records)
Use the Data import wizard if the number of records to be imported is less than 50K. Data import wizard uses a single thread and has a user interface guiding you through the import process. Users can upload files manually and monitor the import progress.
Consider a scenario for importing around 50K organization records into Pega Sales Automation. Below is a sample screenshot of the Organizations.csv file to be imported:
Below is the data import mapping screen:
The best way to import large amounts of data (> 50K records)
When you are importing large volumes of data, for example, millions of rows, use the data import File Listener. The data import File Listener uses multithreading for faster throughput, while the Data import wizard uses single-thread processing.
Place all your data import files in the folder which was configured in the File Listener. When the listener starts from Admin Studio, the importing process runs in the background without any manual intervention.
To configure File Listener for data import, see File Listener.
Recommendations for importing around 100 million contacts
Follow the recommended steps for the best performance while using the data import File Listener:
- Before starting the import for all your records, import a few records to start with and fix any issues.
- The size of each CSV file used for importing, should not exceed 1 million records. It is recommended to split the files into 50k bunches.
- Enable configuration templates that are provided by default.
- In the App Studio application settings, set the InitialDataMigration setting to “true”.
- To support high volumes of data, increase the number of nodes to Depending on time to complete import process. For example, if you are importing 100 million contacts within 10 hours, increase the number of nodes to 10.
- Depending on the system configuration, update the number of threads to around 4 per node.
- The recommended batch size for upload is 1000 records.
- To improve performance and disable creating the audit history, use the Add Only mode for the initial data import.
- To ensure the maximum parallel processing, there must be as many input files for the File Listener as there are threads, because each thread processes one file at a time. In Dev Studio, in the Listener properties section, set up the File Listener properties.
- Benchmark the processing time with a smaller size file with 50K records.
- Turn off the creation of history instances for the initial data load.