Leveraging self-learning models to ensure optimization in new situation
Leveraging self-learning models to ensure optimization in new situation
This article is part of a series about using Pega Customer Decision Hub to communicate with customers during emergency events. For an overview of the process, see Rapid Response to Emergencies. For the previous article in the series, see Adjusting Next-Best-Action catalogue to support changed business priorities.
One of the key features of Pega is that AI is central to our NBA philosophy, specifically Pega's adaptive models which use self-learning statistical models to determine customer propensity.
There are two aspects of Pega's adaptive models that are important to understand:
Automation
Adaptive Decision Manager is a closed-loop system that automates the model creation, deployment, and monitoring process for each action.
Adaptive Decision Manager automatically manages a large number of models without human intervention.
Self-learning algorithms
Predict the likelihood of accepting a positive outcome in real time.
Establishing customer preferences incrementally while collecting historical data.
Model learning from each interaction, system keeps sufficient statistics only.
Adaptive tracking of model performance.
Review current settings
The first two steps are to review your current usage of adaptive models and ensure that it is enabled for each action and in the overall arbitration settings.
- Action-level adaptive
- For each action that you wish to use adaptive as part of the P*V*L formula, ensure that you have Apply AI to calculate action propensity enabled.
- Arbitration-level adaptive
- Go to the Arbitration section of Next-Best-Action Designer.
- Ensure that Apply propensity calculated for every treatment is enabled.
Review performance
Once you have ensured that adaptive models are being applied as required by following the previous steps, you can now review the current models, their performance and ways to optimize them.
Prediction Studio
Prediction Studio is an authoring environment in which you can control the life cycle of AI and machine-learning models (such as model building, monitoring, and update).
Predictions
When you open Prediction Studio it will open at the Predictions work area. Here you can manage, review, and run the predictions that are used in the Next-Best-Action Designer Arbitration. Here you will see all the active and inactive predictions. Here you can review and sort the predictions by name, subject, outcome and performance. You can look for a particular prediction, e.g. Predict Outbound Email Propensity, or you can sort by performance to identify the predictions that are under-performing with a lower than average AUC score.
Adaptive Models
Adaptive models work by recording all customer responses (both positive and negative) and correlating them to different customer details (for example, age, gender, region, and so on). For example, if ten people under 35 years of age accept a particular phone offer, the predicted likelihood that more people under 35 years of age will buy the same phone increases. The likelihood can also go down if a negative response is recorded, from this group. Over time, reliable correlations emerge.
When you click on a particular prediction in the Predictions work area you will be presented with the Models Overview for the underlying model that is driving that prediction. Here you can review the predictive performance and success rate of the individual model. For more information, see Adaptive models monitoring
In this chart there are two factors to consider:
- The location of the bubble on the chart
- Weak Performance / High Success (top left)
- These actions are not considered relevant to the customer, but the action would seem to be attractive or accepted often.
- Weak Performance / Low Success (bottom left)
- These actions are currently not adding any value and should be reviewed as a priority.
- Strong Performance / Low Success (bottom right)
- These actions are already relevant to the customer but for some reason they seem to be unattractive to them.
- Strong Performance / High Success (top right)
- These actions are the top performers and our aspirational goal for all actions.
- Weak Performance / High Success (top left)
- The size of the bubble on the chart
- High number of responses
- These actions being presented often
- Low number of responses
- These actions are rarely being presented
- High number of responses
Tasks for users of adaptive models based on the above:
- Identify technical problems
- For example, are there propositions that are not retrieving any positives (the success rate will be zero, to the bottom of the plot)?
- Identify propositions for which the model is not predictive (weak performance, to the left side in the graph).
- The performance may be increased if new potential predictors are added. Can we add potential predictors that have not yet been considered? This could help, however there is no guarantee as an outcome could be inherently difficult to predict.
- Identify propositions that are not proposed enough (low number of responses, small circle size in the graph)
- Are the eligibility rules in the decision strategy to restrictive? Should we relax the exclusion rules?
- Should we increase the priority of this proposition in the decision strategy?
- Identify propositions that are proposed so much that they be dominating other propositions (a high number of responses, a large circle size in the graph).
- Is this desired from the business perspective? If not, the prioritization in the decision strategy should be adjusted.
- Identify propositions with a low success rate (low success rate, to the bottom of the plot).
- Should the business consider dismissing the proposition?
- If the model performance is weak (to the left of the plot) then there may be room for improvement if the model can be improved, because proposing it to more relevant customers would increase the success rate.
- If the model performance is already strong (to the right of the plot) then the relevance to the customer is already high, but the proposition would seem to be inherently unattractive.
- You can then click on a particular model report to get to a details report for that model at the action level. The Predictors view provides a detailed report of all predictors.
- Look at the highest performing predictor in the list of active predictors.
- Look at the inactive predictors.
