View source for Leveraging self-learning models to ensure optimization in new situation

== Review performance ==
Once you have ensured that adaptive models are being applied as required by following the previous steps, you can now review the current models, their performance and ways to optimize them.

=== Prediction Studio ===
Prediction Studio is an authoring environment in which you can control the life cycle of AI and machine-learning models (such as model building, monitoring, and update).

=== Predictions ===
When you open Prediction Studio it will open at the '''Predictions''' work area. Here you can manage, review, and run the predictions that are used in the Next-Best-Action Designer Arbitration. Here you will see all the active and inactive predictions. Here you can review and sort the predictions by name, subject, outcome and performance. You can look for a particular prediction, e.g. ''Predict Outbound Email Propensity,'' or you can sort by performance to identify the predictions that are under-performing with a lower than average AUC score.

[[File:Outbound Email Predictions.jpg|alt=Outbound Email Predictions|frameless|600x600px]]

=== Adaptive Models ===
Adaptive models work by recording all customer responses (both positive and negative) and correlating them to different customer details (for example, age, gender, region, and so on). For example, if ten people under 35 years of age accept a particular phone offer, the predicted likelihood that more people under 35 years of age will buy the same phone increases. The likelihood can also go down if a negative response is recorded, from this group. Over time, reliable correlations emerge.

When you click on a particular prediction in the Predictions work area you will be presented with the Models Overview for the underlying model that is driving that prediction. Here you can review the predictive performance and success rate of the individual model. For more information, see [https://community.pega.com/knowledgebase/articles/decision-management/85/adaptive-models-monitoring Adaptive models monitoring]

[[File:Models Performance Overview.png|alt=Models Performance Overview|frameless|600x600px]]

In this chart there are two factors to consider:

* The location of the bubble on the chart
** Weak Performance / High Success (top left)
*** These actions are not considered relevant to the customer, but the action would seem to be attractive or accepted often.
** Weak Performance / Low Success (bottom left)
*** These actions are currently not adding any value and should be reviewed as a priority.
** Strong Performance / Low Success (bottom right)
*** These actions are already relevant to the customer but for some reason they seem to be unattractive to them.
** Strong Performance / High Success (top right)
*** These actions are the top performers and our aspirational goal for all actions.
* The size of the bubble on the chart
** High number of responses
*** These actions being presented often
** Low number of responses
*** These actions are rarely being presented

Tasks for users of adaptive models based on the above:

# Identify technical problems
#* For example, are there propositions that are not retrieving any positives (the success rate will be zero, to the bottom of the plot)?
# Identify propositions for which the model is not predictive (weak performance, to the left side in the graph).
#* The performance may be increased if new potential predictors are added. Can we add potential predictors that have not yet been considered? This could help, however there is no guarantee as an outcome could be inherently difficult to predict.
# Identify propositions that are not proposed enough (low number of responses, small circle size in the graph)
#* Are the eligibility rules in the decision strategy to restrictive? Should we relax the exclusion rules?
#* Should we increase the priority of this proposition in the decision strategy?
# Identify propositions that are proposed so much that they be dominating other propositions (a high number of responses, a large circle size in the graph).
#* Is this desired from the business perspective? If not, the prioritization in the decision strategy should be adjusted.
# Identify propositions with a low success rate (low success rate, to the bottom of the plot).
#* Should the business consider dismissing the proposition?
#* If the model performance is weak (to the left of the plot) then there may be room for improvement if the model can be improved, because proposing it to more relevant customers would increase the success rate. 
#* If the model performance is already strong (to the right of the plot) then the relevance to the customer is already high, but the proposition would seem to be inherently unattractive.
# You can then click on a particular model report to get to a details report for that model at the action level. The Predictors view provides a detailed report of all predictors.
#* Look at the highest performing predictor in the list of active predictors.
#* Look at the inactive predictors.
[[File:Predictors Report.png|alt=Predictors Report|frameless|600x600px]]