Arize AI

Share Business Impact

Business Impact and Payoff Curve is a feature in Arize that allows you to define custom formulas that measure your model's performance.

What is Business Impact?

Sometimes, we need other metrics other than traditional statistical measures such as accuracy or recall to define model performance. Business Impact is a way to measure your score model's Payoff at different thresholds (i.e, decision boundary for a score model). The Arize platform allows you to enter custom formulas, mapping model performance to your definition of model performance.
Accuracy Count=TP+TNFPFNBusiness Impact\text{Accuracy Count} = \text{TP} + \text{TN} - \text{FP} - \text{FN} \neq \text{Business Impact}
Imagine your company uses a model to predict whether or not a customer will default on a loan or not. You want to calculate the Business Impact that this model will have on you business. Imagine a scenario where:
  1. 1.
    When the model correctly predicts default (TP), the business makes a $10 profit from admin fees.
  2. 2.
    When the model incorrectly predicts default (FP), the business incurs a $500 loss in order to re-acquire a new customer
  3. 3.
    When the model correctly predicts no-default (TN), the business makes a $1000 profit in the form of customer life-time value.
  4. 4.
    When the model incorrectly predicts no-default (FN), we incur a $300 loss from the contract.
Business Impact=10TP+1000TN500FP300FN\text{Business Impact} = 10 * \text{TP} + 1000 * \text{TN} - 500 * \text{FP} - 300 * \text{FN}

How Can Arize Help?

In the above example, correctly predicting Profitable Loan (TN) offers the most business value, so we should optimize for that metric in production. However, it is often difficult to measure Business Impact production when we consider typical problems such as actual label distributions shifts (i.e, increase in fraud), and prediction drift (i.e, model performance degradation, feature drift)
Arize offers a solution to monitor your model in a way that fits your business decisions, calculating and generating a live, up-to-date production Payoff Curve, showing the most profitable/impactful probabilistic decision boundary for any models that predicts probability scores.

How do I See My Payoff Curve?

For any Score Model, by entering your custom business formula, you can generate a Payoff Curve.
Generating Custom Payoff Curve

How can I compare Model Performances?

By logging new versions of models to validation using arize.log_validation_records, we can validate and visualize their Business Impact before deploying them in production!
Validate Model Impact before Production

How do I share my insights with other team members?

When you have properly constructed a formula that describes the business impact of your model, name the formula something meaningful (e.x. payoff, income, loss) and click save formula. You can now share this page with your broader team so that they can view and track the payoff curve as well.

Example Notebook

Work through one of our quick-start notebooks to experiment with Business Impact in under 15 minutes!
Questions? Email us at [email protected] or Slack us in the #arize-support channel