Currently machine learning not only act actively in research but also in production. But when serving to clients, a black-box machine learning model might not works:

1. Reasonable: Is this decision making procedure reasonable? Our end-user not only want correct solution but also reasonable solution
2. Importance: How can model help our work? What features is the more important so I can make improvement?
3. Impact: For each feature, how they affect model? Positively? Negatively? Like the coefficient in linear regression analysis
4. Case Study: Can I investigate some specific cases like extreme cases, error cases?
5. Corelation: Is there any correlation between features?

In additional to prediction given by model, answering these question could help our end-users understand their dataset, and their problem more clearly. (Machine Learning could performs like a way of EDA)

And we can see from above that machine learning model intepretation is usually restricted into 2 domain:

1. Global: or say dataset perspective → Importance of features
2. Local: or say sample perspective → How model make decision for each sample

And we usually cares about:

1. Feature Importance
2. Prediction procedure / Feature Dependency
3. Feature Corelation

SHAP stands for SHaply Additive exPlanation. The core is shaply value.

Shaply value provides an additve method to calculate the contribution of each feature to model. Mathematically, $y_i = \sum \phi_{i, j}$ where i is index of sample and j is index of feature

A very simple example would be use A and B to predict y. ($X = \{A, B\} \rightarrow Y$)

• If only use A could give prediction 0.3 and with B could give prediction 0.7, shaply value of A and B would be 0.3 and 0.4, respectively.
• If only use B could give prediction 0.5 and with A could give prediction 0.7, shaply value of A and B would be 0.2 and 0.5, respectively

Hence the shaply value of this sample, A and B would be weighted sum so 0.25 and 0.45, respectively. Hence, we know feature B has a higher contribution to model (Compare the absolute value) compared to A, in this sample.