MDI

Overview

MDI (Mean Decrease Impurity) computes parameter importance from the impurity reduction at each split during Random Forest training. Every time a parameter is used for a split, the weighted reduction in MSE is accumulated across all trees.

Scores are normalized to sum to 1.

Comparison with RF-ANOVA

Aspect	MDI	RF-ANOVA
Measured	During training	After training (variance decomposition over leaf boxes, fANOVA)
Bias	Overestimates high-cardinality features	Affected by correlated features
Interpretation	"How useful was it during learning?"	"Fraction of the objective's variance explained by this parameter's main effect"

Formula

Impurity decrease at node $t$:

$$\Delta I(t) = \operatorname{MSE}(y_t) - \left[ \frac{n_L}{n_t} \cdot \operatorname{MSE}(y_L) + \frac{n_R}{n_t} \cdot \operatorname{MSE}(y_R) \right]$$

MDI for parameter $j$ in tree $b$:

$$\operatorname{MDI}$$b(j) = \sum{\text{splits on } j} \frac{n_t}{n_{\text{root}}} \cdot \Delta I(t)

Final normalized score:

$$\operatorname{MDI}(j) = \frac{1}{T} \sum_{b} \operatorname{MDI}_b(j)$$

$$\operatorname{MDI}$${\text{norm}}(j) = \frac{\operatorname{MDI}(j)}{\sum{j'} \operatorname{MDI}(j')}

Hyperparameters

Parameter	Value
Trees	64
Max depth	10
Min leaf samples	2
Random seed	42
Max rows	1,000

R² Interpretation

R² is computed on holdout data:

R²	Meaning
≥ 0.8 (green)	Good fit. Scores are reliable.
0.5–0.8 (yellow)	Moderate. Use as reference.
< 0.5 (red)	Poor fit. Scores are less reliable.

Known Bias

MDI tends to overestimate high-cardinality features (features with many distinct values have more split candidates). When this is a concern, use RF-ANOVA instead.

When to Use

• When you want fast importance with zero additional cost after training.
• For a quick sanity check alongside RF-ANOVA or Permutation.

References

• Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
• Ke, G. et al. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NeurIPS 30. https://proceedings.neurips.cc/paper_files/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html