Sensitivity Analysis
Choosing a method
- Spearman: fast screening of monotonic relationships for numerical parameters.
- Ridge: fast numerical linear contributions; its R² evaluates a numerical-only model.
- MDI: split-based feature importance; fast, but can favor high-cardinality features.
- RF-ANOVA: main-effect variance decomposition over Random Forest leaf boxes.
- Permutation: holdout MSE increase after shuffling; affected by correlated features.
- SHAP: prediction attribution; heavier but useful for explanation.
- Sobol: first-order and total-effect sensitivity, depending on surrogate quality and sampling.
- ARD: relevance from GP length scales, not a variance decomposition.
- PDP: visualize response shape; strong correlations can make it extrapolative.
Start with Spearman/Ridge for numerical screening, then compare methods, inspect PDP, and use Sobol when interactions matter. Scores describe association or predictive contribution, not causality.
Nominal categories and output
Optuna categorical choices are nominal even if their labels look numeric. Spearman and Ridge exclude them and list Unsupported (categorical) separately in the Importance Chart, Sensitivity Heatmap and CSV, never as a scored or ranked zero. All-categorical inputs show No supported numerical parameters and no Ridge R². Mixed Ridge R² is explicitly numerical-only. Spearman and Ridge support Optuna Int/Float parameters. Spearman excludes non-finite parameter/objective pairs before ranking; Ridge excludes rows with any non-finite included numerical feature or objective before sampling and splitting.
Reports use numerical-only |Spearman| importance against the first objective.
HTML/Markdown list unsupported categories without scores; JSON carries
importance.unsupported_categorical separately from scores. The MCP report
uses these same outputs, and MCP study summaries expose
unsupported_categorical_importance. Numerical unavailable correlations are not
categorical unsupported.
Categorical encoding limitations
Tree methods, Sobol and GP ARD encode categorical choices as integer IDs
0, 1, 2, ... in first-appearance order:
- MDI, RF-ANOVA, SHAP and Permutation train trees on numeric codes, not native nominal features. Split thresholds impose an arbitrary ordering. RF-ANOVA additionally integrates numeric leaf boxes.
- Sobol fits its quadratic Ridge surrogate on these codes and samples continuous ranges. Code distances and intermediate points need not correspond to meaningful categories.
- GP ARD importance uses these code distances in its kernel, so length-scale relevance for categories depends on an artificial geometry.
These categorical results are not guaranteed invariant to code assignment or trial order. A bijective relabeling that preserves equality and first appearance keeps the codes, but does not make the assumed order or distances meaningful. They are not nominal-safe substitutes for unsupported Spearman/Ridge. Interpret them cautiously: these methods use neither one-hot encoding nor a category-specific distance metric.