Tunny Icon
TunnyDocs

The next-gen Grasshopper optimization tool.

Sensitivity Analysis

Choosing a method

  • Spearman: fast screening of monotonic relationships for numerical parameters.
  • Ridge: fast numerical linear contributions; its R² evaluates a numerical-only model.
  • MDI: split-based feature importance; fast, but can favor high-cardinality features.
  • RF-ANOVA: main-effect variance decomposition over Random Forest leaf boxes.
  • Permutation: holdout MSE increase after shuffling; affected by correlated features.
  • SHAP: prediction attribution; heavier but useful for explanation.
  • Sobol: first-order and total-effect sensitivity, depending on surrogate quality and sampling.
  • ARD: relevance from GP length scales, not a variance decomposition.
  • PDP: visualize response shape; strong correlations can make it extrapolative.

Start with Spearman/Ridge for numerical screening, then compare methods, inspect PDP, and use Sobol when interactions matter. Scores describe association or predictive contribution, not causality.

Nominal categories and output

Optuna categorical choices are nominal even if their labels look numeric. Spearman and Ridge exclude them and list Unsupported (categorical) separately in the Importance Chart, Sensitivity Heatmap and CSV, never as a scored or ranked zero. All-categorical inputs show No supported numerical parameters and no Ridge R². Mixed Ridge R² is explicitly numerical-only. Spearman and Ridge support Optuna Int/Float parameters. Spearman excludes non-finite parameter/objective pairs before ranking; Ridge excludes rows with any non-finite included numerical feature or objective before sampling and splitting.

Reports use numerical-only |Spearman| importance against the first objective. HTML/Markdown list unsupported categories without scores; JSON carries importance.unsupported_categorical separately from scores. The MCP report uses these same outputs, and MCP study summaries expose unsupported_categorical_importance. Numerical unavailable correlations are not categorical unsupported.

Categorical encoding limitations

Tree methods, Sobol and GP ARD encode categorical choices as integer IDs 0, 1, 2, ... in first-appearance order:

  • MDI, RF-ANOVA, SHAP and Permutation train trees on numeric codes, not native nominal features. Split thresholds impose an arbitrary ordering. RF-ANOVA additionally integrates numeric leaf boxes.
  • Sobol fits its quadratic Ridge surrogate on these codes and samples continuous ranges. Code distances and intermediate points need not correspond to meaningful categories.
  • GP ARD importance uses these code distances in its kernel, so length-scale relevance for categories depends on an artificial geometry.

These categorical results are not guaranteed invariant to code assignment or trial order. A bijective relabeling that preserves equality and first appearance keeps the codes, but does not make the assumed order or distances meaningful. They are not nominal-safe substitutes for unsupported Spearman/Ridge. Interpret them cautiously: these methods use neither one-hot encoding nor a category-specific distance metric.