Tunny Icon
TunnyDocs

The next-gen Grasshopper optimization tool.

Sobol

Overview

Sobol sensitivity indices quantify parameter importance through variance decomposition of the objective function. They work for linear and nonlinear functions alike and can measure interaction effects between parameters. Values are in [0, 1].

In v0.1, the reported values estimate sensitivity of a fitted quadratic Ridge surrogate under independent uniform inputs, not of the empirical joint Trial distribution or the constrained design domain. The input measure and surrogate quality are essential to their interpretation.

Tunny Dashboard provides two indices:

Index Symbol Meaning
First-order SiS_i Fraction of variance explained by xix_i alone
Total-effect STiST_i xix_i alone plus all interactions involving xix_i

Theory

For independent inputs and positive output variance, the model output's variance is decomposed as:

Var⁡(Y)=∑iVi+∑i<jVij+⋯+V1..p\operatorname{Var}(Y) = \sum_i V_i + \sum_{i<j} V_{ij} + \cdots + V_{1..p}

First-order index:

Si=ViVar⁡(Y)S_i = \frac{V_i}{\operatorname{Var}(Y)}

Total-effect index:

STi=1−Var⁡(E[Y∣X∼i])Var⁡(Y)ST_i = 1 - \frac{\operatorname{Var}(E[Y \mid X_{\sim i}])}{\operatorname{Var}(Y)}

where X∼iX_{\sim i} means "all parameters except xix_i."

The theoretical indices satisfy STi≥SiST_i \geq S_i. A large gap (STi−Si)(ST_i - S_i) indicates interactions in the evaluated model under the chosen input measure.

Note on finite-sample estimates: the Saltelli first-order estimator and the Jansen total-effect estimator are computed independently, so raw estimates can violate ST^i≥S^i\hat{ST}_i \geq \hat{S}_i in finite samples. Tunny Dashboard enforces the property after estimation by taking ST^i←max⁡(ST^i,S^i)\hat{ST}_i \leftarrow \max(\hat{ST}_i, \hat{S}_i) before clamping both values to [0,1][0, 1].

How Tunny Dashboard Computes This

A quadratic Ridge surrogate is fitted using only rows where all parameter and objective values are finite. At least two retained rows, one parameter, and one objective are required. Parameter ranges are computed from those retained rows.

For each nondegenerate parameter range, the input measure is Xj∼U(observedminj,observedmaxj)X_j \sim U(\mathrm{observed_min}_j, \mathrm{observed_max}_j), independently across parameters. A and B contain separate independent uniform draws; each ABiAB_i replaces column ii of A with column ii of B. These are observed ranges, not declared search ranges, and sampling does not reproduce empirical Trial frequencies or dependencies.

If ∣max⁡j−min⁡j∣<f64::EPSILON|\max_j - \min_j| < \texttt{f64::EPSILON}, the range is expanded to [min⁡j−1,max⁡j+1][\min_j - 1, \max_j + 1], rather than keeping the parameter fixed. The range helper also has an empty-column fallback of [0,1][0,1], but empty columns are not normal accepted input: at least two valid rows are required.

All A, B, and ABiAB_i outputs are surrogate predictions, with no simulation rerun, domain feasibility validation, or snapping to discrete values. First-order indices use Saltelli (2010); total-effect indices use Jansen (1999). Near-zero surrogate output variance ($\mathrm{Var}_Y < \texttt{f64::EPSILON}$) returns zeros for both indices; otherwise the max-then-clamp post-processing above is applied.

Parameters

Setting Value
Saltelli sample count NN 1,024
Ridge regularization strength α\alpha 1.0
Random number generator ChaCha8 (deterministic, seed 0xDEAD_BEEF_1234_5678)

Interpreting Results

Interpret these patterns as effects in the fitted surrogate under the independent uniform input measure, not as guarantees about the actual objective function:

  • SiS_i high, STi≈SiST_i \approx S_i → parameter has a large independent effect, few interactions.
  • SiS_i small, STiST_i high → parameter mainly acts through interactions with others.
  • Both near zero → parameter has little influence on surrogate output variance under this measure.

Strengths and Limitations

Strengths:

  • Captures linear, quadratic, and pairwise interaction effects in the surrogate.
  • Values in [0, 1] allow easy comparison across parameters.
  • ST−SST - S indicates interactions in the surrogate under this measure.

Limitations:

  • Results depend on surrogate model quality (quadratic Ridge).
  • Accuracy decreases when the true function is highly nonlinear.
  • Correlations and conditional parameter activation in the Trials are not preserved. Independent sampling over the rectangular product of ranges can produce unobserved combinations or points outside constrained/nonrectangular domains. Predictions there can be extrapolations.
  • String categories are label-encoded into IDs 0.0, 1.0, … in order of first appearance; numeric category labels are treated as numeric values. Continuous uniform sampling can draw fractional IDs that are not real categories. The surrogate imposes artificial ordering and distances on these IDs, so this is not categorical-aware sensitivity analysis and results can depend on encoding.

When to Use

  • When interactions between parameters are important.
  • When global sensitivity of a quadratic Ridge surrogate under independent uniform inputs is an appropriate approximation.
  • After Spearman / Ridge screening identifies top candidates.

References

  • Saltelli, A. et al. (2010). Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Computer Physics Communications, 181(2), 259–270. https://doi.org/10.1016/j.cpc.2009.09.018
  • Jansen, M. J. W. (1999). Analysis of variance designs for model output. Computer Physics Communications, 117(1), 35–43. https://doi.org/10.1016/S0010-4655(98)00154-4
  • Sobol, I. M. (1993). Sensitivity estimates for nonlinear mathematical models. Mathematical Modelling and Computational Experiments, 1(4), 407–414.