Robustness
The Robustness widget answers a question the optimizer itself cannot: how stable is a candidate design's predicted performance when its input variables carry scatter? It perturbs a chosen design with Normal, Uniform, or standardized Weibull input noise, propagates the perturbations through a surrogate model fitted to the observed trials, and shows the resulting output distribution — without running a single additional real evaluation. The math behind the widget is described in the Method section below.
Workflow
- Select the objective and the surrogate model, then click Fit Surrogate. The surrogate is trained in the background on all completed trials (the same training requirements as the Surrogate Optimizer; at least 10 trials are required). When the study has constraint columns, one constraint surrogate per column is trained alongside automatically.
- Choose the Center — the candidate design to analyze: the Best trial (direction-aware best of the selected objective) or any pinned trial. Pin candidates in the Trial Table or detail modal first to compare specific designs. If a pinned trial disappears (e.g. after switching studies), the widget falls back to the best trial.
- Choose Noise dist: — Normal, Uniform, or Weibull. For Weibull, set the shape k (default 1.5, range [0.2, 20]). Set Noise % — the standard deviation of the input scatter before clamping, as a percentage of each parameter's declared range — and the sample count (256 / 1024 / 4096).
- Optionally enable Model uncertainty to additionally draw from the GP posterior at each sample, folding the surrogate's own (epistemic) uncertainty into the distribution. Ignored by models without predictive variance (Ridge, LightGBM). This affects objective samples only; GP constraint predictive probabilities are computed independently, even when this toggle is off.
The analysis itself runs instantly on any setting change — only the surrogate fit is a long-running step.
Reading the results
- Histogram: the distribution of the predicted objective across all perturbed samples.
- Nominal (grey dashed line): the surrogate prediction at the unperturbed center. Mean (red line): the average over the perturbed samples. Shift = Mean − Nominal does not prove an asymmetric slope or degradation. Nonlinear surrogate responses, boundary clamping (even for a linear objective), and finite Monte Carlo sampling fluctuations can contribute. A positive shift means a worse model-estimated average objective for minimization, better for maximization; a negative shift means the reverse. This is not a guarantee of real performance.
- P5 / P95 (dashed lines): the empirical 5th/95th percentiles; for a minimized objective, P95 is a pessimistic "bad day" estimate.
- Mean ± Std: the spread is the primary robustness measure: between two candidates with similar nominal values, prefer the one with the smaller std.
- P(feasible): shown when the study has constraints: the estimated probability that a perturbed design still satisfies all constraints (Optuna convention, feasible iff ). It averages pointwise feasibility probabilities, using GP predictive uncertainty or a hard indicator for non-GP models; it is not generally a count of passing predicted means.
- Clipped % (amber warning): the fraction of samples clamped in any parameter dimension because the raw value was outside its bounds. Clamping moves those values to the boundary, accumulating probability mass there; this is not a distribution conditional on staying inside the range.
Notes
- The analysis is a model estimate: its validity is bounded by the surrogate's quality. Fit quality can be assessed with the Surrogate Optimizer's validation plot using the same model kind; a poorly fitted surrogate produces confident-looking but meaningless distributions.
- Results are reproducible: with the same trained surrogate and fixed seed, the same center, noise level, distribution (including Weibull shape), sample count, and Model uncertainty setting reproduce the same statistics. Finite Monte Carlo sampling error remains, so candidate comparisons, especially small differences, are not guaranteed exact. Increasing the sample count generally reduces sampling error, but does not remove surrogate error or guarantee improvement at every increase for a fixed seed. Gaussian draws use the Box-Muller transform.
- The intended workflow is comparison: place two Robustness widgets side by side (each keeps its own settings), point them at two pinned candidates, and compare distributions. A slightly worse nominal with a much tighter spread is often the better engineering choice.
- Categorical parameters are excluded (numeric parameters only), matching the Surrogate Optimizer.
- The CSV export writes the raw output samples for external post-processing.
Method
Overview
An optimum found by an optimizer is a deterministic optimum: it assumes the design variables can be realized exactly. In practice, inputs carry uncertainty — manufacturing tolerances, material scatter, fluctuating operating conditions — and a design that sits on a steep ridge of the objective surface may perform far worse than predicted once that scatter is applied. Robustness analysis asks: how does the predicted performance distribute when the inputs are perturbed around a candidate design?
Evaluating this directly would require many additional real evaluations around the candidate. Instead, the analysis reuses the surrogate model already fitted to the observed trials, and propagates input uncertainty through it by Monte Carlo sampling. This makes the analysis essentially free (milliseconds) once a surrogate is trained.
Formula
Let be the fitted surrogate of an objective, the candidate design (e.g. a pinned or best trial), and let the input perturbation be independent per dimension:
where is the declared range of parameter and is the user-chosen noise level (a fraction of each range, e.g. for ±2 %-of-range scatter at 1σ). If a parameter has no declared range (no lower/upper bound on its distribution), falls back to the observed range of that parameter in the trial data. The selected noise family is shared across dimensions, with independent draws of mean zero and variance one before clamping:
- Normal: .
- Uniform: .
- Weibull: for shape (default 1.5, selectable in [0.2, 20]), draw and standardize it:
Here is the gamma function. Thus all families have the same input standard deviation before clamping, but different tails and skew. Clamping (winsorization) moves out-of-range values to the nearest bound rather than rejecting or resampling them; probability mass accumulates at the boundaries. The resulting distribution is not a conditional truncated distribution, and its mean and variance can change.
For Monte Carlo samples the propagated output set is with
The first form propagates only the input (aleatory) uncertainty through the surrogate mean. The second additionally draws from the GP posterior at each sample, folding in the model (epistemic) uncertainty — useful when the candidate lies in a sparsely sampled region where the surrogate itself is unsure.
From the analysis reports the empirical mean , standard deviation , and the 5th / 50th / 95th percentiles. The mean shift (Shift = Mean − Nominal) does not prove an asymmetric slope or degradation. Nonlinear surrogate responses and boundary clamping can shift the mean; clamping can do so even for a linear objective. Finite Monte Carlo sampling fluctuations can also contribute. A positive shift means a worse model-estimated average objective for minimization, better for maximization; a negative shift means the reverse, with no guarantee of real performance.
With constraint surrogates (trained on the values Optuna stores
under the constraints convention, feasible iff ), the feasibility
probability is estimated by averaging pointwise probabilities:
The product assumes independence between constraints. For a GP constraint surrogate, with predictive mean and standard deviation in original constraint units,
where is the standard normal CDF. Non-GP models without predictive variance use instead. Zero or numerically negligible predictive variance also uses this hard indicator; a constant constraint uses its constant value. Thus the reported probability is generally not the fraction of samples whose predicted constraint means pass. Model uncertainty controls objective posterior sampling only, not these GP constraint probabilities.
Characteristics
- The analysis is only as good as the surrogate. Check the surrogate's cross-validation quality first; a poorly fitted model produces confident-looking but meaningless distributions.
- Clamping at the bounds accumulates probability mass at the boundaries and changes the output distribution; the reported spread is not conditional on staying inside the box. Clipped % counts samples clamped in any dimension.
- Independence between input dimensions is a modeling assumption for all three noise families. Correlated tolerances are not modeled; non-Gaussian scatter can be represented by Uniform or standardized Weibull, but arbitrary noise distributions are not supported.
- Independence between constraints is also assumed when multiplying their pointwise feasibility probabilities.
- Gaussian samples are generated with the Box-Muller transform on a seeded RNG. With the same trained surrogate and fixed seed, the same candidate, noise level, distribution (including Weibull shape), sample count, and Model uncertainty setting reproduce the same statistics. Reproducibility does not eliminate finite Monte Carlo sampling error or guarantee exact candidate comparisons, especially for small differences. More samples generally reduce sampling error, not surrogate error; improvement is not guaranteed at every increase for a fixed seed.
- Comparing the distributions of two candidates (e.g. two pinned trials) is the intended workflow: a slightly worse nominal value with a much tighter distribution is often the better engineering choice.
References
- Box, G. E. P., & Muller, M. E. (1958). A Note on the Generation of Random Normal Deviates. Annals of Mathematical Statistics, 29(2), 610–611. https://doi.org/10.1214/aoms/1177706645