Robustness

The Robustness widget answers a question the optimizer itself cannot: how stable is a candidate design's predicted performance when its input variables carry scatter? It perturbs a chosen design with Gaussian input noise, propagates the perturbations through a surrogate model fitted to the observed trials, and shows the resulting output distribution — without running a single additional real evaluation. The math behind the widget is described in the Method section below.

Workflow

1. Select the objective and the surrogate model, then click Fit Surrogate. The surrogate is trained in the background on all completed trials (the same training requirements as the Surrogate Optimizer; at least 10 trials are required). When the study has constraint columns, one constraint surrogate per column is trained alongside automatically.
2. Choose the Center — the candidate design to analyze: the Best trial (direction-aware best of the selected objective) or any pinned trial. Pin candidates in the Trial Table or detail modal first to compare specific designs. If a pinned trial disappears (e.g. after switching studies), the widget falls back to the best trial.
3. Set the Noise % — the 1σ of the Gaussian input scatter, as a percentage of each parameter's declared range — and the sample count (256 / 1024 / 4096).
4. Optionally enable Model uncertainty to additionally draw from the GP posterior at each sample, folding the surrogate's own (epistemic) uncertainty into the distribution. Ignored by models without predictive variance (Ridge, LightGBM).

The analysis itself runs instantly on any setting change — only the surrogate fit is a long-running step.

Reading the results

• Histogram: the distribution of the predicted objective across all perturbed samples.
• Nominal (grey dashed line): the surrogate prediction at the unperturbed center. Mean (red line): the average over the perturbed samples. A gap between them (also printed as Shift) means the design sits on an asymmetric slope: scatter degrades it on average even though the nominal value looks fine.
• P5 / P95 (dashed lines): the empirical 5th/95th percentiles; for a minimized objective, P95 is a pessimistic "bad day" estimate.
• Mean ± Std: the spread is the primary robustness measure: between two candidates with similar nominal values, prefer the one with the smaller std.
• P(feasible): shown when the study has constraints: the estimated probability that a perturbed design still satisfies all constraints (Optuna convention, feasible iff $c \le 0$).
• Clipped % (amber warning): the fraction of samples that hit a declared parameter bound. When this is large, the center is near a bound and the reported distribution is truncated — interpret it as conditional on staying inside the range.

Notes

• The analysis is a model estimate: its validity is bounded by the surrogate's quality. Fit quality can be assessed with the Surrogate Optimizer's validation plot using the same model kind; a poorly fitted surrogate produces confident-looking but meaningless distributions.
• Results are deterministic: a fixed seed means the same center, noise, and sample count always reproduce the same statistics — comparisons between candidates are never polluted by sampling luck. Gaussian draws use the Box-Muller transform.
• The intended workflow is comparison: place two Robustness widgets side by side (each keeps its own settings), point them at two pinned candidates, and compare distributions. A slightly worse nominal with a much tighter spread is often the better engineering choice.
• Categorical parameters are excluded (numeric parameters only), matching the Surrogate Optimizer.
• The CSV export writes the raw output samples for external post-processing.

Method

Overview

An optimum found by an optimizer is a deterministic optimum: it assumes the design variables can be realized exactly. In practice, inputs carry uncertainty — manufacturing tolerances, material scatter, fluctuating operating conditions — and a design that sits on a steep ridge of the objective surface may perform far worse than predicted once that scatter is applied. Robustness analysis asks: how does the predicted performance distribute when the inputs are perturbed around a candidate design?

Evaluating this directly would require many additional real evaluations around the candidate. Instead, the analysis reuses the GP surrogate already fitted to the observed trials, and propagates input uncertainty through it by Monte Carlo sampling. This makes the analysis essentially free (milliseconds) once a surrogate is trained.

Formula

Let $\hat{f}$ be the fitted surrogate of an objective, $\mathbf{x}_0$ the candidate design (e.g. a pinned or best trial), and let the input perturbation be Gaussian and independent per dimension:

$$\mathbf{x}_i = \mathbf{x}_0 + \boldsymbol{\varepsilon}$$i, \qquad \varepsilon{ij} \sim \mathcal{N}!\left(0,; \sigma_j^2\right), \qquad \sigma_j = \delta \cdot (u_j - l_j)

where $[l_j, u_j]$ is the declared range of parameter $j$ and $\delta$ is the user-chosen noise level (a fraction of each range, e.g. $\delta = 0.02$ for ±2 %-of-range scatter at 1σ). If a parameter has no declared range (no lower/upper bound on its distribution), $[l_j, u_j]$ falls back to the observed range of that parameter in the trial data. Samples are clipped to these bounds, mirroring the physical constraint that the variable cannot leave its feasible box.

For $N$ Monte Carlo samples the propagated output set is ${y_i}$ with

$$y_i = \hat{f}(\mathbf{x}_i) \quad\text{(aleatory only)}, \qquad y_i \sim \mathcal{N}!\left(\hat{f}(\mathbf{x}_i),; \hat{s}^2(\mathbf{x}_i)\right) \quad\text{(aleatory + epistemic)}$$

The first form propagates only the input (aleatory) uncertainty through the surrogate mean. The second additionally draws from the GP posterior at each sample, folding in the model (epistemic) uncertainty $\hat{s}^2(\mathbf{x})$ — useful when the candidate lies in a sparsely sampled region where the surrogate itself is unsure.

From ${y_i}$ the analysis reports the empirical mean $\bar{y}$, standard deviation $s_y$, and the 5th / 50th / 95th percentiles. The mean shift $\bar{y} - \hat{f}(\mathbf{x}_0)$ exposes asymmetry: a design on a one-sided slope degrades on average even though the nominal prediction looks good.

With constraint surrogates $\hat{c}_k$ (trained on the values Optuna stores under the constraints convention, feasible iff $c_k \le 0$), the feasibility rate is

$$P_\text{feas} \approx \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}!\left[\hat{c}_k(\mathbf{x}_i) \le 0 ;; \forall k\right]$$

i.e. the fraction of perturbed designs that remain feasible — a Monte Carlo counterpart of the reliability ("six-sigma") measures used in robust design optimization.

Characteristics

• The analysis is only as good as the surrogate. Check the surrogate's cross-validation quality first; a poorly fitted model produces confident-looking but meaningless distributions.
• Clipping at the declared bounds truncates the noise distribution for candidates near a bound; the reported spread is then conditional on staying inside the box.
• Independent Gaussian noise per dimension is an assumption. Correlated tolerances or non-Gaussian scatter are not modeled.
• Gaussian samples are generated with the Box-Muller transform on a seeded RNG. Deterministic seeding makes the sample set reproducible: the same candidate, noise level, and sample count always produce the same statistics.
• Comparing the distributions of two candidates (e.g. two pinned trials) is the intended workflow: a slightly worse nominal value with a much tighter distribution is often the better engineering choice.

References

• Box, G. E. P., & Muller, M. E. (1958). A Note on the Generation of Random Normal Deviates. Annals of Mathematical Statistics, 29(2), 610–611. https://doi.org/10.1214/aoms/1177706645