Surrogate Optimizer

The Surrogate Optimizer fits a response surface (surrogate model) to the sampled trials and then optimizes on that surface to estimate the optimal parameter values — without running any additional trials.

Workflow

1. Select the objective, the surrogate model, and the optimization method.
2. Click Run Optimization. The surrogate is trained on all completed trials and the optimizer searches within each parameter's declared range from the log (falling back to the sampled parameter ranges when the declared range is unavailable).
3. The estimated optimum is shown in a table styled like the Trial Table widget, with the predicted objective value. In multi-objective mode the predicted Pareto front is shown as a scatter plot inside the widget, and observed trials can be overlaid (see "Results display" below).

Results display

• Optimal variable combination: the estimated optimum (in multi-objective mode, each predicted Pareto-front point) is shown in a table styled like the Trial Table widget, with striped rows, resizable columns, and horizontal scroll, listing parameter values alongside the predicted objective value(s).
• Predicted Pareto front scatter (multi-objective): the predicted front is displayed as an objective-space scatter plot inside the widget. For two objectives it is a fixed 2D (obj0 × obj1) plot; for three or more a 3D view toggle switches between 2D and 3D, with X/Y (and Z in 3D) axis selectors. The front is also overlaid in gold on the Pareto Scatter widgets (2D and 3D).
• Observed-point overlay: existing trials can be overlaid on the scatter so you can compare the predicted front against real results. Points are classified using the same convention as the other scatter charts: observed Pareto front (red) / dominated (blue) / infeasible (grey), each with an individual on/off toggle. The axis scale encompasses both front and observed points.
• Optimization-history plot (single-objective): the objective-value history across all trials, with the predicted optimum highlighted via a horizontal line and a star marker.

The response-surface (heatmap) slice display has been removed. Surrogate fit can be assessed via the validation plot, and optimization improvement via the predicted front or history plot.

Surrogate models

• GP-FITC: Gaussian process regression (Kriging) with ARD Matérn 5/2 kernel backed by egobox-gp. FITC sparse approximation with M = min(N, 100) inducing points; trains on up to 2000 trials (larger sets are subsampled, see below). Provides predictive uncertainty ( $\pm 1.96\sigma$). Default GP choice.
• GP-VFE: Same architecture as GP-FITC but uses the Variational Free Energy bound instead of FITC likelihood. Produces a slightly smoother, more conservative fit; recommended when GP-FITC surface looks overfit or spiky.
• GP-MOE: Mixture-of-experts GP via egobox-moe. Clusters the input space with a Gaussian Mixture Model and trains one FITC expert per cluster (up to 3, selected by cross-validation on ≤ 500 points). Best for discontinuous or regime-switching objectives. If training fails, an error is reported rather than silently falling back.
• Ridge: Linear ridge regression. Fast baseline; the surface is a plane, so the optimum always lies on the boundary of the declared range (search range).
• LightGBM: Gradient-boosted-tree-derived model run in LightGBM's Random Forest mode. Handles non-smooth, discontinuous response surfaces well, but predictions are piecewise-constant and extrapolate flat beyond the training range; no predictive uncertainty is provided.

Automatic model selection

Choosing Auto (cross-validated) in the model selector lets the tool pick the surrogate for you. Each candidate is cross-validated and the one with the highest mean CV R² is trained.

• Candidates: Ridge, GP-FITC, GP-VFE, LightGBM. GP-MOE is excluded from Auto — its cross-validated cluster search is expensive and it degenerates to a single GP on smooth or linear data, giving a poor cost/benefit ratio. Pick GP-MOE manually when you know the response is discontinuous or multi-modal.
• Criterion: mean k-fold CV R² (the same metric shown in the validation panel), which rewards generalization rather than in-sample fit.
• Tie-break: candidates whose CV R² is within 1e-3 of the best are treated as tied, and the earlier (simpler, cheaper) candidate in the order above is chosen. On perfectly linear data, where both Ridge and a GP reach R² ≈ 1, this keeps the simpler Ridge.

After an Auto fit, the widget shows which model was chosen and the ranked candidate CV R² scores. The chosen concrete model is used for everything downstream (acquisition suggestions, constraints).

Validation plot (predicted vs actual)

After Fit & Validate, an out-of-fold predicted-vs-actual scatter is shown. The closer the points cluster along the $y = x$ diagonal, the better the surrogate generalizes to unseen data.

For multi-objective fits, Pareto-front (non-dominated, rank 0) trials are highlighted in red (all others in blue), and the out-of-fold $R^2$ / RMSE computed on front points only is shown above the plot. Even when the overall fit looks good, the front region that actually drives optimization can still be systematically off; this breakdown lets you judge approximation accuracy specifically near the front. Single-objective fits have no front concept, so all points remain blue.

How to choose the optimization method

The optimization methods the Surrogate Optimizer runs on the fitted surface (surrogate model) fall into two groups: single-objective optimization (estimating the optimum) and next-trial suggestion (recommending parameters for real evaluation). All of them work with non-GP surrogates too, but the acquisition functions (EI / LCB / EHVI) need predictive uncertainty and therefore only support GP models (GP-FITC / GP-VFE / GP-MOE).

Surface optimization (estimating the optimum)

Method	Derivative info	Search unit	Multi-obj	Good for	When to pick
Multi-start L-BFGS	numerical gradient	single point (multi-start)	no	smooth, unimodal	Surface is smooth and strongly unimodal; converges in few evaluations
CMA-ES	none (distribution adaptation)	single adaptive distribution	no	mild multimodal, correlated, ill-scaled	Strong default for continuous problems; robust to distortion and mild multimodality
NSGA-II	none	population (n individuals)	yes	strong multimodal, discontinuous	Strongly multimodal or discontinuous surfaces, and automatic predicted-Pareto-front computation in multi-objective mode
Random Search	none	random points	no	any	Robust baseline that always works

• If the surface is smooth and strongly unimodal, Multi-start L-BFGS converges in few evaluations.
• If there is mild multimodality or correlation / scale distortion, CMA-ES adapts its distribution and is robust.
• For strongly multimodal or discontinuous surfaces (including piecewise-constant models like LightGBM), NSGA-II fits, and in multi-objective mode it is used automatically to compute the predicted Pareto front.
• When unsure, Random Search gives a robust baseline to compare against.

Acquisition functions (next-trial suggestion)

After fitting a GP surrogate (GP-FITC / GP-VFE / GP-MOE), Suggest next trials recommends parameters for the next real evaluations. These are all acquisition functions; the choice depends on whether you have one objective or several.

• EI (Expected Improvement): for single-objective optimization; the default that balances predicted improvement and uncertainty.
• LCB (Lower Confidence Bound): for single-objective optimization; picks the point with the lowest lower bound on the objective, more exploration-oriented.
• EHVI (Expected Hypervolume Improvement): for multi-objective optimization; recommends the points expected to grow the dominated hypervolume of the Pareto front the most (GP only).
• Batch candidates (up to 10) are generated with the Constant Liar strategy and exported via Copy enqueue JSON for Optuna's study.enqueue_trial().

Suggesting next trials (acquisition functions)

After fitting a GP surrogate, the Suggest next trials section appears below the optimization results. It uses an acquisition function to recommend parameter settings for the next real evaluations. These are all acquisition functions; the choice depends on whether you have one objective or several.

• EI (Expected Improvement): for single-objective optimization; balances predicted improvement and uncertainty (default).
• LCB (Lower Confidence Bound): for single-objective optimization; selects points where the lower bound on the objective is smallest.
• EHVI (Expected Hypervolume Improvement): for multi-objective optimization; in multi-objective mode a Suggest next trials (EHVI) section appears below the predicted Pareto front (GP surrogates only) and recommends parameter settings whose evaluation is expected to grow the dominated hypervolume of the Pareto front the most. It is estimated by Monte-Carlo with a fixed common-random-numbers sample matrix, which makes the suggestions deterministic and the acquisition surface smooth enough for gradient-based optimization.

Batch candidates (up to 10) are generated with the Constant Liar strategy: each selected candidate is temporarily added as a "lie" observation before refitting and selecting the next one.

The Copy enqueue JSON button copies the suggested parameters as a JSON array for use with Optuna's study.enqueue_trial().

Multi-objective (EHVI) operation

• Set the number of candidates (1–10, default 3) and click Suggest next trials (EHVI).
• The results table lists, per candidate, the parameter values, each objective's predicted value ± standard deviation, and the EHVI score.
• The button is disabled (with a hint) when any objective uses a non-GP model or while a computation is running.
• Batch candidates use the same Constant Liar strategy as the single-objective path, and the Copy enqueue JSON button copies the suggested parameters for Optuna's study.enqueue_trial().

Constraint-aware optimization

When the study has constraint columns (Optuna convention: value ≤ 0 means feasible), a Use constraints (N) checkbox appears in the Fit section, where N is the number of constraint columns.

Enabling this checkbox makes the optimizer:

1. Fit a surrogate for each constraint using the same model kind as the objective (a GP yields a smooth feasibility probability; only a perfectly linear, noise-free constraint whose GP fit degenerates falls back to Ridge for that constraint).
2. Add a constraint penalty to the cost function during optimization:

$$\text{cost}(x) = \text{sign} \cdot \hat{\mu}_y(x) + 100 \cdot \sum_i \max(0, \hat{c}$$i(x) - z{0,i})

where $z_{0,i}$ is the feasibility boundary in the normalized constraint space and $\text{sign} = 1$ for minimization, $-1$ for maximization. 3. Display the P(feasible) percentage and per-constraint predicted values in the results.

The feasibility probability shown is the product over all constraints of $P(c_i \le 0)$ (a smooth $\Phi$-based probability for GP constraints, or a hard indicator when a constraint fell back to Ridge):

$$P_\text{feas}(x) = \prod_i P(c_i \le 0 \mid x)$$

In the Suggest next trials section, the P(feas) column in the candidate table shows the feasibility probability for each suggested candidate; values below 0.5 are highlighted in orange/red as a warning.

Notes

• The search is bounded by each parameter's declared range recorded in the log (the search space). The optimizer can explore unobserved-but-valid regions within that range but never leaves it. Parameters without a declared numeric range (categorical variables or legacy Uniform-format parameters) fall back to the sampled range. GP predictions grow increasingly uncertain away from the data, so check the validation plot and predictive uncertainty alongside the results.
• The objective direction (minimize / maximize) is taken from the study metadata.
• $R^2$ reports how well the surrogate fits the training data. A low $R^2$ means the estimated optimum should not be trusted.
• The predicted optimum is a model estimate — validate it with a real evaluation before relying on it.
• Categorical parameters are excluded; only numeric parameters are optimized.
• The constraint checkbox is only shown for the single-objective path. Multi-objective mode does not currently incorporate constraints.
• While fitting, a progress bar and the current stage (model selection, cross-validation, final fit, …) are shown, and a Cancel button aborts the run. Cancellation takes effect at the next fit-step boundary.

Pareto-front-focused surrogates

For multi-objective fits, when there are more trials than the GP inducing budget (100), the inducing points are concentrated on the non-dominated (Pareto-front) trials so the surrogate is most accurate near the Pareto front, where improvements matter. With 100 trials or fewer, all points are used as inducing points (Z = X) and this focusing has no effect.

Large-data subsampling

When the number of training trials exceeds the cap (2000), the data is subsampled to a representative subset before fitting. GP-FITC training cost scales roughly linearly with the number of trials $N$, and Fit & Validate trains the same model 7 times (one 8:2 holdout + 5-fold CV + one final fit on all data), so an $N$ around ten thousand takes over a minute. Because FITC compresses the data into 100 inducing points, training on a representative subset loses almost no response-surface fidelity.

Subsampling keeps an elite band (half the budget) that always survives, then fills the rest randomly from the non-elite trials (seed-fixed, deterministic).

• Single objective: both value extremes (best and worst) are kept as elites. The fit itself is direction-agnostic, so retaining both tails preserves the optimum region whether the objective is minimized or maximized.
• Multi objective: elites are taken in ascending non-dominated rank — starting from rank 0 (the Pareto front) and expanding into rank 1, 2, … when the budget is not yet filled. The same subset is shared across all objectives so the predicted Pareto front stays consistent.

Random fill is used instead of space-filling because Optuna trials concentrate in good regions, and random sampling preserves that density; space-filling would flatten it and dilute the good region that matters for optimization.

With 2000 trials or fewer, no subsampling is performed and all trials are used for training.