Covariance Matrix Adaptation Evolution Strategy

CMA-ES is a stochastic, derivative-free optimization algorithm for single-objective continuous optimization. It belongs to the family of Evolution Strategies (ES) and is particularly effective when:

variables are correlated (the objective is non-separable), or
dimensions have very different scales (ill-conditioned landscape)

Unlike genetic algorithms such as NSGA-II, CMA-ES does not use crossover. Instead, it samples new candidates from a multivariate normal distribution and adapts that distribution based on the history of successful search steps.

How CMA-ES Works

Optimization by CMA-ES is performed by repeatedly executing STEP2 to STEP4.

STEP1: Initialization

Three quantities are initialized:

Mean m: the starting point (typically the center of the search range)
Step size σ: controls the overall scale of the search
Covariance matrix C = I: starts as the identity matrix (isotropic, no correlation between variables)

STEP2: Sampling

A population of λ candidate solutions is drawn from a multivariate normal distribution:

x_i ~ m + σ · N(0, C),  i = 1, ..., λ

Each candidate represents one set of variable values to evaluate.

STEP3: Selection and Mean Update

Candidates are ranked by their objective value. The best μ candidates (μ < λ) are selected, and the mean is updated as a weighted centroid of those candidates:

m ← Σ w_i · x_i   (weighted average of the top μ)

STEP4: Covariance Matrix and Step Size Adaptation

This is the defining step of CMA-ES:

Covariance matrix update: C is adapted to reflect the direction and magnitude of successful steps. Over iterations, C learns the correlations between variables and reshapes the sampling ellipsoid to match the problem's geometry.
Step size update (σ): increased when progress is steady (the search moves consistently in one direction) and decreased when oscillating. This keeps exploration efficient throughout the run.

STEP2 through STEP4 are repeated until the number of trials is reached.

Image of behavior

The animation below shows how the sampling distribution (ellipse) gradually rotates and scales to align with the objective function's contours.

CMA-ES behavior

Key Properties

Property	Detail
Objective type	Single-objective only
Variable type	Continuous
Invariance	Rotation- and scale-invariant
Recommended number of variables	Up to ~100

Rotation invariance means CMA-ES performs equally well regardless of how the coordinate axes are oriented relative to the problem. This is a significant advantage over methods that assume axis-aligned search directions (such as independent random or QMC sampling).

When to Use CMA-ES

Single-objective, continuous optimization
Variables are correlated or the landscape is not axis-aligned
The number of variables is moderate (roughly 2–100)
Bayesian methods (TPE, GP) are not converging well

Derivative Samplers (added in v1.5.0)

As of v1.5.0, what used to be a single CMA-ES sampler is offered as three separate samplers, so you pick the variant directly from the sampler list instead of toggling an option inside CMA-ES.

Restart CMA-ES restarts the search with a new population once the run stagnates, using either the IPOP-CMA-ES strategy (population size grows on each restart) or BIPOP-CMA-ES (alternates large- and small-population restarts). This helps escape local optima that a single CMA-ES run without restarts can get stuck in. It supports continuous variables only, single objective, and no constraints.
CMA-ES with Refinement splits the trial budget into three phases: a Sobol quasi-Monte Carlo pass for space-filling initialization, a CMA-ES phase for covariance matrix adaptation, and a quasi-random Gaussian refinement phase that does a local search around the best point found so far. Phase lengths scale automatically with the number of trials you configure. It is also single-objective only.

Start with plain CMA-ES; move to Restart CMA-ES if a single run keeps converging to the same suboptimal point, and to CMA-ES with Refinement if you want the last portion of the trial budget spent polishing the best point rather than exploring further.

CMA-ES