-jiiSSKJr SSKJr SSKrSSKrSSKrSSKJr SSKJ r SSKJ r SSKJ r SSK r SSKrSSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJr SSKJr SSKJ r SSK!J"r" SSK#J$r$ SSK%J&r& SSK'J(r( SSK'J)r) \ (a+SSK*r*\ \*RV\*RX\*RZ4r.O\"S5r*\R^"\05r1Sr2Sr3"SS\5r4SSjr5g)) annotations)SequenceN)Any)cast) TYPE_CHECKING)Union) _deprecated)logging)warn_experimental_argument) _LazyImport)_SearchSpaceTransform) optuna_warn)BaseDistribution)FloatDistribution)IntDistribution) BaseSampler)&_INDEPENDENT_SAMPLING_WARNING_TEMPLATE)LazyRandomState)IntersectionSearchSpace)StudyDirection) FrozenTrial) TrialStatecmaesg|=ic \rSrSrSrSSSSSSSSSS.SSjjjrSSjrSS jrSS jr\ SS j5r \ SS j5r SS jr SSjr S SjrS!SjrS"SjrS#SjrS$SjrS%SjrS&SjrS'SjrSrg)( CmaEsSampler/u$A sampler using `cmaes `__ as the backend. Example: Optimize a simple quadratic function by using :class:`~optuna.samplers.CmaEsSampler`. .. code-block:: console $ pip install cmaes .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y sampler = optuna.samplers.CmaEsSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=20) Please note that this sampler does not support CategoricalDistribution. However, :class:`~optuna.distributions.FloatDistribution` with ``step``, (:func:`~optuna.trial.Trial.suggest_float`) and :class:`~optuna.distributions.IntDistribution` (:func:`~optuna.trial.Trial.suggest_int`) are supported. If your search space contains categorical parameters, I recommend you to use :class:`~optuna.samplers.TPESampler` instead. Furthermore, there is room for performance improvements in parallel optimization settings. This sampler cannot use some trials for updating the parameters of multivariate normal distribution. For further information about CMA-ES algorithm, please refer to the following papers: - `N. Hansen, The CMA Evolution Strategy: A Tutorial. arXiv:1604.00772, 2016. `__ - `A. Auger and N. Hansen. A restart CMA evolution strategy with increasing population size. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2005), pages 1769–1776. IEEE Press, 2005. `__ - `N. Hansen. Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed. GECCO Workshop, 2009. `__ - `Raymond Ros, Nikolaus Hansen. A Simple Modification in CMA-ES Achieving Linear Time and Space Complexity. 10th International Conference on Parallel Problem Solving From Nature, Sep 2008, Dortmund, Germany. inria-00287367. `__ - `Masahiro Nomura, Shuhei Watanabe, Youhei Akimoto, Yoshihiko Ozaki, Masaki Onishi. Warm Starting CMA-ES for Hyperparameter Optimization, AAAI. 2021. `__ - `R. Hamano, S. Saito, M. Nomura, S. Shirakawa. CMA-ES with Margin: Lower-Bounding Marginal Probability for Mixed-Integer Black-Box Optimization, GECCO. 2022. `__ - `M. Nomura, Y. Akimoto, I. Ono. CMA-ES with Learning Rate Adaptation: Can CMA-ES with Default Population Size Solve Multimodal and Noisy Problems?, GECCO. 2023. `__ .. seealso:: You can also use `optuna_integration.PyCmaSampler `__ which is a sampler using cma library as the backend. Args: x0: A dictionary of an initial parameter values for CMA-ES. By default, the mean of ``low`` and ``high`` for each distribution is used. Note that ``x0`` is sampled uniformly within the search space domain for each restart if you specify ``restart_strategy`` argument. sigma0: Initial standard deviation of CMA-ES. By default, ``sigma0`` is set to ``min_range / 6``, where ``min_range`` denotes the minimum range of the distributions in the search space. seed: A random seed for CMA-ES. n_startup_trials: The independent sampling is used instead of the CMA-ES algorithm until the given number of trials finish in the same study. independent_sampler: A :class:`~optuna.samplers.BaseSampler` instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for :class:`~optuna.samplers.CmaEsSampler` is determined by :func:`~optuna.search_space.intersection_search_space()`. If :obj:`None` is specified, :class:`~optuna.samplers.RandomSampler` is used as the default. .. seealso:: :class:`optuna.samplers` module provides built-in independent samplers such as :class:`~optuna.samplers.RandomSampler` and :class:`~optuna.samplers.TPESampler`. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case. restart_strategy: Strategy for restarting CMA-ES optimization when converges to a local minimum. If :obj:`None` is given, CMA-ES will not restart (default). If 'ipop' is given, CMA-ES will restart with increasing population size. if 'bipop' is given, CMA-ES will restart with the population size increased or decreased. Please see also ``inc_popsize`` parameter. .. warning:: Deprecated in v4.4.0. ``restart_strategy`` argument will be removed in the future. The removal of this feature is currently scheduled for v6.0.0, but this schedule is subject to change. From v4.4.0 onward, ``restart_strategy`` automatically falls back to ``None``, and ``restart_strategy`` will be supported in OptunaHub. See https://github.com/optuna/optuna/releases/tag/v4.4.0. popsize: A population size of CMA-ES. inc_popsize: Multiplier for increasing population size before each restart. This argument will be used when ``restart_strategy = 'ipop'`` or ``restart_strategy = 'bipop'`` is specified. .. warning:: Deprecated in v4.4.0. ``inc_popsize`` argument will be removed in the future. The removal of this feature is currently scheduled for v6.0.0, but this schedule is subject to change. From v4.4.0 onward, ``inc_popsize`` is no longer utilized within Optuna, and ``inc_popsize`` will be supported in OptunaHub. See https://github.com/optuna/optuna/releases/tag/v4.4.0. consider_pruned_trials: If this is :obj:`True`, the PRUNED trials are considered for sampling. .. note:: Added in v2.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.0.0. .. note:: It is suggested to set this flag :obj:`False` when the :class:`~optuna.pruners.MedianPruner` is used. On the other hand, it is suggested to set this flag :obj:`True` when the :class:`~optuna.pruners.HyperbandPruner` is used. Please see `the benchmark result `__ for the details. use_separable_cma: If this is :obj:`True`, the covariance matrix is constrained to be diagonal. Due to reduce the model complexity, the learning rate for the covariance matrix is increased. Consequently, this algorithm outperforms CMA-ES on separable functions. .. note:: Added in v2.6.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.6.0. with_margin: If this is :obj:`True`, CMA-ES with margin is used. This algorithm prevents samples in each discrete distribution (:class:`~optuna.distributions.FloatDistribution` with ``step`` and :class:`~optuna.distributions.IntDistribution`) from being fixed to a single point. Currently, this option cannot be used with ``use_separable_cma=True``. .. note:: Added in v3.1.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.1.0. lr_adapt: If this is :obj:`True`, CMA-ES with learning rate adaptation is used. This algorithm focuses on working well on multimodal and/or noisy problems with default settings. Currently, this option cannot be used with ``use_separable_cma=True`` or ``with_margin=True``. .. note:: Added in v3.3.0 or later, as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. source_trials: This option is for Warm Starting CMA-ES, a method to transfer prior knowledge on similar HPO tasks through the initialization of CMA-ES. This method estimates a promising distribution from ``source_trials`` and generates the parameter of multivariate gaussian distribution. Please note that it is prohibited to use ``x0``, ``sigma0``, or ``use_separable_cma`` argument together. .. note:: Added in v2.6.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.6.0. NF)consider_pruned_trialsrestart_strategypopsize inc_popsizeuse_separable_cma with_marginlr_adapt source_trialscUcU S:wa2[RRSSSS9n[US3[5 XlX lU=(d [RRUS9Ul X0l XPl [U5Ul[5UlXplXlXlXlXlXlUR&(aSUlO UR((aS UlOS UlUR"(a [1S 5 UR&(a [1S 5 UR,b [1S 5 UR((a [1S5 UR*(a [1S5 UbUcUb [3S5eUbU (a [3S5eU (aU (dU (a [3S5eUR&(aUR((a [3S5egg)Nrz`restart_strategy`z4.4.0z6.0.0)named_verr_verz~ From v4.4.0 onward, `restart_strategy` automatically falls back to `None`. `restart_strategy` will be supported in OptunaHub.)seedzsepcma:zcmawm:zcma:rr"r%r#r$zQIt is prohibited to pass `source_trials` argument when x0 or sigma0 is specified.zNIt is prohibited to pass `source_trials` argument when using separable CMA-ES.z]It is prohibited to pass `use_separable_cma` or `with_margin` argument when using `lr_adapt`.zMCurrently, we do not support `use_separable_cma=True` and `with_margin=True`.)r _DEPRECATION_WARNING_TEMPLATEformatr FutureWarning_x0_sigma0optunasamplers RandomSampler_independent_sampler_n_startup_trials_warn_independent_samplingr_cma_rngr _search_space_consider_pruned_trials_popsize_use_separable_cma _with_margin _lr_adapt_source_trials _attr_prefixr ValueError)selfx0sigma0n_startup_trialsindependent_samplerwarn_independent_samplingr*rrr r!r"r#r$r%msgs P/home/james-whalen/.local/lib/python3.13/site-packages/optuna/samplers/_cmaes.py__init__CmaEsSampler.__init__s$  ';"+<;;BB)CC %MM   $7$c6??;X;X^b;X;c!!1*C''- 46'=$ "3'!+  " " )D     (D  &D   ' ' &'? @  " " &': ;    * & 7    &} 5 >> &z 2  $".F">u"E"K"K"M D""$$l->,PQQ!- #NrJcFURU5 [U5S:Xa0$URU5n[U5UR:a0$[ X0R (+SS9nUR U5nUcURXQR5nUR[UR5:waSUR(a@URRRn[R!SUS35 SUl 0$UR#XFR$5n[U5UR&:GaK/n USUR&Hn U R(cS5e[+U[,R.5(a$[0R2"U R4S5n OUR7U R85n UR[:R<:Xa U R(O U R(*n U R?X45 M URAU 5 [BRD"U5RG5n URIU 5nUH,nURJRMURNXU5 M. URPRRRUS S 5URV-nURXR[U5 [+U[,R.5(aIUR]5unnURJRMURNSUR_55 OUR]5nUR`nURJRMURNUUR$5 URcU5nU$) NrT)transform_step transform_0_1z7`CmaEsSampler` does not support dynamic search space. `z$` is used instead of `CmaEsSampler`.Fz"completed trials must have a value x_for_telli)2_raise_error_if_multi_objectivelen _get_trialsr4r r;_restore_optimizer_init_optimizer directiondimboundsr5r3 __class____name___loggerwarning_get_solution_trials generationpopulation_sizevaluerTrCMAwMnparray system_attrs transformparamsrMINIMIZEappendtellpickledumpshex_split_optimizer_str_storageset_trial_system_attr _trial_idr6rngrandintnumber_rngr*asktolist_attr_key_generation untransform)r@rUrVrWcompleted_trialstrans optimizerind_sampler_namesolution_trials solutionstxy optimizer_stroptimizer_attrskeyr*rur^generation_attr_keyexternal_valuess rGsample_relativeCmaEsSampler.sample_relativees ,,U3 |  !I++E2  4#9#9 9I& ->->)>d ++,<=  ,,UOODI ==C - -..#'#<#<#F#F#O#O ())MO38/I334DFZFZ[  9#<#< <8:I$%@y'@'@Aww*P,PP*i55 !=>A1A$.2I2IIAGGPQPWPWx  !(B NN9 %#LL3779M"77 FO&44U__c[^K_`'}}  ((E2U\\AD! i - -!* FJ NN 0 0z/@/@/B ]]_F"77 ,, OO0)2F2F  ++F3rJc URS-$)Nrmr>rNs rGr!CmaEsSampler._attr_key_generations  <//rJc URS-$)NrrrNs rG_attr_key_optimizer CmaEsSampler._attr_key_optimizers  ;..rJcb^^SRUU4Sj[[T5555$)Nc3L># UHnTTRSU3v M g7f):N)r).0irr@s rG 7CmaEsSampler._concat_optimizer_attrs..s, Hc1Ot778!= >Hcs!$)joinrangera)r@rs``rG_concat_optimizer_attrs$CmaEsSampler._concat_optimizer_attrss,ww HMcRaNbHc   rJc[U5n0n[[R"U[- 55H7nU[-n[ US-[-U5nXUX0R SU3'M9 U$)Nr_r)rarmathceil_SYSTEM_ATTR_MAX_LENGTHminr)r@r optimizer_lenattrsrstartends rGr|!CmaEsSampler._split_optimizer_strswM* tyy1H!HIJA//Eq1u 77GC7D37OE--.as3 4K rJcj[U5HnURR5VVs0sH)up4URUR5(dM'X4_M+ nnn[ U5S:XaMdUR U5n[R"[RU55s $ gs snnf)Nr) reversedrsrR startswithrrarryloadsbytesfromhex)r@rrVrrorrs rGrcCmaEsSampler._restore_optimizers ./E#("4"4":":"<">$":":; "<  ?#q( 88IM<< m <= =0s &B/B/c URSS2S4nURSS2S4n[UR5nURclURc X4U- S- -nOUR UR5nUR c[ R"XC- S- 5nO UR nSnGO [R/n UR(aU R[R5 U[R:XaSOSn URV s/sHin U RU ;dM[!XR"5(dM1UR U R$5U ['[(U R*5-4PMk n n [U 5S:Xa [-S5e[.R0"U 5upgn[3U[45nUR6(a~[UR5S:Xa[9S[:5 OT[.R<"UUURUR>R@RCSS5S U-URDS 9$URF(Ga7[ RH"[URJ5[(S 9n [MURJRO55Hup[QU[R[T45(deURVbURX(aS X'MFURZUR\:XaS X'MfURVUR\URZ- - X'M [.R^"UUURU UUR>R@RCSS5S U-URDS9$[.R`"UUUURUR>R@RCSS5S U-URDURbS9$s sn f)Nrr_rzNo compatible source_trialszSeparable CMA-ES does not operate meaningfully on single-dimensional search spaces. The setting `use_separable_cma=True` will be ignored.i )meansigmargr*n_max_resamplingrn)dtypegg?)rrrgstepscovr*rrn)rrrrgr*rrnr$)2rgrar=r.rtr/rqrrCOMPLETEr8rwPRUNEDrrvstate_is_compatible_search_space distributionsrurfloatror?rget_warm_start_mgdmax_EPSr:r UserWarningSepCMAr6rrr9r;emptyr7 enumeratevaluesrTrrsteploglowhighrpCMAr<)r@rre lower_bounds upper_bounds n_dimensionrrBrexpected_statessignrsource_solutionsrrdists rGrdCmaEsSampler._init_optimizersZ ||AqD) ||AqD) %,,'    &xx#l'Ba&GGtxx0||#!< ABC)223O++&&z'8'89"^%<%<<1"D,, ,A77o-I0GI*D4qww3G,GH,   #$) !>??!& 8 89I J D#VT"  " "5<< A%[ ||  <<**221i@%'+%5$(MM     HHS!4!45UCE$U%8%8%?%?%AB!$:K(LMMMM99$"EHXX*"EH#yyDII,@AEHC;;||]]&&..q)>3E3EFY[]3^%^6___s #AAc:URRX5 grL)r3 before_trial)r@rUrVs rGrCmaEsSampler.before_trialrs !!..ur6r8r3r<r4r9r7r/r=r:r5r;r.)NNr_NTN)rAzdict[str, Any] | NonerBz float | NonerCintrDzBaseSampler | NonerEboolr* int | Nonerrrz str | Noner rr!rr"rr#rr$rr%zlist[FrozenTrial] | NonereturnNone)rr)rU'optuna.Study'rV'optuna.trial.FrozenTrial'rdict[str, BaseDistribution])rUrrVrrWrrzdict[str, Any])rstr)rdict[str, str]rr)rrrr)rz 'list[optuna.trial.FrozenTrial]'rz'CmaClass' | None)rr rerrz 'CmaClass') rUrrVrrrrrrr)rVrrrrr)rUrrlist[FrozenTrial])rr rmrrr )rUz optuna.StudyrVrrr) rUrrVrrrrzSequence[float] | Nonerr)ri __module__ __qualname____firstlineno____doc__rHrMrYrpropertyrrrr|rcrdrrrbrlrr__static_attributes__rJrGrr/sIFT%)# !26*.U(-'+""'!26!U !UU U 0 U $( UU!%U%UUU UUU 0!U" #Un/#,F $$KK*K2 K  KZ00// : $] $] "]   ] ~  *   -   $  &`'`58` ` =KK*K K ' K  KrJrc[[URR55R UR555nU[UR5s=:H=(a [U5:H$s $rL)rasetr7keys intersection)rrWintersection_sizes rGrrs`C 3 3 8 8 :;HHIZIZI\]^ E$7$7 8 M MC r's"$   ;'3(120'H>78$#UYY ekk9:H  E   X & M K;M K`N N0KN NrJ