-ji-TFSSKJr SSKJr SSKJr SSKrSSKrSSKJ r SSKJ r SSK J r SSK J r SS K Jr SS K Jr SS KJr SS KJr SS KJr SSKJr SSKJr \(aJSSKJr SSKJr SSKrSSKJs Jr SSK!Js J"r" SSK#Js J$r$ SSK%Js J&r& SSK'Js J(r) SSK*J+r+ SSKJ,r, O6SSK-J.r. \."S5r\."S5r)\."S5r"\."S5r$\."S5r \."S5r&SSK/r/\/R`"\15r2Sr3S!Sjr4\ "S5"SS\55r5S"S jr6g)#) annotations)Any) TYPE_CHECKINGN)experimental_class)warn_experimental_argument)_CONSTRAINTS_KEY)&_INDEPENDENT_SAMPLING_WARNING_TEMPLATE) _process_constraints_after_trial) BaseSampler)LazyRandomState)StudyDirection)_is_pareto_front) FrozenTrial) TrialState)Callable)Sequence)BaseDistribution)Study) _LazyImporttorchzoptuna._gp.search_spacez optuna._gp.gpzoptuna._gp.optim_mixedzoptuna._gp.acqfzoptuna._gp.priorg|=c[R"U5n[R"USS9n[R"USS9nX- [R "[ U5- nXBU4$)Nraxis)gpwarn_and_convert_infnpmeanstdmaximumEPS)valuesclipped_valuesmeansstdsstandardized_valuess U/home/james-whalen/.local/lib/python3.13/site-packages/optuna/samplers/_gp/sampler.py_standardize_valuesr'5sV,,V4N GGN +E 66.q )D)1RZZT5JJ t ++z3.6.0c\rSrSrSrSSSSSSS.SSjjrSS jrSS jrSS jrSS jr SS jr SSjr SSjr SSjr SSjrSSjrSrg) GPSampler=aSampler using Gaussian process-based Bayesian optimization. This sampler fits a Gaussian process (GP) to the objective function and optimizes the acquisition function to suggest the next parameters. The current implementation uses Matern kernel with nu=2.5 (twice differentiable) with automatic relevance determination (ARD) for the length scale of each parameter. The hyperparameters of the kernel are obtained by maximizing the marginal log-likelihood of the hyperparameters given the past trials. To prevent overfitting, Gamma prior is introduced for kernel scale and noise variance and a hand-crafted prior is introduced for inverse squared lengthscales. As an acquisition function, we use: - log expected improvement (logEI) for single-objective optimization, - log expected hypervolume improvement (logEHVI) for Multi-objective optimization, and - the summation of logEI and the logarithm of the feasible probability with the independent assumption of each constraint for (black-box inequality) constrained optimization. For further information about these acquisition functions, please refer to the following papers: - `Unexpected Improvements to Expected Improvement for Bayesian Optimization `__ - `Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization `__ - `Bayesian Optimization with Inequality Constraints `__ Please also check our articles: - `[Optuna v4.5] Gaussian Process-Based Sampler (GPSampler) Can Now Perform Constrained Multi-Objective Optimization `__ - `[Optuna v4.2] Gaussian Process-Based Sampler (GPSampler) Can Now Handle Inequality Constraints `__ - `Introducing Optuna's Native GPSampler `__ The optimization of the acquisition function is performed via: 1. Collect the best param from the past trials, 2. Collect ``n_preliminary_samples`` points using Quasi-Monte Carlo (QMC) sampling, 3. Choose the best point from the collected points, 4. Choose ``n_local_search - 2`` points from the collected points using the roulette selection, 5. Perform a local search for each chosen point as an initial point, and 6. Return the point with the best acquisition function value as the next parameter. Decoupled optimizer update with a batched evaluation is employed to perform a batch of local searches simultaneously, specifically speeding up Step 5 above. The following paper details the methodology: - `Batch Acquisition Function Evaluations and Decouple Optimizer Updates for Faster Bayesian Optimization `__ Note that the procedures for non single-objective optimization setups are slightly different from the single-objective version described above, but we omit the descriptions for the others for brevity. The local search iteratively optimizes the acquisition function by repeating: 1. Gradient ascent using l-BFGS-B for continuous parameters, and 2. Line search or exhaustive search for each discrete parameter independently. The local search is terminated if the routine stops updating the best parameter set or the maximum number of iterations is reached. We use line search instead of rounding the results from the continuous optimization since EI typically yields a high value between one grid and its adjacent grid. .. note:: This sampler requires ``scipy`` and ``torch``. You can install these dependencies with ``pip install scipy torch``. Args: seed: Random seed to initialize internal random number generator. Defaults to :obj:`None` (a seed is picked randomly). independent_sampler: Sampler used for initial sampling (for the first ``n_startup_trials`` trials) and for conditional parameters. Defaults to :obj:`None` (a random sampler with the same ``seed`` is used). n_startup_trials: Number of initial trials. Defaults to 10. deterministic_objective: Whether the objective function is deterministic or not. If :obj:`True`, the sampler will fix the noise variance of the surrogate model to the minimum value (slightly above 0 to ensure numerical stability). Defaults to :obj:`False`. Currently, all the objectives will be assume to be deterministic if :obj:`True`. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible. If ``constraints_func`` returns more than one value for a trial, that trial is considered feasible if and only if all values are equal to 0 or smaller. The ``constraints_func`` will be evaluated after each successful trial. The function won't be called when trials fail or are pruned, but this behavior is subject to change in future releases. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler, meaning that no GP model is used in the sampling. Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case. N FT)seedindependent_samplern_startup_trialsdeterministic_objectiveconstraints_funcwarn_independent_samplingc[U5UlU=(d [RR US9Ul[R R5UlX0l [RUl [RUlSUlSUlX@lXPlX`lUb [)S5 SUlSUlSUlg)N)r-r1ir,g-C6?)r _rngoptunasamplers RandomSampler_independent_sampler search_spaceIntersectionSearchSpace_intersection_search_space_n_startup_trialspriordefault_log_prior _log_priorDEFAULT_MINIMUM_NOISE_VAR_minimum_noise_gprs_cache_list_constraints_gprs_cache_list_deterministic_constraints_func_warn_independent_samplingr_n_preliminary_samples_n_local_search_tol)selfr-r.r/r0r1r2s r&__init__GPSampler.__init__s$D) $7$c6??;X;X^b;X;c!*0*=*=*U*U*W'!1DID[D[%*%D%D>BIM)5!1*C'  ' &'9 :,0#! r(c[R"UURURRR URR SS9n[ RU5 g)Nz2dynamic search space is not supported by GPSampler) param_name trial_numberindependent_sampler_name sampler_namefallback_reason)r formatnumberr8 __class____name___loggerwarning)rJtrialrNmsgs r&_log_independent_sampling#GPSampler._log_independent_samplingsO4;;!%)%>%>%H%H%Q%Q00P   r(cURRR5 URR 5 gN)r4rngr-r8 reseed_rng)rJs r&r`GPSampler.reseed_rngs(  !!,,.r(c0nURRU5R5H upEUR5(aMXSU'M" U$r^)r; calculateitemssingle)rJstudyrYr9name distributions r&infer_relative_search_space%GPSampler.infer_relative_search_spacesS "&"A"A"K"KE"R"X"X"Z D""$$!- #[ r(c Ub[UR5S:Xde[R"UUURUR UR URRS9up4U$)N)!warmstart_normalized_params_arrayn_preliminary_samplesn_local_searchtolr_) lenshape optim_mixedoptimize_acqf_mixedrGrHrIr4r_)rJacqf best_paramsnormalized_params _acqf_vals r&_optimize_acqfGPSampler._optimize_acqfsh "c+*;*;&<&AAA'2'F'F .9"&"="=//  ( $! r(c >[U*5upEnURb7[URSR5UR:waSUlUR n/n/n U*[R"[U5- R5n [UR5HlupUR bUR U OSn [R"UU UURUR U UR"S9n UR%U 5 Mn XlX4$)NrXYis_categorical log_prior minimum_noise gpr_cacher0)r'rBrqinverse_squared_lengthscalesdimrCrrrr tolist enumerateTrfit_kernel_paramsr?rArDappend)rJconstraint_valsinternal_search_spacerwstandardized_constraint_valsr#r$rconstraints_gprsconstraints_threshold_listivalscachegprs r&_get_constraints_acqf_args$GPSampler._get_constraints_acqf_argss$5HHX4Y1$T  ! ! -D))!,IIJ$(()15D -.==%'"',frzz#t/D&D%L%L%N" !=!?!?@GA44@11!4  &&#-//"11(,(;(;C  # #C (A"-=);;r(cU[U*SS9n[U5n[URS-U5nURR R XESS9nX6$)NF)assume_unique_lexsortedrl)sizereplace)rrqminrHr4r_choice)rJrwstandardized_score_vals pareto_params n_pareto_solsrchosen_indicess r&$_get_best_params_for_multi_objective.GPSampler._get_best_params_for_multi_objective!sf * 55u U M* 4''1,m<--mPU-V,,r(c  U0:Xa0$[R4nURSUSS9n[U5UR:a0$[ R "U5nURU5n[R"URVs/sHo[R:XaSOSPM sn5n [U [R"UVs/sHo"RPM sn5-5un n URb7[URSR 5UR":waSUl/n U R$Sn UR&n[)U 5HonURbURUOSnU R+[,R."UU SS2U4UUR0UR2UUR4S95 Mq XlUR6cU S :Xaj[U 5S :Xde[8R:"U SUU SS2S4R=5S 9nU[R>"U 5[R@4nGO[8RB"U U[DRF"U 5S URHRJRMS 5S 9nUROXz5nGOU S :Xa[U 5S :Xde[QX5unn[RR"UU SS2S4[RT*5nURWUXg5unn[R>"U5nUUn[8RX"U SUUUUS9nUR$SSUR$:Xde[RZ"U5(aSOUU[R@4nO[QX5unnURWUXg5unn[]U5(+n[8R^"U UU(d[DRF"U U5OSS URHRJRMS 5UUS9nU(dUROUUU U5OSnURaUU5nURcU5$s snfs snf)NFTdeepcopystates use_cachegg?rr|)rr9 thresholdi@)gpr_listr9Y_train n_qmc_samplesqmc_seed)rr9rconstraints_gpr_listr)rr9 Y_feasiblerrrr)2rCOMPLETE _get_trialsrqr<gp_search_space SearchSpaceget_normalized_paramsrarray directionsr MINIMIZEr'r!rBrrrrrrangerrrr?rArDrE acqf_moduleLogEImaxargmaxnewaxisLogEHVIr from_numpyr4r_randintr$_get_constraint_vals_and_feasibilitywhereinfrConstrainedLogEIisneginfanyConstrainedLogEHVIryget_unnormalized_param)rJrfrYr9rtrialsrrwd_signr_ gprs_list n_objectivesrrrrurvr is_feasible y_with_neginfconstr_gpr_listconstr_threshold_listi_optbest_feasible_yis_all_infeasiblenormalized_params r&sample_relativeGPSampler.sample_relative/sq 2 I%%'""E&D"Q v;// /I / ; ;L I1GGOQVQaQabQaA~'>'>">$CGQabc(; BHH?ull?@ @) %A  ! ! -D))!,IIJ$(()%)D ! .44R8 .==|$A040E0E0QD))!,W[E   $$'-ad3#1"oo"&"5"5#,0,?,? %!*  ! ! )q 9~***"((!! !65ad;??A 0 :Q0RTVT^T^0^_ "**&!6!,,-DE"%!YY]]227; #GG% q 9~***/STY/b, "6MaQRd6SVXV\V\U\ ] :>9X9X#%::6!6 -0"/"6"33!! !6-)8/D )..s3}7J7JJJJKK88D>OPUWYWaWaPa>b0TTY/b,9=9X9X#%::6!6),K(8$8!"55&!6 1(()@)MN!"%!YY]]227;)8/D $- ==)+6/ <   ..t[A$;; snm[ U4SjT55(a [ S5e[R"T5n[UR5S:XdS5e[R"US:*SS9n[U[R5(aS 5eX44$s snf) Nrc3X># UHn[TS5[U5:gv M! g7f)rN)rq).0c_constraint_valss r& 7_get_constraint_vals_and_feasibility..s' H7G!3" #s1v -7Gs'*z:The number of constraints must be the same for all trials.rlz#constraint_vals must be a 2d array.rrrz#MyPy Redefinition for NumPy v2.2.0.)_storageget_trial_system_attrs _trial_idgetrr ValueErrorrrrqrrall isinstancebool_)rfrrYrrrs @r&rrs E --eoo>BBCSUWX H7G HHHUVVhh/0O $$ % *Q,QQ *&&A-A6K+rxx00W2WW 0  ''sAC()r!rrz)tuple[np.ndarray, np.ndarray, np.ndarray])rfrrzlist[FrozenTrial]rztuple[np.ndarray, np.ndarray])7 __future__rtypingrrnumpyrr5optuna._experimentalrroptuna.samplers._baserr r r "optuna.samplers._lazy_random_stater optuna.studyr optuna.study._multi_objectiver optuna.trialrrcollections.abcrrroptuna._gp.acqf_gprur optuna._gp.gproptuna._gp.optim_mixedrsoptuna._gp.priorr=optuna._gp.search_spacer9roptuna.distributionsrroptuna._importsrlogging getLoggerrVrWr r'r*rrr(r&rs "  3;2HB->':$#(())00$$555"+  E!";