-jiAXSrSSKJr SSKrSSKJr SSKJr SSKrSSK J r SSK J r SSK Jr \(aSS KJr SSKrSSKrOSS KJr \"S 5r\"S 5r\"\5rSS jr"SS\R0R25r"SS5rSSSjjrg)aNotations in this Gaussian process implementation X_train: Observed parameter values with the shape of (len(trials), len(params)). y_train: Observed objective values with the shape of (len(trials), ). x: (Possibly batched) parameter value(s) to evaluate with the shape of (..., len(params)). cov_fX_fX: Kernel matrix X = V[f(X)] with the shape of (len(trials), len(trials)). cov_fx_fX: Kernel matrix Cov[f(x), f(X)] with the shape of (..., len(trials)). cov_fx_fx: Kernel scalar value x = V[f(x)]. This value is constant for the Matern 5/2 kernel. cov_Y_Y_inv: The inverse of the covariance matrix (V[f(X) + noise_var])^-1 with the shape of (len(trials), len(trials)). cov_Y_Y_inv_Y: `cov_Y_Y_inv @ y` with the shape of (len(trials), ). max_Y: The maximum of Y (Note that we transform the objective values such that it is maximized.) sqd: The squared differences of each dimension between two points. is_categorical: A boolean array with the shape of (len(params), ). If is_categorical[i] is True, the i-th parameter is categorical. ) annotationsN)Any) TYPE_CHECKING)*single_blas_thread_if_scipy_v1_15_or_newer) optuna_warn) get_logger)Callable) _LazyImportscipytorchc [R"U5n[R"U5(aU$[S5 [R"USS9n[R "U[R "U[R"[R "X[R5SS9S5[R "U[R"[R "X[R*5SS9S55$)NzDClip non-finite values to the min/max finite values for GP fittings.r)axis) npisfiniteallranyclipwheremininfmax)valuesis_values_finite is_any_finites G/home/james-whalen/.local/lib/python3.13/site-packages/optuna/_gp/gp.pywarn_and_convert_infr/s{{6* vv VWFF+!4M 77 rxx0@"&&'QXY Z\_` rxx0@266''RYZ []`a c<\rSrSr\SSj5r\SSj5rSrg)Matern52Kernel?c[R"SU-5n[R"U*5nUSU-U-S--nSUS--U-nURU5 U$)a This method calculates `exp(-sqrt5d) * (1/3 * sqrt5d ** 2 + sqrt5d + 1)` where `sqrt5d = sqrt(5 * squared_distance)`. Please note that automatic differentiation by PyTorch does not work well at `squared_distance = 0` due to zero division, so we manually save the derivative, i.e., `-5/6 * (1 + sqrt5d) * exp(-sqrt5d)`, for the exact derivative calculation. Notice that the derivative of this function is taken w.r.t. d**2, but not w.r.t. d. g?g)r sqrtexpsave_for_backward)ctxsquared_distancesqrt5dexp_partvalderivs rforwardMatern52Kernel.forward@shA 00199fW%5$44v=ABFQJ'(2 e$ rc&URunX!-$)z Let x be squared_distance, f(x) be forward(ctx, x), and g(f) be a provided function, then deriv := df/dx, grad := dg/df, and deriv * grad = df/dx * dg/df = dg/dx. ) saved_tensors)r(gradr-s rbackwardMatern52Kernel.backwardSs $$|rN)r(rr) torch.Tensorreturnr6)r(rr2r6r7r6)__name__ __module__ __qualname____firstlineno__ staticmethodr.r3__static_attributes__r5rrr r ?s($rr c\rSrSrS Sjr\S Sj5rS SjrSSSjjrSSSjjr SSjr SS jr S r g) GPRegressor]cXlX lX0lURS5URS5- R 5UlURR 5(aTUR SUR4S:R[R5UR SUR4'SUl SUl X@l XPl X`lg)N.r)_is_categorical_X_train_y_train unsqueezesquare__squared_X_diffrtyper float64 _cov_Y_Y_chol_cov_Y_Y_inv_Yinverse_squared_lengthscales kernel_scale noise_var)selfis_categoricalX_trainy_trainrNrOrPs r__init__GPRegressor.__init__^s .  ' 1 1" 58I8I"8M MVVX    # # % %$$S$*>*>%>?#Ed5==!  d&:&:!: ;3737,H)("rcS[R"URR5R 5R 55- $)Ng?)rr%rNdetachcpunumpy)rQs r length_scalesGPRegressor.length_scalesvs7RWWT>>EEGKKMSSUVVVrc ZURc URbS5e[R"5 UR 5R 5R 5R5nSSS5 W[R"URRS5==URR5- ss'[RRU5n[ RR#UR$[ RR#X R&R 5R5SS9SS9n[R("U5Ul[R("U5UlUR*R 5UlSUR*lUR.R 5UlSUR.lURR 5Ul SURlg!,(df  GN=f)Nz(Cannot call cache_matrix more than once.rT)lowerF)rLrMr no_gradkernelrXrYrZr diag_indicesrEshaperPitemlinalgcholeskyr solve_triangularTrF from_numpyrNr2rO)rQcov_Y_Y cov_Y_Y_chol cov_Y_Y_inv_Ys r _cache_matrixGPRegressor._cache_matrixzs!!)d.A.A.I 6 I]]_kkm**,00288:G  3 3A 678DNN##F+d.?.???rc jURb URcS5eURS:HnU(dUOURS5n[R R URU5=oPR5n[R RUR[R RURRUSSS9SSS9nU(abU(aS5eURXD5nXRURSS 55- n U RS SS 9RS 5 O?URnU[R R XW5- n U RS 5 U(a"URS5U RS54$Xi4$) a This method computes the posterior mean and variance given the points `x` where both mean and variance tensors will have the shape of x.shape[:-1]. If ``joint=True``, the joint posterior will be computed. The posterior mean and variance are computed as: mean = cov_fx_fX @ inv(cov_fX_fX + noise_var * I) @ y, and var = cov_fx_fx - cov_fx_fX @ inv(cov_fX_fX + noise_var * I) @ cov_fx_fX.T. Please note that we clamp the variance to avoid negative values due to numerical errors. z+Call cache_matrix before calling posterior.r$rTF)upperleftz3Call posterior with joint=False for a single point.rB)dim1dim2r)rLrMrorGr rdvecdotr`rfrgrp transposediagonal clamp_min_rOsqueeze) rQxjointis_single_pointx_ cov_fx_fXmeanV cov_fx_fxvar_s r posteriorGPRegressor.posteriorsr!!-$2E2E2Q 9 Q&&A+%Q1;;q>||"" B#?9ATATU LL ) )    LL ) )$*<*<*>*> QU\a ) b *  & ](] ]& B+Ixx (;(;B(CDDD MMrM + 6 6s ;))Iu||229@@D OOC 5D Qa1V4,Vrc0URRSnSU-[R"S[R-5-nUR 5UR [R"U[RS9--n[RRU5nUR5R5R5*n[RRX@RSS2S4SS9SS2S4nSXf--nXR-U-$)a% This method computes the marginal log-likelihood of the kernel hyperparameters given the training dataset (X, y). Assume that N = len(X) in this method. Mathematically, the closed form is given as: -0.5 * log((2*pi)**N * det(C)) - 0.5 * y.T @ inv(C) @ y = -0.5 * log(det(C)) - 0.5 * y.T @ inv(C) @ y + const, where C = cov_Y_Y = cov_fX_fX + noise_var * I and inv(...) is the inverse operator. We exploit the full advantages of the Cholesky decomposition (C = L @ L.T) in this method: 1. The determinant of a lower triangular matrix is the diagonal product, which can be computed with N flops where log(det(C)) = log(det(L.T @ L)) = 2 * log(det(L)). 2. Solving linear system L @ u = y, which yields u = inv(L) @ y, costs N**2 flops. Note that given `u = inv(L) @ y` and `inv(C) = inv(L @ L.T) = inv(L).T @ inv(L)`, y.T @ inv(C) @ y is calculated as (inv(L) @ y) @ (inv(L) @ y). In principle, we could invert the matrix C first, but in this case, it costs: 1. 1/3*N**3 flops for the determinant of inv(C). 2. 2*N**2-N flops to solve C @ alpha = y, which is alpha = inv(C) @ y. Since the Cholesky decomposition costs 1/3*N**3 flops and the matrix inversion costs 2/3*N**3 flops, the overall cost for the former is 1/3*N**3+N**2+N flops and that for the latter is N**3+2*N**2-N flops. rgdtypeNF)rx)rErbmathlogpir`rPr eyerKrdrersumrfrF)rQn_pointsconstriL logdet_partinv_L_y quad_parts rmarginal_log_likelihood#GPRegressor.marginal_log_likelihoods4==&&q)x$((1tww;"77++-$..599XU]]3["[[ LL ! !' *zz|'')--// ,,//==D3IQV/WXY[\X\]G-. "Y..rc .^^^^^ TRRSm [R"[R"TR R 5R5R55[R"TRR55[R"TRR5ST-- 5//5nS UUUU U4Sjjn[5 [RRUUSSSU0S9nSSS5 WR (d[#SUR$35e[&R("UR*5n[&R,"UST 5Tl[&R,"UT 5Tl T(a#[&R."T[&R0S 9OT[&R,"UT S-5-Tl TR35 T$!,(df  N=f) Nr$gGz?c>[R"U5RS5n[R"5 [R"UST5Tl[R"UT5TlT(a#[R"T[RS9O[R"UTS-5T-Tl TR5*T"T5- nUR5 URTS-nT(aUS:XdeSSS5 WR5URR5R5R!54$!,(df  NT=f)NTrr$r)r rhrequires_grad_ enable_gradr&rNrOtensorrKrPrr3r2rcrXrYrZ) raw_paramsraw_params_tensorlossraw_noise_var_graddeterministic_objective log_prior minimum_noisen_paramsrQs r loss_func1GPRegressor._fit_kernel_params..loss_funcs& % 0 0 < K KD Q ""$49II>OPYQY>Z4[1$)II.?.I$J!/LLemmD#4X\#BCmS 44664H %6%;%;HqL%I"26HA6MMM%99; 1 6 6 = = ? C C E K K MM M%$s CE  E.Tzl-bfgs-bgtol)jacmethodoptionszOptimization failed: r)r np.ndarrayr7ztuple[float, np.ndarray])rErbr concatenaterrNrXrYrZrOrcrPrr optimizeminimizesuccess RuntimeErrormessager rhrr&rrKrl) rQrrrrinitial_raw_paramsrresraw_params_opt_tensorrs ```` @r_fit_kernel_paramsGPRegressor._fit_kernel_paramss==&&q) ^^t88??AEEGMMOPFF4,,1134FF4>>..04-3GGH    N N"8 9..))"! *C:{{!6s{{mDE E % 0 0 7,1II6KIX6V,W)!II&;H&EF' LLemm <+@A+N!OO   -: 9s 5$H H) rErLrMrDrIrFrNrOrP)rRr6rSr6rTr6rNr6rOr6rPr6r7None)r7r)r7r)NN)rrtorch.Tensor | Nonersrr7r6)F)rr6rboolr7z!tuple[torch.Tensor, torch.Tensor])r7r6) r%Callable[[GPRegressor], torch.Tensor]rfloatrrrrr7r?) r8r9r:r;rUpropertyr[rlr`rrrr=r5rrr?r?]s#$## # '3 # # # # #0WW#8IM@%@2E@ @<#WJ!/F@8@@"& @  @  @rr?c J^^^^[R"TRSS-[RS9mS UUUU4SjjnU"5n UcU"5nSn Xi4H|n [ [R "T5[R "T5[R "T5U R U RU RS9RUUUUS9s $ [RSU S35 U"5n U R5 U $![a n U n Sn A MSn A ff=f) Nr$rrc >[[R"T5[R"T5[R"T5TSSR5TSR5TSR5S9$)NrBrzrRrSrTrNrOrP)r?r rhclone)XYdefault_kernel_paramsrRsr _default_gpr'fit_kernel_params.._default_gprKso ++N;$$Q'$$Q')>s)C)I)I)K.r288:+B/557   rr)rrrrz/The optimization of kernel parameters failed: z< The default initial kernel parameters will be used instead.)r7r?)r onesrbrKr?rhrNrOrPrrloggerwarningrl)rrrRrrr gpr_cacherrdefault_gpr_cacheerrorgpr_cache_to_usee default_gprrs``` @rfit_kernel_paramsr>s%"JJqwwqzA~U]]K  % N E': $//?((+((+-=-Z-Z-::*44 ! #+(? ! ;$ NN :5'BF F.K  E sA7D  D"DD")rrr7r)Ng{Gz?)rrrrrRrrrrrrrrzGPRegressor | Nonerrr7r?)__doc__ __future__rrtypingrrrZr"optuna._gp.scipy_blas_thread_patchroptuna._warningsroptuna.loggingrcollections.abcr r r optuna._importsr r8rrautogradFunctionr r?rr5rrrs&#  Y(%(+  E  E H   U^^,,<^^P%)77775 7  7 " 7"7 77r