hPSSKJrJrJrJrJr SSKJrJr SSK J r "SS5r g))DictIterableListUnioncast) has_torch_amptorch)is_torch_arrayc \rSrSrSrSS\S\S\S\S\4 SjjrS r SS \ S \ S 4S \ S \ S 44S jjr SS S\SS\4SjrSr\S5rSrSrSrg)PyTorchGradScalera Gradient scaler for the PyTorch shim. Gradients with small magnitudes are not representable in half-precision and will underflow to zero. A gradient scaler counters this issue by scaling up the loss before backpropagation, increasing the gradients by the same magnitude. A large enough scale will avoid that the gradients underflow. The gradients are unscaled in single precision after backpropagation, to provide the unscaled gradients to the optimizer. enabled init_scalebackoff_factor growth_factorgrowth_intervalcXlX@lX0lXPl[R "SS[R S9Ul[R "SU5UlSUl g)a Construct a gradient scaler for the PyTorch shim. enabled (bool): Sets whether the gradient scalar is enabled. If it is disabled, the methods of the grad scaler are no-ops. init_scale (float): The initial scale used to increase the gradient magnitude. backoff_factor (float): The scale will be multiplied by this factor if any of the gradients overflows. growth_factor (float): The scale will be multiplied by this factor when none of the gradients overflowed for "growth_interval" steps. growth_interval (int): When no overflows were found for this number of steps, the scale will be multiplied by "growth_factor". r)dtypeFN) _enabled_growth_factor_backoff_factor_growth_intervalr fullint_growth_tracker_scale _found_inf)selfrrrrrs Y/home/james-whalen/.local/lib/python3.13/site-packages/thinc/shims/pytorch_grad_scaler.py__init__PyTorchGradScaler.__init__sM< +- /$zz$Cjjz2 cURRU5UlURRU5UlgN)rtor)r!devices r"to_PyTorchGradScaler.to_:s/#3366v>kknnV, r%tensors torch.TensorreturncpUR(d [SU5$[S5n[5n[ U5(a[SU5nUR XTU5$[ U[5(a@/nUH6n[ U5(dUeURUR XTU55 M8 U$Ue)z)Scale up the values in the given tensors.r-z>Input to gradient scaling must be a Tensor or Iterable[Tensor]) rr ValueErrordictr _scale_tensor isinstancerappend)r!r,inplaceincorrect_typescale_per_devicetensorscaled_tensorss r"scalePyTorchGradScaler.scale>s}}0 0# L  BF ' " ".'2F%%fH H  * *N!%f--((%%&&vI "" !r%r8r7)z torch.devicer-r5c[(d [S5eUR(d Sn[U5eURnXR;aURR US9X%'X%nU(aUR U5$X-$)NzHGradient scaling is not supported, requires capable GPU and torch>=1.9.0zGradient scaling is only supported for CUDA tensors. If you are using PyTorch models, you can avoid this error by disabling mixed-precision support.r))r r0is_cudar)rr(mul_)r!r8r7r5msgr)r:s r"r2PyTorchGradScaler._scale_tensor^s }Z ~~>  S/ !  )'+{{~~V~'D  $ ( ;;u% %> !r%c[5nUH0nURUR/5nURU5 M2 U$r')r1 setdefaultr)r4)r!r,tensors_per_devicer8device_tensorss r"_tensors_per_device%PyTorchGradScaler._tensors_per_device|s@!VF/::6=="MN  ! !& )"!r%cUR$r')r )r!s r" found_infPyTorchGradScaler.found_infs r%cUR(dgURR5R5R 5nUR U5nUR 5H]upE[R"SSUS9nURUS9n[R"XVU5 [US:g5(dMVSUl M_ UR$)zNUnscale the given tensors. Returns True if any of the gradients were infinite.Frr=rT) rrdouble reciprocalfloatrFitemsr rr(*_amp_foreach_non_finite_check_and_unscale_boolr )r!r, inv_scalerDr)rEfound_inf_deviceinv_scale_devices r"unscalePyTorchGradScaler.unscales}}KK&&(335;;= "55g>&8&>&>&@ "F$zz$FC (||6|:   < <2B $)**"&'Ar%cJUR(dg[R"SUR(aSOSURR S9n[R "URURUURURUR5 SUlg)z} Update the scale factor and clear information about infinities. This method should be called after each optimization step. Nrg?rLr=F) rr rr rr)_amp_update_scale_rrrr)r!rTs r"updatePyTorchGradScaler.updates} }}  :: #c$++:L:L    KK        ! !   r%)rrr rrrrN)Fg@g?g@i)F)__name__ __module__ __qualname____firstlineno____doc__rRrOrr#r*rrrr:rr2rFpropertyrIrVrZ__static_attributes__r%r"r r s # #"# % % %  %  %  % N- QV^Xn-EEF ~tN33 4@""=>" "<". r%r N) typingrrrrrcompatr r utilr r rcr%r"rgs44)!n n r%