ȅiSSKrSSKrSSKJr SSKJrJr SSKJrJ r \ "SS5r \R\R\R\R\R/r\R"\R$/r\Vs0sH9o\R)U5R*\R)U5R,4_M; snr\R1\Vs0sHEo\"\R5U5R*5\"\R5U5R,54_MG sn5 Sr\ R9S5 \"\ S S 5S \R:S \S \S\S\S\R>S\R:4Sj5r \"\ S S5S \R:S \S \S\S\S\R>S\R:4Sj5r!\ R9S5 \"\ SS 5S \R:S \R:S \R:S\S\S\R>S\R:4Sj5r"\"\ SS5S \R:S \R:S \R:S\S\S\R>S\R:4Sj5r#\ R9S5 \"\ SS 5S \R:S \R:S \R:S\R:S\R:S\R>S\R:4Sj5r$\"\ SS5S \R:S \R:S \R:S\R:S\R:S\R>S\R:4Sj5r%\ R9S5 \"\ SS 5SS.S \R:S \S \S\S\S\R>S \R>S-S\R:4S!jj5r&\"\ SS5SS.S \R:S \R:S \R:S\S\S\R>S \R>S-S\R:4S"jj5r'\ R9S#5 \"\ S$S 5SS.S \R:S \R:S \R:S\S\S\R>S \R>S-S\R:4S%jj5r(\"\ S$S5SS.S \R:S \R:S \R:S\S\S\R>S \R>S-S\R:4S&jj5r)\ R9S'5 \"\ S(S 5SS.S \R:S \R:S \R:S\R:S\R:S\R>S \R>S-S\R:4S)jj5r*\"\ S(S5SS.S \R>S-S\R:4S*jj5r+\ R9S+5 \"\ S,S 5S \R:S-\S.\S/\S\R>S\,\R:\R:44 S0j5r-\ R9S15 \"\ S2S 5S \R:S-\S.\S/\S\R>S\,\R:\R:44 S3j5r.\"\ S,S5S \R:S\S\S/\S\R>S\,\R:\R:44 S4j5r/\"\ S2S5S \R:S\S\S/\S\R>S\,\R:\R:44 S5j5r0S6r1\ R9S75 \"\ S8S 5S \R:S9\R:S:\R:S;\S\S\S\R>S\R:4S<j5r2\"\ S8S5S \R:S9\R:S:\R:S;\S\S\S\R>S\R:4S=j5r3\ R9S>5 \"\ S?S 5SS.S \R:S9\R:S:\R:S-S;\S\S\S\R>S \R>S-S\R:4S@jj5r4\"\ S?S5SS.S \R:S9\R:S:\R:S-S;\S\S\S\R>S \R>S-S\R:4SAjj5r5\ R9SB5 \"\ SCS 5S \R:S\R>S\,\R:\R:44SDj5r6\"\ SCS5S \R:S\R>S\,\R:\R:44SEj5r7\ R9SF5 \"\ SGSH5S \R:S\R>S\,\R:\R:44SIj5r8\ R9SJ5 \"\ SKS 5S \R:S\R>S\,\R:\R:44SLj5r9\"\ SKS5S \R:S\R>S\,\R:\R:44SMj5r:SNr;\ R9SO5 \"\ SPS 5S \R:S9\R:S:\R:S\S\S\R>4 SQj5r<\"\ SPS5S \R:S9\R:S:\R:S\S\S\R>4 SRj5r=\ R9SS5 \"\ STS 5\R|4S \R:S9\R:S:\R:S\S\S\R>SU\R>4SVjj5r?\"\ STS5\R|4S \R:S9\R:S:\R:S\S\S\R>SU\R>4SWjj5r@\ R9SX5 \"\ SYS 5SmS \R:S9\R:S:\R:S\S\S\R>4 S[jj5rA\"\ SYS5SmS \R:S9\R:S:\R:S\S\S\R>4 S\jj5rB\ R9S]5 \"\ S^S 5SZ\R|4S_\R:S9\R:S:\R:S-S\S\S\R>S`\SU\R>4Sajj5rC\ R9Sb5 "ScSd\RR5rF\"\ SeSf5S \R:S9\R:S:\R:S;\S\S\S\R:4Sgj5rG\"\ SeS5S \R:S9\R:S:\R:S;\S\S\S\R:4Shj5rH\ R9Si5 \"\ SjS 5S \R:S\R>S\R:4Skj5rI\"\ SjS5S \R:S\R>S\R:4Slj5rJgs snfs snf)nN)_unsqueeze_multiple)determine_qparamsvalidate_qmin_qmax)implLibraryquantized_decomposedDEFcU[;a[SU35e[Uup4X:a[SUSU35eX:a[SUSU35eg)NzUnsupported dtype: z9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: )_DTYPE_TO_QVALUE_BOUNDS ValueErrorAssertionError) quant_min quant_maxdtypequant_min_lower_boundquant_max_upper_bounds ^/home/james-whalen/.local/lib/python3.13/site-packages/torch/ao/quantization/fx/_decomposed.py_quant_min_max_bounds_checkrs ++.ug6773J53Q0( &&;% Tensorquantize_per_tensorCompositeExplicitAutogradinputscale zero_pointrrrreturncUR[R[R4;aUR [R 5nUR[R :wa[ SUR35e[X4U5 SU- n[R"[R"X-5U-X45R U5$)aAffine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead Tensorzquantize_per_tensor.tensorcUR5S:wa[SUR535eUR5S:wa[SUR535e[UUR5UR5UUU5$zAffine quantization for the Tensor using the same quantization parameters to map from floating point to quantized values Same as `quantize_per_tensor` but scale and zero_point are Scalar Tensor instead of scalar values >Expecting zero_point tensor to be one element, but received : 9Expecting scale tensor to be one element, but received : numelr ritemr+s rquantize_per_tensor_tensorr5ps QLZM]M]M_L` a   {{}G  W       rcUR[R[R4;aUR [R 5nUR 5S:wa[SUR 535eUR 5S:wa[SUR 535eUR[R :wa[SUR35e[R"XS9$)Nr/r0r1rr)) rrr r!r"r#r3r r*r+s rquantize_per_tensor_tensor_metar7s {{u}}enn55'QLZM]M]M_L` a   {{}G  W   {{emm#J5;;- X     E //rzquantize_per_tensor.tensor2(Tensor input, Tensor scale, Tensor zero_point, Tensor quant_min, Tensor quant_max, ScalarType dtype) -> Tensorzquantize_per_tensor.tensor2cRUR5S:wa[SUR535eUR5S:wa[SUR535e[UUR5UR5UR5UR5U5$r.r2r+s rquantize_per_tensor_tensor2r9s QLZM]M]M_L` a   {{}G  W       rc"[UUUUUU5$N)r7r+s r quantize_per_tensor_tensor2_metar<s# +    rzdequantize_per_tensor(Tensor input, float scale, int zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_tensor out_dtyper?cURU:wa[SUSUR35eUc[RnU[;aUR U5U- U-$[ SU35e)aAffine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per tensor quantized Tensor if combined with quantization parameters in the argument of this function (scale/zero_point) scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_min (int): minimum quantized value for input Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for input Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): dtype for input Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor Expecting input to have dtype: , but got ,Unsupported dtype in dequantize_per_tensor: )rr rr#r r"r rrrrrrr?s rr=r=szJ {{e-eWJu{{m L  MM  ''#j0E99GwOPPrcPUc[Rn[R"XS9$Nr))rr#r*rDs rdequantize_per_tensor_metarG&s$MM   E 33rzdequantize_per_tensor.tensor(Tensor input, Tensor scale, Tensor zero_point, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensorzdequantize_per_tensor.tensorc UR5S:wa[SUR535eUR5S:wa[SUR535e[UUR5UR5UUUUS9$zAffine dequantization for the Tensor using the same quantization parameters to map from quantized values to floating point values Same as `dequantize_per_tensor` but scale and zero_point are Scalar Tensor instead of scalar values r/r0r1r>r3r r=r4rDs rdequantize_per_tensor_tensorrK<s(QLZM]M]M_L` a   {{}G  W   !    rcUc[RnUR5S:wa[SUR535eUR5S:wa[SUR535eURU:wa[SUSUR35eU[ ;a[R "XS9$[SU35e)Nr/r0r1rArBr)rC)rr#r3r rr r*r rDs r!dequantize_per_tensor_tensor_metarMcsMM QLZM]M]M_L` a   {{}G  W   {{e-eWJu{{m L   ''77GwOPPrzdequantize_per_tensor.tensor2(Tensor input, Tensor scale, Tensor zero_point, Tensor quant_min, Tensor quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensorzdequantize_per_tensor.tensor2c PUR5S:wa[SUR535eUR5S:wa[SUR535e[UUR5UR5UR5UR5UUS9$rIrJrDs rdequantize_per_tensor_tensor2rOs(QLZM]M]M_L` a   {{}G  W   !    rc [XX#XEUS9$)Nr>)rMrDs r"dequantize_per_tensor_tensor2_metarQs - jY rzrchoose_qparams.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams.tensorqminqmaxepsc UR[R[R[R4;a[ SUR35eU[ ;a#[ S[ R5SU35e[X5 [R"U5upV[UUUUU[R"U/5SS9$)3Given an input Tensor, derive the per tensor affine quantization parameter (scale and zero_point) for target quantized Tensor from the Tensor Args: input (torch.Tensor): floating point input Tensor quant_min (int): minimum quantized value for target quantized Tensor quant_max (int): maximum quantized value for target quantized Tensor dtype (torch.dtype): dtype for target quantized Tensor Returns: scale (float): quantization parameter for the target quantized Tensor zero_point (int): quantization parameter for the target quantized Tensor CExpecting input to have dtype torch.float32/16/b16, but got dtype: $Expecting target dtype to be one of , but got: F)has_customized_qrange) rrr#r r!r r keysraminmaxrTensorrrRrSrTrmin_valmax_vals rchoose_qparams_tensorras" {{    QRWR]R]Q^ _   ++23J3O3O3Q2RR]^c]d e  t"}}U+G     cU# rz|choose_qparams_symmetric.tensor(Tensor input, int quant_min, int quant_max, float eps, ScalarType dtype) -> (Tensor, Tensor)zchoose_qparams_symmetric.tensorc UR[R[R[R4;a[ SUR35eU[ ;a#[ S[ R5SU35e[X5 [R"U5upV[UUUUU[R"U/5S[RS9$)rVrWrXrYF)rZqscheme) rrr#r r!r r r[rr\rr]per_tensor_symmetricr^s rchoose_qparams_symmetric_tensorres* {{    QRWR]R]Q^ _   ++23J3O3O3Q2RR]^c]d e  t"}}U+G     cU#**  rcUR[R[R[R4;a[ SUR35eX:a[ SUSU35e[R "S[RURS9[R "S[RURS94$)NrWzCExpecting quant_min to be smaller than quant_max but received min: z max: r/rdevice) rrr#r r!r emptydoublerhint64rrrrTrs rchoose_qparams_tensor_metarm(s {{    QRWR]R]Q^ _  QR[Q\\bclbm n   ;;q U\\ BEKK U\\E rc[R"S[RURS9[R"S[RURS94$)Nr/rg)rrirjrhrkrls r$choose_qparams_symmetric_tensor_metaro=sA ;;q U\\ BEKK U\\E rc[[UR555nSX!'XS'UR[ U55nX24$)Nr)listrangedimpermutetuple)xaxis new_axis_listys r_permute_to_axis_zerorzGsBquuw(MM! % &'A rzquantize_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max, ScalarType dtype) -> Tensorquantize_per_channelscales zero_pointsrwcUR[R[R4;aUR [R 5nUR[R :wa[ SUR35eX0R5:a[ SUR535e[XEU5 [X5upS/UR5-nURSUS'URU5nURU5n[R"[R"USU- -5U-XE5n U R[U55n U R U5$)a<Affine per channel quantization for the Tensor using the same quantization parameters for each channel/axis to map from floating point to quantized values Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_point (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead rExpecting axis to be < r/rr)rrr r!r"r#r rsrrzshapeviewr$r%rtru) rr|r}rwrrrpermute_axis_list new_shaperesouts rr{r{Us'6 {{u}}enn55' {{emm#J5;;- X   yy{6uyy{mDEE e<4UAEeiik!I<<?IaL [[ #F""9-K ++ ES6\*+k99 C ++e-. /C 66%=rcUR[R[R4;aUR [R 5nUR[R :wa[ SUR35eX0R5:a[ SUR535e[XEU5 [R"XS9$)Nrrr)) rrr r!r"r#r rsrr*)rr|r}rwrrrs rquantize_per_channel_metars {{u}}enn55' {{emm#J5;;- X   yy{6uyy{mDEE e<   E //rzdequantize_per_channel(Tensor input, Tensor scales, Tensor? zero_points, int axis, int quant_min, int quant_max, ScalarType dtype, *, ScalarType? out_dtype=None) -> Tensordequantize_per_channelcURU:wa[SUSUR35eUc[RnX0R 5:a[SUR 535e[ XEU5 [ X5upS/UR 5-n URSU S'URU 5nUbXRU 5- U-n OX-n U RU5n U R[U55n U $)aOAffine per channel dequantization for the Tensor using the same quantization parameters for each channel/axis to map from quantized values to floating point values Args: input (torch.Tensor): Tensor with dtype matching `dtype` argument, e.g. (`torch.uint8`), it is a per channel quantized Tensor if combined with quantization parameter in the argument of this function (scales/zero_points/axis) scales (torch.Tensor): a list of scale quantization parameter for affine quantization, one per channel zero_points (torch.Tensor): a list of zero_point quantization parameter for affine quantization, one per channel quant_min (int): minimum quantized value for output Tensor (not used in computation, reserved for pattern matching) quant_max (int): maximum quantized value for output Tensor (not used in computation, reserved for pattern matching) dtype (torch.dtype): requested dtype for output Tensor (not used in computation, reserved for pattern matching) out_dtype (torch.dtype?): optional dtype for output Tensor Returns: dequantized float32 Tensor rA, but got dtype: rr/r) rr rr#rsrrzrrr"rtru) rr|r}rwrrrr?rrrrs rrrsP {{e-eW4Eekk] S  MM  yy{6uyy{mDEE e<4UAEeiik!I<<?IaL [[ #F'' 22f<n && C ++e-. /C JrcURU:wa[SUSUR35eUc[RnX0R 5:a[SUR 535e[ XEU5 [R "XS9$)NzExpecting input to have dtype rrr))rr rr#rsrr*)rr|r}rwrrrr?s rdequantize_per_channel_metars {{e,UG3DU[[M R  MM  yy{6uyy{mDEE e<   E 33rzLchoose_qparams_per_token(Tensor input, ScalarType dtype) -> (Tensor, Tensor)choose_qparams_per_tokenchUR5RSSS9nUR[R:XaUR 5nU[R :XaSnSUS- -S- nO[SU35eURSS 9RU5n[R"U5nX%4$) Choose quantization parameters for per token quantization. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32/float16 Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor Returns: scales and zero_points, both float32 Tensors Trskeepdimr/z/unsupported dtype in choose_qparams_per_token: gh㈵>min) absamaxrrr floatint8 Exceptionr$div zeros_like)rrr|n_bitsrr}s rrrs,YY[  "d  3F ||u}}$ LLN   &1*%) =eW E  \\d\ # ' ' 2F""6*K  rc[URSS5S/-n[R"U[RUR S9[R"U[R UR S94$Nrr/rgrqrrrirjrhrkrrsizes rchoose_qparams_per_token_metar-]  CR !QC 'D ;;t5<< Eu{{ EKK H rz]_choose_qparams_per_token_asymmetric_impl(Tensor input, ScalarType dtype) -> (Tensor, Tensor))_choose_qparams_per_token_asymmetric_implCompositeImplicitAutogradcSup#[R"USSS9n[R"USSS9n[R"U[R"U55n[R "U[R"U55n[R "[R5RnXv- [X2- 5- n U RUS9n Xi- n Xy- n X*-n X;-n [R"X-S:X*- X;- 5n[R"XU5R5nU R5UR54$)r)irTrrr)raminrrrmaxfinfor#rTrr$wherer%r"float64rk)rrrRrSr_r` min_val_neg max_val_posrTr descaled_min descaled_maxzero_point_from_min_errorzero_point_from_max_errorrs rrrAs,,JDjjB5GjjB5G))GU%5%5g%>?K))GU%5%5g%>?K ++emm $ ( (C &% *< >>rzWchoose_qparams_per_token_asymmetric(Tensor input, ScalarType dtype) -> (Tensor, Tensor)#choose_qparams_per_token_asymmetricc[X5$r;)rrrs rrrvs 5U BBrc[URSS5S/-n[R"U[RUR S9[R"U[R UR S94$rrrs r(choose_qparams_per_token_asymmetric_metarrrc,[R"[UR55SS5nX1R 5:wa[ SUSUR535eX2R 5:wa[ SUSUR535eg)Nrz num_tokens: z scales: z zero_points: )mathprodrqrr3r )rr|r} num_tokenss r!_per_token_quant_qparam_dim_checkrs4 -cr23J\\^#|J Tensorquantize_per_tokenc[X4U5 [XU5 URSU- 5RU5R 5R X45R U5nU$)aPer token quantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per token affine quantization zero_points (int32 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r)rrmuladdr%r$r"rr|r}rrrs rrrsX6 e<%e[A #, [   y $ E   LrcB[X4U5 [R"XS9$rFrrr*rs rquantize_per_token_metars e<   E //rzdequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensordequantize_per_token output_dtypec4X- nX-nURU5$)aPer token dequantization for the Tensor using the quantization parameters to map from floating point to quantized values. This means for a N dimension Tensor (M1, M2, ...Mn, N), we calculate scales/zero_points for each N elements and quantize every N elements with the same quantization parameter. The dimension for scales/zero_points will be (M1 * M2 ... * Mn) Args: input (torch.Tensor): quantized Tensor (uint8, int8 etc.) scales (float64 torch.Tensor): quantization parameter for per token affine quantization zero_points (int64 torch.Tensor): quantization parameter for per token affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` )r"rr|r}rrrrs rrrs"8  E NE 88L !!rcB[X4U5 [R"XS9$rFrrs rdequantize_per_token_metars e<   E 66rzquantize_per_channel_group(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size) -> Tensorquantize_per_channel_groupcUS::a [S5eX`RS:a"URSS:XaURSnURSU-S:wa [S5eUR5S:wa [S5eURSU5n[R "U5R 5S:wa [S5eURSS5nURSS5nURS U- 5RU5R5RX45RU5RU5nU$) Nr/group_size must be > 1rr/input.shape[-1] must be divisible by group_sizerinput must be 2-dimensionalzto_quant must not contain NaNsr) r rrsreshaperisnansumrrr%clamp_r" reshape_as) rr|r}rrr group_sizeto_quant input_int8s rrrs$Q566KKO# R(8A(=[[_  {{2#q(NOO yy{a:;;}}R,H {{8  "a'=>> ^^B "F%%b!,K  S6\" [    % E E   rc4US::a [S5eX`RS:a"URSS:XaURSnURSU-S:wa [S5eUR5S:wa [S5e[R"XS9$) aGroupwise quantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): original float32 or bfloat16 Tensor scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for output Tensor quant_max (int): maximum quantized value for output Tensor dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor Returns: Tensor with requested dtype (e.g. torch.uint8), note the quantization parameters are not stored in the Tensor, we are storing them in function arguments instead r/rrrrrrr))r rrsrr*)rr|r}rrrrs rquantize_per_channel_group_metar>s8Q566KKO# R(8A(=[[_  {{2#q(NOO yy{a:;;   E //rzdequantize_per_channel_group(Tensor input, Tensor scales, Tensor? zero_points, int quant_min, int quant_max, ScalarType dtype, int group_size, ScalarType output_dtype) -> Tensordequantize_per_channel_groupw_int8rc\US::a [S5eX`RS:a"URSS:XaURSnURSU-S:wa [S5eUR5S:wa [S5eURSU5nURSS5nUbURSS5n O.[R "/[R URS9n URU 5RU5RU5RU5n U $) aGroupwise dequantization within each channel for an 2-d Tensor using the quantization parameters to map from floating point to quantized values. This means for each row of a 2-d Tensor (M, N), we calculate scales/zero_points for each `group_size` elements and quantize every `group_size` elements with the same quantization parameter. The dimension for scales/zero_points will be (M * ceil(N, group_size),) Args: input (torch.Tensor): quantized Tensor (uint8/int8 etc.) scales (float32 torch.Tensor): quantization parameter for per channel group affine quantization zero_points (int32 torch.Tensor): quantization parameter for per channel group affine quantization quant_min (int): minimum quantized value for input Tensor quant_max (int): maximum quantized value for input Tensor dtype (torch.dtype): dtype (e.g. torch.uint8) for input Tensor output_dtype (torch.dtype): dtype (e.g. torch.float32) for output Tensor Returns: dequantized Tensor with dtype `output_dtype` r/rrrz0w_int8.shape[-1] must be divisible by group_sizerzw_int8 must be 2-dimensionalrg) r rrsrrzerosint32rhsubrrr") rr|r}rrrrrw_int8_groupedzpw_dqs rrrms DQ566LL$$b)9Q)>\\"%  ||B*$)OPP zz|q;<<^^B 3N ^^B "F  Q ' [[5;;v}} E   b ! % %f - 8 8 @ C CL QD Krzyfake_quant_per_channel(Tensor input, Tensor scales, Tensor zero_points, int axis, int quant_min, int quant_max) -> Tensorc4\rSrSr\S5r\S5rSrg)FakeQuantPerChannelic.UR[R:waUR[R5nUR[R:waUR[R5nUR[R:wa[ SUR35eXAR 5:a[ SUR 535e[[U55[[US-UR55-n[X'5n[X75n [R"USU- -5U -n [R"XU5U - U-n [R"X:X:*5n URU 5 U $)Nrrr/r)rrr#r"rr rsrqrrndimrr%r$ logical_andsave_for_backward) ctxrr|r}rwrrbroadcast_dimsunsqueeze_scalesunsqueeze_zero_pointstemprmasks rforwardFakeQuantPerChannel.forwardsC <<5== (YYu}}-F    +%..5K ;;%-- ' Nu{{m\  99;  #:599;-!HI IeDk*T%q%**2M-NN.vF 3K P{{5C*:$:; Tensorzconvert_element_type.no_fusech[RRRR X5$r;)ropsprimsconvert_element_typedefaultrs rr r s# 99?? / / 7 7 EErc*[R"XS9$rFrrs rconvert_element_type_metars   E //r)r)Krr torch._refsrtorch.ao.quantization.utilsrr torch.libraryrrquantized_decomposed_libuint8ruint16int16r_INTEGER_DTYPES float8_e5m2 float8_e4m3fn _FLOAT_DTYPESiinforrr updateintrrdefiner]rrrr,r5r7r9r<r=rGrKrMrOrQrurarermrorzr{rrrrrrrrrrrr#rrrrrautogradFunctionrrrr r)ks0rr!s +M' ##95A;; ELL%++u{{S""E$7$78 :I9HA AEKKN..//DQRMqU[[^   #ekk!n&8&8"9::MR  $@  57RS" <<" "" "  " ;; " \\"T"J 5v>0 <<0 00 0  0 ;; 0 \\0?0"@ :  =vF << << ||  ||  ;;  \\G,_  79TU%)0Q <<0Q 0Q0Q 0Q  0Q ;; 0Q{{T!0Q \\0QV0Qf 7@%) 4 << 4 << 4  4 4  4 ;; 4{{T! 4 \\ 4A 4_ "%) << <<     ;; {{T! \\  D >G%)Q <<Q <<Q Q Q  Q ;; Q{{T!Q \\QHQ>e #%) << << ||  ||  ;; {{T! \\  D ?H%) {{T!  \\ I 7  79TU( <<("(*-(49(BG++( 5<< %&(V(V7 % ( <<("(*-(49(BG++( 5<< %&(  (V 7@ <<$'47>CLQKK 5<< %&A( A6J <<$'47>CLQKK 5<< %&K@  68ST. <<. LL..  .  .  . ;;. \\.U.b 6?0 <<0 LL00  0  0  0 ;;0 \\0@02_  8:UV%)= <<= LL=$=  =  =  = ;;={{T!= \\=W=@ 8&A%)4 <<4 LL4$4  4  4  4 ;;4{{T!4 \\4B4.R   <<  ;;  5<< %&   F   << ;; 5<< %&  c / (? <<(? ;;(? 5<< %&(?  (?V] ) C <<C ;;C 5<< %&C  C)   << ;; 5<< %&   @  46QR# <<# LL## #  # ;; #S#L 4f= 0 << 0 LL 0 0 0  0 ;; 0> 0Y  68ST!& " <<" LL"" "  " ;; "++"U"B 6?!&  7 << 7 LL 7 7 7  7 ;; 7++ 7@ 7A:G0U\\0%++0%,,0H0E'Ss=Aw"A w'