hl}z %SSKrSSKJr SSKJrJrJrJr SSKrSSK J r SSK J r J r JrJr SSKJr \"S\405r\\S'\R*"\5r\R0\R2\R4\R61r\R:S\R<S \R>S \R@S \RBS \RDS \RFS0r$\RJS\RLS\RNS\RPS0r)\*\\RV\4\,\-\-44\S'\)R]\$5 S\RVS\/4Sjr0Sr1Sr2SSSSSS\Rf4S\RhS\RhS\ S\,\-S4S\RVS \\-S!\\-S"\\5S#\\RVS$\\RVS%\/S&\\S\,\Rh\Rh44S'jjr6S;S(\\RhS\7S\\-S\RVS \\\-\5\/4S!\\\-\5\/4S"\\5S#\\RVS$\\RVS%\/S&\\7S\\RhS\\RhS\,\Rh\Rh44S)jjr8\Rr"5SS\Rf4S(\RhS\,\-S4S*\RhS+\\RhS,\RVS \\\-\54S!\\\-\54S&\\S\Rh4S-jj5r:SS\RfRv4S(\RhS\<\-S*\RhS+\\RhS,\RVS \\\-\5\/4S!\\\-\5\/4S&\\7S\Rh4S.jjr=\RfRv4S(\RhS\<\-S*\RhS+\\RhS \\-\54S!\\-\54S&\\7S\Rh4S/jjr>SS\Rf4\R~S0.S(\RhS\,\-S4S*\RhS+\\RhS1\RVS \\\-\54S!\\\-\54S&\S,\RVS\Rh4S2jjjr@SS\RfRv\R~4S(\RhS\<\-S*\RhS+\\RhS1\RVS \\\-\5\/4S!\\\-\5\/4S&\\7S,\RVS\Rh4S3jjrA\RfRv\R~4S(\RhS\<\-S*\RhS+\\RhS \\-\54S!\\-\54S&\\7S,\RVS\Rh4S4jjrB"S5S6\ 5rC"S7S8\ 5rD"S9S:\ 5rEg)<N)ABCMeta)AnyListOptionalUnion) Granularity)AffineQuantizedObserverBase MappingType TorchAODTypeZeroPointDomain)get_block_sizeABC)r)r)r)r)r)r?)r)r)ir)ii)ii_DTYPE_TO_QVALUE_BOUNDSdtypereturnc[R[R[R[R1nX;$N)torch float8_e4m3fnfloat8_e4m3fnuz float8_e5m2float8_e5m2fnuz)r fp8_typess h/home/james-whalen/.local/lib/python3.13/site-packages/torchao/quantization/pt2e/_affine_quantization.py_is_float8_typer#:s8     I  c>U[;a@[R"U5R[R"U5RpCO#U[ ;a[ SU35e[ Uup4UcUnUcUnX:d SUSU35eX$::d SUSU35eX4$)zGet quant_min and quant_max args based on dtype and also verify that they are within the range of possible quant_min/quant_max for dtype zUnsupported dtype: z9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: ) FP8_TYPESrfinfominmaxr ValueError)r quant_min quant_maxquant_min_lower_boundquant_max_upper_bounds r"_get_and_check_qmin_qmaxr/Es   KK  " " KK  " " 5 - -.ug6777Nu7U4) )  - ""7!8 YK Q -  - ""7!8 YK Q -  r$c [U5[U5:Xde/n/nSn[[U55HnXX:warXS:ajXX-S:XdSUSXSUSX35eURXX-5 URX5 URUS-5 US- nMURX5 XS:waURU5 US- nM X#4$)aGiven block_size and input size find the parameters for reduction: Output: shape_for_reduction: the shape we use to `view` input to prepare it for reduction reduction_dims: the dims we'll do reduction over Example:: Input: block_size: (3, 3, 2, 10) input_size: (3, 3, 10, 10) Output: shape_for_reduction: (3, 3, 5, 2, 10) reduction_dim: [0, 1, 3, 4] rrzExpecting input size at z dimension: z" to be divisible by block_size at )lenrangeappend) block_size input_sizeshape_for_reductionreduction_dimscur_dimis r"_get_reduction_paramsr;ds z?c*o -- -NG 3z? # =JM )jma.?=:=0A5 *1#\=/!CA3lS]S`Rac 5 & &z} 'E F  & &z} 5  ! !'A+ . qLG & &z} 5}!%%g. qLG%$&  ..r$Tmin_valmax_val mapping_typer5. target_dtyper+r,eps scale_dtypezero_point_dtype preserve_zerozero_point_domainc n[SURUUUUUUU U U bU RUU5 $SUU5 $)aLA variant of :func:`~torchao.quantization.quant_primitives.choose_qparams_affine` operator that pass in min_val and max_val directly instead of deriving these from a single input. This is used for observers in static quantization where min_val and max_val may be obtained through tracking all the data in calibration data set. Args: Mostly same as :func:`~torchao.quantization.quant_primitives.choose_qparams_affine`. with one difference: instead of passing in `input` Tensor and use that to calculate min_val/max_val and then scale/zero_point, we pass in min_val/max_val directly N)_choose_qparams_affinename) r<r=r>r5r?r+r,r@rArBrCrDs r""choose_qparams_affine_with_min_maxrHsa0 "  "3"? FJ r$inputc \ [X4U5upEU[RR[RR[R R4;d SU35eU[ ;a(U[RR:Xd SU35eUbUc URnUc URnUc*[R"UR5Rn[U5UR5:XdSUR5SU35e[X R55upURU 5n[R "XSS9n [R""XSS9n OyU bU cS5eU RU R:XdS 5eUc U RnUc U RnUc*[R"U R5RnU (aW[R$"U [R&"U 55n[R("U [R&"U 55nOU nU nU[RR:XdU[RR:XGaU[RR:Xa,[R("U*U5nU[+XT- 5S - - nOYU[RR:XdeU[+U5- nU[+U5- nUU:n[R,"UUU5nU (d [/S 5eU b)U [0R2R:wa [/S 5e[R4"UUS 9n[R6"U[9XT-S-S - 55nOU[R R:XdeUU- [+XT- 5- n[R4"UUS 9nU [0R:R:XaSnOrU (a4U[R<"UU- 5- n[R4"UXE5nO7U [0R>R:XdS5eXT-S-S - nUUU--nUbURAUS9nURAUS9U4$)aWop definition that has compatible signatures with custom op library The op does the following: 1. figure out the dimension for reduction based on block_size 2. find min_val/max_val based on the dimension for reduction 3. calculate quantization parameters based on min_val/max_val based on args like `preserve_zero` and `zero_point_domain` zUnsupported mapping type: zr5r?r+r,r@rArBrCrDr<r=r7r8 min_val_neg max_val_posscalesminsmaxmask zero_point mid_points r"rFrFsI.4LYWI ""--22## 3 $L>2 3  y {44999 J<. Y 9   ++K  #${{  ;++ekk*..C:%))+- UYY[M | D -/D  / + ./**UF**UF"w': c :}} - k -  !--K  #&}}  ;++gmm,00Cii)9)9')BC ii)9)9')BC     --222 ;@@EE E ;0055 5))[L+>K5)>#?!#CDE;#H#H#M#MM MMy!11Dy!11D$;DKKdD1ET   )!_%8%8%=%==f  Es+__UC1F1Ja0O,PQ {55:::::{*eI4I.JJ Es+  4 4 9 9 9J&)II "[[YJ (O,A,A,F,FFNF'2Q6!; (59+<< ]])9]: 88+8 & 22r$rdrh output_dtypec L[UUUUUUUUbUR5$S5$)a Args: input (torch.Tensor): original float32, float16 or bfloat16 Tensor block_size: (Tuple[int, ...]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization output_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for output Tensor, if not specified, it will be derived from dtype quant_max (Optional[int]): maximum quantized value for output Tensor, if not specified, it will be derived from dtype zero_point_domain (ZeroPointDomain): the domain that zero_point is in, should be either integer or float if zero_point is in integer domain, zero point is added to the quantized integer value during quantization if zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) value during quantization default is ZeroPointDomain.INT Note: How can block_size represent different granularities? let's say we have a Tensor of size: (3, 3, 10, 10), here is the table showing how block_size represents different granularities: granularity type | block_size per_tensor | (3, 3, 10, 10) per_axis (axis=0) | (1, 3, 10, 10) per_axis (axis=1) | (3, 1, 10, 10) per_group (groupsize=2) | (3, 3, 10, 2) per_group (groupsize=2) for axis = 3 | (3, 3, 2, 10) Output: quantized tensor with requested dtype N)_quantize_affinerGrIr5rdrhrjr+r,rDs r"quantize_affinern:sGZ   "3"?  FJ  r$c [XEU5upVU[;a[Rn[ UUUUUUU5R U5$)aop definition that has compatible signatures with custom op library Note: zero_point_domain is optional specifies how we quantize the floating point to quantized data: INT: quantized_val = (float_val / scale) (integer) + zero_point (integer) FLOAT: quantized_val = (float_val - (zero_point (float) - scale * mid_point)) / scale None: quantized_val = (float_val / scale) | this is primarily used for floatx quantization Where we do not want to round values to nearest integer and instead scale and cast. )r/_SUB_BYTE_UINT_BOUNDSruint8_quantize_affine_no_dtype_castrarms r"rlrlssU&4LYWI,,{{ )    br$cUR[R[R[R4;dSUR35e[ U5UR 5:XdSUR 5SU35e[XR55upxURn URU5nUn UHn SX'M URU 5nUbURU 5nU[RR:Xa5[R"[R"USU- -5U-XE5n OU[R R:Xa<UbS5e[R"[R"USU- -5XE5n OUc2UbS5e[R"XR#5-XE5n OaU[R$R:XdeXT-S-S- n X2U -- n[R"[R"X- U- 5XE5n U RU 5n U $) a? The op does the following: 1. figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. quantize the input based on the quantization parameters scale and zero_point and args like zero_point_domain 3. reshape the quantized result to origianl shape zUnsupported input dtype: rKrLrg?8zero_point should be None when zero_point_domain is NONE8zero_point should be None when zero_point_domain is Noner1)rrfloat32float16bfloat16r2rNr;rSshaperTr rZrGr[r_r^ reciprocalr`)rIr5rdrhr+r,rDr7r8original_shapeshape_after_reductionr:quantrir<s r"rrrrs*$ ;;    1 #5;;-0 1  z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E__%:; O//444 KKu- . ;Y  o2277 7! F ! EKKu(=> U  "! F ! E$4$4$66 M O$9$9$>$>>>>*Q.!3 y00 KKE1 2I  JJ~ &E Lr$rj input_dtypec H[UUUUUUUUbURUS9 $SUS9 $)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 zero_point_domain (ZeroPointDomain): the domain that zero_point is in, should be either integer or float if zero_point is in integer domain, zero point is added to the quantized integer value during quantization if zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) value during quantization default is ZeroPointDomain.INT Output: dequantized Tensor, with requested dtype or fp32 Nr~)_dequantize_affinerG rIr5rdrhrr+r,rDrjs r"dequantize_affinersMD   "3"?!  FJ!  r$c U[;a'URU:XdSUSUR35eU[R[R[R 4;d SU35e[ XEU5upV[UUUUUUUU5$)zCop definition that has compatible signatures with custom op libraryz Expected: z, got: zUnsupported output dtype: )rprrrvrwrxr/!_dequantize_affine_no_dtype_checkrs r"rrs//{{k)  WU[[M : )     3 $L>2 3  4KIVI ,    r$c8[U5UR5:XdSUR5SU35e[XR55upURn UR U5nUn U Hn SX'M UR U 5nUbUR U 5nU[ RR:XaXUR[RSS9n Ub!XR[R5- n U RU5n X-n OU[ RR:Xa UbS5eURU5n X-n OUcNUbS5e[UR5(dSUR35eURU5n X-n ORU[ RR:Xd S U35eXT-S-S - nX- n U RU5n X-n UbX- n U R U 5RU5$) aThis function converts AQT tensors to their high precision floating point representation The op does the following: 1. figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. dequantize the input based on the quantization parameters scale and zero_point and args like zero_point_domain 3. reshape the quantized result to origianl shape and change dtype to the output_dtype rKrLrT)copyrtruzNdequantiztion with no zero point domain is only supported with FP8 types, got zUnexpected zero point domain: r1)r2rNr;rSryrTr rZrGrarint32r^r#rr`)rIr5rdrhr+r,rDrjr7r8r{r|r:dequantris r"rr)s1$ z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E__%:; O//444((5;;T(2  ! ekk ::G**\*/ o2277 7! F !((<(/  "! F !u{{++ \]b]h]h\i j +((<(/ O$9$9$>$>> ,->,? @ >*Q.!3 #**\*  !  !G << ' * *< 88r$cv\rSrSrS\R 4SjrS\\R \R 44SjrSr g)AffineQuantizedMinMaxObserverisrIc4UR5S:XaU$UR5nURUlURcS5e[ UR UR5Ul[URUR55up4URU5n[R"X$SS9n[R"X$SS9n[US5(a[US5(dXPlX`lU$URR UR :Xd+SURR SUR 35eUR R UR :Xd+S UR R S UR 35e[R""URU5n[R$"UR U5nURR'U5 UR R'U5 U$ Nrzgranularity is NoneFrMr<r=z=Can't update existing min_val - shape mismatch, self.min_val:z != min_val:z=Can't update existing max_val - shape mismatch, self.max_val z != max_val:)numeldetachroriginal_dtype granularityr ryr5r;rSrTrrUrVhasattrr<r=r(r)copy_selfrIinput_detachedr7r8r<r=s r"forward%AffineQuantizedMinMaxObserver.forwardts ;;=A L,22+B-BB+()=)=t?O?OP.C OO^002/ +(,,-@A**^O**^OtY''wtY/G/G"L"L <<%%6 OPTP\P\PbPbOccopwp}p}o~ 6<<%%6 OPTP\P\PbPbOccopwp}p}o~ 6ii g6Gii g6G LL  w ' LL  w ' r$rc\[US5(a[US5(dS5e[URURUR/UR UR URURURURURUR5 $Nr<r=zhExpecting the observer has min_val and max_val, please run the observer before calling calculate_qparams rrHr<r=r>r?r+r,r@rArBrCrDrs r"calculate_qparams/AffineQuantizedMinMaxObserver.calculate_qparamsstY''GD),D,D v D2 LL LL        NN NN HH     ! !     " "  r$)r5r=r<rN) __name__ __module__ __qualname____firstlineno__rTensorrtupler__static_attributes__r$r"rrss1U\\@ 5u||)C#D r$rc@^\rSrSrSSSSSSSS\R 4 S\S\RS\ S \ \ S \ \ S \ \ S \ \RS \ \RS\ S\ \4U4SjjjrS\R4SjrS\\R\R44SjrSrU=r$)*AffineQuantizedMovingAverageMinMaxObserverig{Gz?NFTr>r?rr+r,r@rArBrCrDc >XlX@lU(a(URS:wa[SUR35e[TU]UUUUUUU U U U S9 g)Nrz[MovingAverageMinMaxObserver doesn't support dynamic quantization for averaging constant of r>r?rr+r,r@rArBrCrD) is_dynamicaveraging_constantNotImplementedErrorsuper__init__)rr>r?rrr+r,r@rrArBrCrDkwargs __class__s r"r3AffineQuantizedMovingAverageMinMaxObserver.__init__sv"%"4 $11Q6%))-)@)@(AC  %%##-'/  r$rIcPUR5S:XaU$UR5nURUlURcS5e[ UR UR5Ul[URUR55up4URU5n[R"X$SS9n[R"X$SS9n[US5(a[US5(dXPlX`lU$URR UR :Xd+SURR SUR 35eUR R UR :Xd+S UR R S UR 35eURUR"XPR- --nUR UR"X`R - --nURR%U5 UR R%U5 U$r)rrrrrr ryr5r;rSrTrrUrVrr<r=rrrs r"r2AffineQuantizedMovingAverageMinMaxObserver.forwards ;;=A L,22+B-BB+()=)=t?O?OP.C OO^002/ +(,,-@A**^O**^OtY''wtY/G/G"L"L <<%%6 OPTP\P\PbPbOccopwp}p}o~ 6<<%%6 OPTP\P\PbPbOccopwp}p}o~ 6llT%<%<,,@V%WWGllT%<%<,,@V%WWG LL  w ' LL  w ' r$rc\[US5(a[US5(dS5e[URURUR/UR UR URURURURURUR5 $rrrs r"rr?rr+r,r@rArBrCrDc >>Xpl[T U] UUUUUUUU U U S9 g)Nr)rrr)rr>r?rr+r,r@rrArBrCrDrrs r"r+AffineQuantizedPlaceholderObserver.__init__s: % %%##-'/  r$cr[URUR5UlURUlU$r)r ryrr5rr)rrIs r"r*AffineQuantizedPlaceholderObserver.forward%s+(d6F6FG#kk r$c[S5e)Nz>calculate_qparams should not be called for PlaceholderObserver) Exceptionrs r"r4AffineQuantizedPlaceholderObserver.calculate_qparams*s L  r$)r5rr)rrrrr rZr rrrrr]rXrrrrrrrs@r"rrs $(#'#-126"7F7J7J ! kk !  C=  C=  e_ ekk* #5;;/  $O4  >   r$r) NNNNNTrZNN)Floggingabcrtypingrrrrrtorchao.quantizationr"torchao.quantization.pt2e.observerr r r r torchao.quantization.utilsr objectr__annotations__ getLoggerrloggerrrrr r&uint1uint2uint3uint4uint5uint6uint7rprqint8int16rrdictrrr]updaterr#r/r;rZrrXrHstrrFno_gradrnrGlistrlrrrvrrrrrrrr$r"rs-- , 6 56)R (S(   8 $        KK KK KK KK KK KK KK  KK JJ  KK& KK& TeEKK$=>c3hOP 455;;4 >'/` $#)-.23B3F3F& \\& \\&&c3h & ++ & } &}& %&%++&&u{{+&& 0& 5<< %&&\4837)-.2',&*&*@3 ELL !@3@3S @3++ @3 c5$./0 @3 c5$./0 @3 %@3%++&@3u{{+@3@3 }@3ell #@3ell #@3 5<< %&@3F.2-13B3F3F5 <<5c3h5 <<5& 5 ++ 5 c5j)* 5c5j)*5 05 \\55|4837'6':':'?'?  << S   << &  ++  c5$./0 c5$./0  }  \\ T(7':':'?'?> <<>S > <<>& > S%Z > S%Z > }> \\>N.2-1)8)<)<,!& , <<,c3h, <<,& ,  , c5j)* ,c5j)*,',++, \\,j4837'6':':'?'? %   << S   << &    c5$./0 c5$./0  } ++  \\ T(7':':'?'? % G9 <<G9S G9 <<G9& G9 S%Z G9 S%Z G9 }G9++G9 \\G9T2 $?2 jZ 1LZ z( )D( r$