hSm(%SSKrSSKJrJr SSKJrJrJrJrJ r J r SSK r SSK J r JrJr SSKJrJr /SQr"SS\5r"S S \5r"S S \5r\ R.R1\\/5 \ R2\ R4\ R6\ R81r\ R<S \ R>S\ R@S\ RBS0r"\\ \ RF\4\ \$\$44\%S'\RLS\RNS\RPS\RRS\RTS\RVS\RXS\ R<S\ R>S\ R@S\ RBS0 r-\\ \ RF\4\ \$\$44\%S'0r.\\ \ RF\4\ \$\$44\%S'\RLS\RNS\RPS \RRS!\RTS"\RVS#\RXS$0r/\\ \ RF\4\ \$\$44\%S%'\ R`S&\ RbS'\ RdS(\ RfS)\ RhS*\ RjS+\ RlS,0r.\-Ro\ R`S\ RbS\ RdS\ RfS\ RhS\ RjS\ RlS05 \/Ro\ RpS\ RrS\ RtS \ RvS!\ RxS"\ RzS#\ R|S$05 \-Ro\ RpS\ RrS\ RtS\ RvS\ RxS\ RzS\ R|S05 \"Ro\.5 \"Ro\/5 \-R5\"R5:XdeS-r@\A"S5Vs/sH n\"U5PM snrB\ RRS.S/5rE\"\E5rF"S0S1\ RR5rI"S2S3\ RR5rJS4rKS5rL\ R"5SS6\ RS7\ \$S84S9\ RS:\\ RS;\ RFS<\\ \$\O4S=\\ \$\O4S>\ R4S?jj5rP\FSS6\ RS7\\$S9\ RS:\\ RS;\ RFS<\\ \$\O\Q4S=\\ \$\O\Q4S>\ R4S@jj5rRS6\ RS7\\$S9\ RS:\\ RS<\ \$\O4S=\ \$\O4S>\ R4SAjrSSS6\ RS7\\$S9\ RS:\\ RS;\ RFS<\\ \$\O\Q4S=\\ \$\O\Q4S>\ R4SBjjrTSS6\ RS7\ \$S84S9\ RS:\\ RS<\\ \$\O4S=\\ \$\O4S>\ R4SCjjrUSS6\ RS7\\$S9\ RS:\\ RS;\ RFS<\\ \$\O\Q4S=\\ \$\O\Q4S>\ R4SDjjrVSS6\ RS7\ \$S84S9\ RS:\\ RS<\\ \$\O4S=\\ \$\O4S>\ R4SEjjrWS\ RSF.S6\ RS7\ \$S84S9\ RS:\\ RSG\ RFS<\\ \$\O4S=\\ \$\O4S;\ RFS>\ R4SHjjjrY\FSS\ R4S6\ RS7\\$S9\ RS:\\ RSG\ RFS<\\ \$\O\Q4S=\\ \$\O\Q4S;\ RFS>\ R4SIjj5rZ\ R4S6\ RS7\\$S9\ RS:\\ RS<\ \$\O4S=\ \$\O4S;\ RFS>\ R4SJjjr[\ R4S6\ RS7\\$S9\ RS:\\ RS<\ \$\O4S=\ \$\O4S;\ RFS>\ R4SKjjr\S\ RSF.S6\ RS7\ \$S84S9\ RS:\\ RSG\ RFS<\\ \$\O4S=\\ \$\O4S;\ RFS>\ R4SLjjjr]\ R4S6\ RS7\\$S9\ RS:\\ RS<\ \$\O4S=\ \$\O4S;\ RFS>\ R4SMjjr^S\ RSF.S6\ RS7\ \$S84S9\ RS:\\ RSG\ RFS<\\ \$\O4S=\\ \$\O4S;\ RFS>\ R4SNjjjr_SS\R4S6\ RS7\ \$S84S9\ RS:\\ RSO\ RFS<\\ \$\O4S=\\ \$\O4SP\S>\ R4SQjjraSS\R4S6\ RS7\ \$S84S9\ RS:\\ RSO\ RFS<\\ \$\O4S=\\ \$\O4SP\S>\ \ R\ R44SRjjrbSS\R4S6\ RS7\ \$S84S9\ RS:\\ RSO\ RFS<\\ \$\O4S=\\ \$\O4SP\S>\ \ R\ R44SSjjrc\ R"5SSSS\ RB4S6\ RST\S7\ \$SU\ RFS<\\ \$\O4S=\\ \$\O4SV\\OSW\\ RFSX\\ RFS>\ \ R\ R44SYjj5rd\ R"5SS6\ RST\S7\ \$SU\ RFS<\\ \$\O4S=\\ \$\O4SV\\OSW\\ RFSX\\ RFS>\ \ R\ R44SZjj5reSS6\ RST\S7\ \$SU\ RFS<\\ \$\O\Q4S=\\ \$\O\Q4SV\\OSW\\ RFSX\\ RFS>\ \ R\ R44S[jjrfSSSSSS\\R4S]\ RS^\ RST\S7\ \$S84SU\ RFS<\\$S=\\$SV\\OSW\\ RFSX\\ RFS_\QSP\S>\ \ R\ R44S`jjrg\FSS6\\ RST\hS7\\$SU\ RFS<\\ \$\O\Q4S=\\ \$\O\Q4SV\\OSW\\ RFSX\\ RFS>\ \ R\ R44Sajj5riSb\ RSc\$Sd\$S>\ \ R\ R\ R44SejrjS6\\ RS7\\$SU\ RFS>\ \ R\ R\ R\ R44SfjrkS6\ RS7\\$SU\ RFSg\ RSh\ RSi\ RSj\ RS>\ R4SkjrlSS6\ RS7\\$SU\ RFSg\ RSh\ RSi\ RSj\ RS;\\ RFS>\ R4SljjrmSSb\ RSm\ RSn\ RSc\$Sd\$S;\\ RF4 SojjrnSp\ RSq\OSr\OS>\ R4Ssjro\ R"5SSSStSuSvSwSxS\Sy.4Sz\ RS9\ RS{\ RS|\qS}\$S~\ \ RFS4S\ \hS4S\QS\rS>\s4Sjj5rtS\$S\$S>\Q4SjruS\ RS9\ RS{\ RS\$S\ \\ \ R4S>\ 4 SjrwSSS\S\ RSStSt\t4 Sz\ RS\OSd\$S\QS}\$S\ RFS\hS\QS\QS\S>\s4Sjjry\ R"5SxStSS.S\ RS7\\$S\$S\$S\$S\QS\OS>\ \ R\ R44Sjj5rzSz\ RS\$S\$S>\ R4Sjr{Sz\ RS9\ RS\$S\$S>\ R4 Sjr|\ R4Sz\ RS9\ RS\$S\$S;\ RFS>\ R4 Sjjr}\F\ R2\ RSS4Sz\ RS7\\$S\ RFSW\ RFS\\OS\\OS>\ R4Sjj5r~S9\ RS\ RS>\ R4Sjr\ R24Sz\ RS9\ RS\ RFS>\ R4Sjjr\ R4Sz\ RS9\ RS;\ RFS>\ R4Sjjr\"\ESt5\ R24Sz\ RS9\ RS\ RFS>\ R4Sjj5r\"\ES5\ R24Sz\ RS9\ RS\ RFS>\ R4Sjj5r\"\ESt5\ R4Sz\ RS9\ RS;\ RFS>\ R4Sjj5r\"\ES5\ R4Sz\ RS9\ RS;\ RFS>\ R4Sjj5rgs snf)N)Enumauto)CallableDictListOptionalTupleUnion)_f32_to_floatx_unpacked_floatx_unpacked_to_f32_n_ones)_register_custom_op_register_meta_op)choose_qparams_affine"choose_qparams_affine_with_min_maxquantize_affinedequantize_affine MappingTypeZeroPointDomain TorchAODType_choose_qparams_affine_tinygemm)_choose_qparams_affine_dont_preserve_zero_choose_qparams_affine_floatx'_choose_qparams_and_quantize_affine_hqq+_choose_qparams_and_quantize_scale_only_hqq'_choose_qparams_and_quantize_affine_qqq_choose_scale_float8_choose_qparams_gguf_quantize_affine_no_zero_point_quantize_affine_tinygemm_quantize_affine_floatx_quantize_affine_float8_quantize_gguf _dequantize_affine_no_zero_point_dequantize_affine_tinygemm_dequantize_affine_floatx_dequantize_affine_qqq_dequantize_affine_float8_dequantize_gguf_fake_quantize_affine_fake_quantize_affine_cachemaskcB\rSrSrSr\"5r\"5r\"5rSr g)r7aHow floating point number is mapped to integer number symmetric mapping means floating point range is symmetrically mapped to integer range let's say we have floating point range (-3.5, 10.2) and integer range (-8, 7) (int4) we'll use (-10.2, 10.2) as the range for floating point and map that to (-8, 7) e.g. scale = (10.2 - (-10.2)) / (7 - (-8)) SYMMETRIC_NO_CLIPPING_ERR is a variant of symmetric mapping, where the scale is the max of smin and smax, where smin = min_val_neg / quant_min, and smax = max_val_pos / quant_max. By calculating smin and smax individually, there can be less round error on negative values, and no out-of-range of all floating point values. asymmetric mapping means we just directly map the floating point range to integer range, for the above example, we will map (-3.5, 10.2) to (-8, 7) and calculate quantization parameter based on this mapping e.g. scale = (10.2 - (-3.5)) / (7 - (-8)) N) __name__ __module__ __qualname____firstlineno____doc__r SYMMETRICSYMMETRIC_NO_CLIPPING_ERR ASYMMETRIC__static_attributes__r._/home/james-whalen/.local/lib/python3.13/site-packages/torchao/quantization/quant_primitives.pyrr7s$I $Jr8rcB\rSrSrSr\"5r\"5r\"5rSr g)rOa8Enum that indicate whether zero_point is in integer domain or floating point domain integer domain: quantized_val = (float_val / scale) (integer) + zero_point (integer) float domain: quantized_val = (float_val - (zero_point (float) - scale * mid_point)) / scale none domain: quantized_val = (float_val / scale) r.N) r/r0r1r2r3rINTFLOATNONEr7r.r8r9rrOs &C FE 6Dr8rcz\rSrSrSr\"5r\"5r\"5r\"5r \"5r \"5r \"5r Sr g)r\z? Placeholder for dtypes that do not exist in PyTorch core yet. r.N)r/r0r1r2r3rINT1INT2INT3INT4INT5INT6INT7r7r.r8r9rr\s: 6D 6D 6D 6D 6D 6D 6Dr8r)r))ii)ii_DTYPE_TO_QVALUE_BOUNDS _DTYPE_TO_BIT_WIDTH_SUB_BYTE_UINT_BOUNDS)r)rL)rN)irR)i)i)i?_SUB_BYTE_INT_BOUNDSrrL)rrN)rrR)rr[)rr\)rr])rrJtorchaoFRAGMENTc\rSrSrSr\S\RS\R4Sj5r\S\RS\R4Sj5r Sr g ) _Roundz> Implementation of generic round operation with backward STE. xreturnc.[R"U5$N)torchround)ctxrfs r9forward_Round.forwards{{1~r8gycU$rir.rlros r9backward_Round.backwards r8r.N) r/r0r1r2r3 staticmethodrjTensorrmrrr7r.r8r9rdrds[ %,,5<<r8rdc\rSrSrSr\S\RS\RS\R4Sj5r \S\RS\R4Sj5r S r g ) _RoundToFloat8z@ Implementation of `tensor.to(float8_dtype)` with backward STE. rf float8_dtypergc$URU5$ri)to)rlrfrys r9rm_RoundToFloat8.forwardsttL!!r8roc US4$rir.rqs r9rr_RoundToFloat8.backwards 4xr8r.N) r/r0r1r2r3rtrjrudtypermrrr7r.r8r9rwrwsf" "EKK"ELL""%,,5<<r8rwc>U[;a@[R"U5R[R"U5RpCO#U[ ;a[ SU35e[ Uup4UcUnUcUnX:d SUSU35eX$::d SUSU35eX4$)aGet quant_min and quant_max args based on dtype and also verify bounds. Args: dtype: Target quantization dtype (e.g., torch.uint8, torch.int8, or FP8 types) quant_min: Minimum quantized value, or None to use dtype default quant_max: Maximum quantized value, or None to use dtype default Returns: Tuple[int/float, int/float]: Validated (quant_min, quant_max) values Raises: ValueError: If dtype is unsupported AssertionError: If quant_min/quant_max are out of bounds for dtype zUnsupported dtype: z9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: ) FP8_TYPESrjfinfominmaxrK ValueError)r quant_min quant_maxquant_min_lower_boundquant_max_upper_bounds r9_get_and_check_qmin_qmaxrs  KK  " " KK  " " 5 - -.ug6777Nu7U4) )  - ""7!8 YK Q -  - ""7!8 YK Q -  r8c [U5[U5:Xde/n/nSn[[U55HnXX:warXS:ajXX-S:XdSUSXSUSX35eURXX-5 URX5 URUS-5 US- nMURX5 XS:waURU5 US- nM X#4$)aGiven block_size and input size find the parameters for reduction: Output: shape_for_reduction: the shape we use to `view` input to prepare it for reduction reduction_dims: the dims we'll do reduction over Example:: Input: block_size: (3, 3, 2, 10) input_size: (3, 3, 10, 10) Output: shape_for_reduction: (3, 3, 5, 2, 10) reduction_dim: [0, 1, 3, 4] rrLzExpecting input size at z dimension: z" to be divisible by block_size at rM)lenrangeappend) block_size input_sizeshape_for_reductionreduction_dimscur_dimis r9_get_reduction_paramsrs* z?c*o -- -NG 3z? # =JM )jma.?=:=0A5 *1#\*-Hjkljmmy{E{HzIJ 5 & &z} 'E F  & &z} 5  ! !'A+ . qLG & &z} 5}!%%g. qLG#$$  ..r8inputr.scale zero_point output_dtyperrrgc $[UUUUUUU5$)a Args: input (torch.Tensor): original float32, float16 or bfloat16 Tensor block_size: (Tuple[int, ...]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization output_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for output Tensor, if not specified, it will be derived from dtype quant_max (Optional[int]): maximum quantized value for output Tensor, if not specified, it will be derived from dtype Note: How can block_size represent different granularities? let's say we have a Tensor of size: (3, 3, 10, 10), here is the table showing how block_size represents different granularities: granularity type | block_size per_tensor | (3, 3, 10, 10) per_axis (axis=0) | (1, 3, 10, 10) per_axis (axis=1) | (3, 1, 10, 10) per_group (groupsize=2) | (3, 3, 10, 2) per_group (groupsize=2) for axis = 3 | (3, 3, 2, 10) Output: quantized tensor with requested dtype )_quantize_affinerrrrrrrs r9rr@s'J    r8c[XEU5upVU[;a[Rn[ UUUUUU5R U5$)a#Quantize tensor using affine quantization with integer zero point domain. Op definition that has compatible signatures with custom op library. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) output_dtype: Target quantized dtype (e.g., torch.uint8, torch.int8) quant_min: Minimum quantized value, derived from dtype if None quant_max: Maximum quantized value, derived from dtype if None Returns: Quantized tensor with requested dtype Note: zero_point_domain is pre-defined as INT, meaning: quantized_val = (float_val / scale) (integer) + zero_point (integer) )rrWrjuint8_quantize_affine_no_dtype_castr{rs r9rrpsR<4LYWI,,{{ )     br8cUR[R[R[R4;dSUR35e[ U5UR 5:XdSUR 5SU35e[XR55upgURnURU5nUn UHn SX'M URU 5nUb&UR5S:aURU 5nOSn[R"[RUSU- -5U-XE5n U RU5n U $)aaQuantize tensor using affine quantization without dtype casting. Performs quantization with integer zero point domain without casting to target dtype. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) quant_min: Minimum quantized value quant_max: Maximum quantized value Returns: Quantized tensor without dtype casting The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale and zero_point with zero_point_domain = INT 3. Reshape the quantized result to original shape Unsupported input dtype: Got input dim:, block_size: rLNr?rrjfloat32float16bfloat16rdimrsizeshapeviewnumelclamprdapply rrrrrrrroriginal_shapeshape_after_reductionrquants r9rrsG> ;;    1 #5;;-0 1  z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E*"2"2"4q"8__%:;  KK UcEk*+j8) E JJ~ &E Lr8c[XEU5upVU[;a[Rn[ UUUUUU5R U5$)aQuantize tensor using affine quantization with float zero point domain for tinygemm. Specialized quantization for tinygemm int4mm kernel where zero point is in floating point domain. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) output_dtype: Target quantized dtype (e.g., torch.uint8, torch.int8) quant_min: Minimum quantized value, derived from dtype if None quant_max: Maximum quantized value, derived from dtype if None Returns: Quantized tensor with requested dtype The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale and zero_point with zero_point_domain = FLOAT 3. Reshape the quantized result to original shape Note: zero_point_domain is pre-defined as FLOAT, meaning: quantized_val = (float_val - (zero_point (float) - scale * mid_point)) / scale )rrWrjr'_quantize_affine_tinygemm_no_dtype_castr{rs r9r r sSF4LYWI,,{{ 2     br8cUR[R[R[R4;dSUR35e[ U5UR 5:XdSUR 5SU35e[XR55upgURnURU5nUn UHn SX'M URU 5nUb&UR5S:aURU 5nOSnXT-S-S- n X2U -- n [R"[RX - U- 5XE5n U RU5n U $)aQuantize tensor using affine quantization with float zero point domain without dtype casting. Specialized quantization for tinygemm int4mm kernel where zero point is in floating point domain. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) quant_min: Minimum quantized value quant_max: Maximum quantized value Returns: Quantized tensor without dtype casting The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale and zero_point with zero_point_domain = FLOAT 3. Reshape the quantized result to original shape rrrrLNrrMr)rrrrrrrrrrr mid_pointmin_valrs r9rrsZ> ;;    1 #5;;-0 1  z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E*"2"2"4q"8__%:;  &*a/I9,,G KK eo%>? VE JJ~ &E Lr8c[XEU5upVU[;a[Rn[ UUUUUU5R U5$)aQuantize tensor using affine quantization without zero point. Specialized quantization for cases where zero point is not needed (e.g., floatx quantization). Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (ignored, should be None) output_dtype: Target quantized dtype (e.g., torch.uint8, torch.int8) quant_min: Minimum quantized value, derived from dtype if None quant_max: Maximum quantized value, derived from dtype if None Returns: Quantized tensor with requested dtype The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale with zero_point_domain = NONE 3. Reshape the quantized result to original shape Note: zero_point_domain is pre-defined as NONE, meaning: quantized_val = (float_val / scale) | This is primarily used for floatx quantization where we do not want to round values to nearest integer and instead scale and cast. )rrWrjr,_quantize_affine_no_zero_point_no_dtype_castr{rs r9rrOsSH4LYWI,,{{ 7     br8cUR[R[R[R4;dSUR35e[ U5UR 5:XdSUR 5SU35e[XR55upgURnURU5nUn UHn SX'M URU 5nUb&UR5S:aURU 5nOSn[R"[RUSU- -5XE5n U RU5n U $)aQuantize tensor using affine quantization without zero point and without dtype casting. Specialized quantization for cases where zero point is not needed without casting to target dtype. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (ignored, should be None) quant_min: Minimum quantized value quant_max: Maximum quantized value Returns: Quantized tensor without dtype casting The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale with zero_point_domain = NONE 3. Reshape the quantized result to original shape rrrrLNrrrrs r9rrs=> ;;    1 #5;;-0 1  z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E*"2"2"4q"8__%:;  KK UcEk%:;Y RE JJ~ &E Lr8r input_dtypec "[UUUUUUUUS9$)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 Default value for zero_point is in integer domain, zero point is added to the quantized integer value during quantization Output: dequantized Tensor, with requested dtype or fp32 r)_dequantize_affinerrrrrrrrs r9rrs)8   !  r8c U[;a'URU:XdSUSUR35eU[R[R[R 4;d SU35e[ XEU5upV[UUUUUUU5$)aDequantize tensor using affine dequantization with integer zero point domain. Op definition that has compatible signatures with custom op library. Args: input: Quantized tensor to dequantize block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) input_dtype: Expected dtype of input tensor (e.g., torch.uint8, torch.int8) quant_min: Minimum quantized value for input tensor quant_max: Maximum quantized value for input tensor output_dtype: Target output dtype (default: torch.float32) Returns: Dequantized tensor with requested output dtype Expected: , got: Unsupported output dtype: )rWrrjrrrr!_dequantize_affine_no_dtype_checkrs r9rrs://{{k)  WU[[M : )     3 $L>2 3  4KIVI ,   r8c[U5UR5:XdSUR5SU35e[XR55upxURn UR U5nUn UHn SX'M UR U 5nUbUR U 5nUR USS9n UbXR U5- n X-n U R U 5R U5$)aDequantize tensor using affine dequantization without dtype checking. Converts quantized tensors to their high precision floating point representation. Args: input: Quantized tensor to dequantize block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) quant_min: Minimum quantized value for input tensor quant_max: Maximum quantized value for input tensor output_dtype: Target output dtype (default: torch.float32) Returns: Dequantized tensor with requested output dtype The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Dequantize the input based on the quantization parameters scale and zero_point 3. Reshape the quantized result to original shape and change dtype to the output_dtype rrrLT)copyrrrrrrr{ rrrrrrrrrrrrdequants r9rrs> z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E__%:; hh|$h/GMM,77oG << ' * *< 88r8c[U5UR5:XdSUR5SU35e[XR55upxURn UR U5nUn UHn SX'M UR U 5nUbS5eUR U5n X-n U R U 5R U5$)aDequantize tensor using affine dequantization without zero point and without dtype checking. Converts quantized tensors to their high precision floating point representation without zero point. Args: input: Quantized tensor to dequantize block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (ignored, should be None) quant_min: Minimum quantized value for input tensor quant_max: Maximum quantized value for input tensor output_dtype: Target output dtype (default: torch.float32) Returns: Dequantized tensor with requested output dtype The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Dequantize the input based on the quantization parameters scale (no zero point) 3. Reshape the quantized result to original shape and change dtype to the output_dtype rrrLz>zero_point should be None for _dequantize_affine_no_zero_pointrrs r9/_dequantize_affine_no_zero_point_no_dtype_checkrSs> z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E  H hh|$GoG << ' * *< 88r8c U[;a'URU:XdSUSUR35eU[R[R[R 4;d SU35e[ XEU5upV[UUUUUUU5$)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization, no zero point is used for this op input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 Default value for zero_point is in integer domain, zero point is added to the quantized integer value during quantization Output: dequantized Tensor, with requested dtype or fp32 rrr)rWrrjrrrrrrs r9r$r$s://{{k)  WU[[M : )     3 $L>2 3  4KIVI :   r8c[U5UR5:XdSUR5SU35e[XR55upxURn UR U5nUn UHn SX'M UR U 5nUbUR U 5nXT-S-S- n X - n U R U5n X-n UbX- n U R U 5R U5$)aThis function converts AQT tensors to their high precision floating point representation The op does the following: 1. figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. dequantize the input based on the quantization parameters scale and zero_point and args like zero_point_domain 3. reshape the quantized result to origianl shape and change dtype to the output_dtype rrrLrMr)rrrrrrrrrrrrrrs r9*_dequantize_affine_tinygemm_no_dtype_checkrs" z?eiik )  ^J<@ )+@JJL+'[[N JJ* +E/ #$  JJ, -E__%:; &*a/IGjj&G G << ' * *< 88r8c U[;a'URU:XdSUSUR35eU[R[R[R 4;d SU35e[ XEU5upV[UUUUUUU5$)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 Default value for zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) Output: dequantized Tensor, with requested dtype or fp32 rrr)rWrrjrrrrrrs r9r%r%s://{{k)  WU[[M : )     3 $L>2 3  4KIVI 5   r8 quant_dtypezero_point_domainc Uc [S5eU[RLaUb [S5e[UUUUUUUU5upU $)aR General fake quantize op for quantization-aware training (QAT). This is equivalent to calling `quantize_affine` + `dequantize_affine` but without the dtype casts. Args: input (torch.Tensor): original float32, float16 or bfloat16 Tensor block_size: (Tuple[int, ...]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_dtype (torch.dtype): desired quantized dtype for determining and validating quant_min and quant_max values. quant_min (Optional[int]): minimum quantized value for output Tensor, if not specified, it will be derived from dtype quant_max (Optional[int]): maximum quantized value for output Tensor, if not specified, it will be derived from dtype zero_point_domain (ZeroPointDomain): the domain that zero_point is in, should be either integer or float if zero_point is in integer domain, zero point is added to the quantized integer value during quantization if zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) value during quantization default is ZeroPointDomain.INT /Please use ZeroPointDomain.NONE instead of None8zero_point should be None when zero_point_domain is NONE)rrr>_do_fake_quantize_affine) rrrrrrrr_fqs r9r*r*s`> JKK o22 2z7MSTT&   GQ Ir8c Uc [S5eUcUb [S5e[UUUUUUUU5up[R"X:X:*5n X4$)a0 General fake quantize op for quantization-aware training (QAT). This is equivalent to calling `quantize_affine` + `dequantize_affine` but without the dtype casts. Note: Compared to :func:`~torchao.quantization.quant_primitives._fake_quantize_affine`, this consumes more memory and returns an additional outlier mask for intermediate quantized values. Args: Same as :func:`~torchao.quantization.quant_primitives._fake_quantize_affine`. Returns: A 2-tuple of ( final fake quantized values, outlier mask for intermediate quantized values ) rr)rrrj logical_and) rrrrrrrrqdqmasks r9r+r+Isq: JKK  "z'=STT&   GQ   an @D :r8c PURn[XEU5upVU[R:Xa [n [ n OPU[R :Xa [n [n O/U[R:Xa [n [n O[SU35eU "UUUUUU5n U "U UUUUUUS9n X4$)aHelper function for fake quantization that returns both intermediate and final values. Performs quantization followed by dequantization without dtype casting, returning both the intermediate quantized values and the final dequantized values. Args: input: Input tensor to fake quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) quant_dtype: Target quantized dtype for determining quant_min/quant_max quant_min: Minimum quantized value, derived from dtype if None quant_max: Maximum quantized value, derived from dtype if None zero_point_domain: Domain of zero point (INT, FLOAT, or NONE) Returns: Tuple of (intermediate quantized values, final dequantized values) Helper function for `_fake_quantize_affine` that returns both the intermediate quantized values and the final dequantized values. z Unrecognized zero point domain: r) rrrr<rrr=rrr>rrr) rrrrrrrrrrrrrs r9rrxs>++K3KIVIO///9> o33 3BG o22 2GL; "    r8c [X4U5upEU[RLd SU35eUc URnUc*[R "UR5R n[U5UR5:XdSUR5SU35e[X R55upURU 5n[R"X SS9n [R"X SS9n U n U nX- [XT- 5- n[R"XS9nXT-S-S- nXU--nUc URnUR!US 9nUR!XpR"S 9U4$) a Specialized version of choose_qparams_affine This is used for tinygemm int4mm kernel where zero point is in floating point domain and zero does not have to be exactly representable. Args: input (torch.Tensor): fp32, bf16, fp16 input Tensor mapping_type (MappingType): determines how the qparams are calculated, symmetric or asymmetric block_size: (Tuple[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam target_dtype (torch.dtype): dtype for target quantized Tensor quant_min (Optional[int]): minimum quantized value for target quantized Tensor quant_max (Optioanl[int]): maximum quantized value for target quantized Tensor eps (Optional[float]): minimum scale, if not provided, default to eps of input.dtype scale_dtype (torch.dtype): dtype for scale Tensor zero_point_dtype (torch.dtype): dtype for zero_point Tensor Output: Tuple of scales and zero_points Tensor with requested dtype Unsupported mapping type: rrFrkeepdimrrLrMrrdevice)rrr6rrjrrrrrrraminamaxfloatrr{r)rrrrrrrrrrrrmax_val min_val_neg max_val_posrrrs r9rrsr@4LYWI ;11 1 $\N3 1kk  {kk%++&** z?eiik )  ^J<@ )+@JJL+' JJ* +EjjEBGjjEBGKK  &% 0E*F FE KK 'E&*a/Iy00J ;;%56J 88+ll8 ;Z GGr8c [X4U5upEU[R:Xd SU35eUc URnUc*[R "UR5R n[U5UR5:XdSUR5SU35e[X R55upURU 5n[R"X SS9n [R"X SS9n U n U nX- [XT- 5- n[R"XS9nU[ R#X- 5- n[R"UXE5nUc[R$nUR'XpR(S9UR'US94$) a|Specialized version of choose_qparams_affine with zero_point_domain=ZeroPointDomain.INT and preserve_zero=False. Args: input (torch.Tensor): fp32, bf16, fp16 input Tensor mapping_type (MappingType): determines how the qparams are calculated, asymmetric only block_size: (Tuple[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam target_dtype (torch.dtype): dtype for target quantized Tensor quant_min (Optional[int]): minimum quantized value for target quantized Tensor quant_max (Optioanl[int]): maximum quantized value for target quantized Tensor eps (Optional[float]): minimum scale, if not provided, default to eps of input.dtype scale_dtype (torch.dtype): dtype for scale Tensor zero_point_dtype (torch.dtype): dtype for zero_point Tensor Now removed params default values: zero_point_domain (ZeroPointDomain): the domain that zero_point is in, defaults to Integer preserve_zero (bool): whether to preserve zero in the quantized Tensor, defaults to False Output: Tuple of scales and zero_points Tensor with requested dtype rrrFrrrr)rrr6rrjrrrrrrrrrrrrdrint32r{r)rrrrrrrrrrrrrrrrrs r9rr-s{<4LYWI ;11 1 $\N3 1kk  {kk%++&** z?eiik )  ^J<@ )+@JJL+' JJ* +EjjEBGjjEBGKK  &% 0E*F FE KK 'EV\\+*=>>JZ>J ;; 88+ll8 ;Z]]>K> r8Trr preserve_zeroc U c [S5e[XEU5upVU[R[R[R 4;d SU35eUbUcS5eUR UR :XdS5eUc UR nUc*[R"UR 5RnURn U (aW[R"U[R"U55n [R"U[R"U55nOUn UnU[R:XdU[R:XGaU[R:Xa,[R"U *U5nU[Xe- 5S- - nOOU[R:XdeU [U5- nU[U5- nUU:n[R"UUU5nU (d [S5eU [ R":Xa [S5eU [ R$:XaSnO([R&"U[)Xe-S -S- 55n[R*"XS 9nOU[R :XdeX- [R,"[Xe- 5XS 9- n[R*"XS 9nU [ R$:XaSnOU [ R.:XaEU[0R3X- 5- n[R*"UXV5nU c[R4n O,U [ R":XdS 5eXe-S -S- nXU--nUbUR7U S 9nUR7XRS 9U4$)aLA variant of :func:`~torchao.quantization.quant_primitives.choose_qparams_affine` operator that pass in min_val and max_val directly instead of deriving these from a single input. This is used for observers in static quantization where min_val and max_val may be obtained through tracking all the data in calibration data set. Args: Mostly same as :func:`~torchao.quantization.quant_primitives.choose_qparams_affine`. with one difference: instead of passing in `input` Tensor and use that to calculate min_val/max_val and then scale/zero_point, we pass in min_val/max_val directly Nrrz@Need to provide `min_val` and `max_val`, got: {min_val, max_val}z]Expecting `min_val` and `max_val` to have the same dtype, got: {min_val.dtype, max_val.dtype}rMzBpreserve_zero == False is not supported for symmetric quantizationzbzero_point_domain should be ZeroPointDomain.INT or ZeroPointDomain.NONE for symmetric quantizationrLrrzGzero_point must be in FLOAT/INT/None domain for asymmetric quantizationr)rrrr4r5r6rrjrrrr zeros_likerrwhererr=r> full_likeintrtensorr<rdrrr{)rrrrrrrrrrrr scale_devicerrrsminsmaxrrrs r9rrqsU0 JKK3LYWI -- 3 $L>2 3   7#6J 6 ==GMM )g )mm  {kk'--(,,>>Lii)9)9')BC ii)9)9')BC     --- ;@@ @ ;00 0))[L+>K5)>#?!#CDE;#H#HH HHy!11Dy!11D$;DKKdD1ET   5 5 5t   4 4 4JY5JQ5NRS4S0TUJ E+{55555*ell )' ( /   E+  4 4 4J /"5"5 5"V\\+2E%FFJZFJ'#(;; $(=(== Y =#.2a7I%y'88J]])9]: 88+nn8 =z IIr8c [X4U5upEU[RR[RR[R R4;d SU35eUc UR nUc*[R"UR 5Rn[U5UR5:XdSUR5SU35e[X R55upURU 5n[R"X SS9n [R "X SS9n [R""U [R$"U 55n [R&"U [R$"U 55nU[RR:XdU[RR:XaU[RR:Xa,[R&"U *U5nU[)XT- 5S- - nOYU[RR:XdeU [)U5- nU[)U5- nUU:n[R*"UUU5n[R,"U[/XT-S-S- 55n[R0"XS9nOU[R R:XdeX- [)XT- 5- n[R0"XS9nU[2R5X- 5- n[R0"UXE5nUc[R6nUR9XpR:S 9UR9US 94$) aWop definition that has compatible signatures with custom op library The op does the following: 1. figure out the dimension for reduction based on block_size 2. find min_val/max_val based on the dimension for reduction 3. calculate quantization parameters based on min_val/max_val based on args like `preserve_zero` and `zero_point_domain` rrrFrrMrLrrr)rrr4rr5r6rrjrrrrrrrrrrrrrrrrrrdrrr{r)rrrrrrrrrrrrrrrrrrrrs r9rrs(4LYWI ""--22## 3 $L>2 3  kk  {kk%++&** z?eiik )  ^J<@ )+@JJL+' JJ* +EjjEBGjjEBG))GU%5%5g%>?K))GU%5%5g%>?K  --222 ;@@EE E ;0055 5))[L+>K5)>#?!#CDE;#H#H#M#MM MMy!11Dy!11D$;DKKdD1E__UC1F1Ja0O,PQ  E+{55:::::*eI4I.JJ E+k.A!BB [[YB  #${{  88+ll8 ;Z]]>K> r8wnum_bits group_sizec^ ^ US:Xd SU35eURum m USST 4;d SU35eURnUS:XaT nUT :GaURSU45nSU-S- nUS-S-n[R"[R "U5SSS 9nUSU- -n[ RX- 5R5nXu- n[R"US U5nXu- R5U-nU U 4S jn U "U5nU "U5n[R"[R "U5SSS 9n U S -n X- R5RS S5R[R5n U R5U -nU RSS5R[RS9n URT S5R5U - R[RS9nOSUS- -S- n[R"[R "U5SSS 9n X-n [ RX - 5R5n[R"Xt*U5nUR5U -n[R "/[RUS9nU SSU- --n U RT S5R5R[R5n XvX4$)NrOUnsupported num_bits = rXUnsupported groupsize = rMrLT)rrcJ>URTT45R5nU$rireshape contiguousrsize_ksize_ns r9 reshape_w:_choose_qparams_and_quantize_affine_qqq..reshape_wZ$ 66*+668AHr8g_@rIrJrrrS)rrr rjrabsrdrrrhalfrkr{int8rr r)rrr orig_device max_q_val half_q_vals_groupq_ww_refr s_channelt_int8rrs @@r9rr;s q=>3H:>>=WWNFF "c6* *S.Fzl,SS *((KR F IIr:& 'xK!O !m) **UYYq\2t<1y= ll1;'++- kk#q),!'')G3 n% JJuyy/TB U #**,224=@@L )%%b!,//ekk/B ??62.99;iGKK**L (Q,'!+ JJuyy|R>  ll1=)--/kk#z95 Y&,,rKHQ1x<(( %%fb1<<>AA%++N  ))r8c8URn[XR55upEURU5n[R "XSS9n[R "XSS9nSnSn U[X- 5S- - n Xv- [X- 5- n Un [US-S:XdeS[US-4n [XR55upEU RU5n U RU5n UR5n UHnSX'M [R "[R"U 5USS9n[R "[R"U 5USS9nSnSnU[UU- 5- nU[UU- 5- nURU 5nURU 5n[R"U U- UU5n[R"U U- UU5nURU5URU5URU5URU54$) a There are two sets of qparams: quantized_block_scale, quantized_block_min and super_block_scale_scale and super_block_min_scale the relationship is the following: block_scale = quantized_block_scale * super_block_sclae block_min = quantized_block_min * super_block_min quantized_val = (float_val - block_min) / block_scale + quant_min first we calculate block_scale and block_min then we calculate super_block_scale_scale and super_block_min_scale after that we can calculate quantized_block_scale and quantized_min_scale the returned values are: super_block_scale_scale, super_block_min_scale, quantized_block_scale and quantized_min_scale Frr[rrMrXrLr]) rrrrrjrrr _GGUF_QK_Krrrr{)rrrrrrrrrr block_scale block_minsuper_block_sizerrblock_scale_absmaxblock_min_absmaxqparam_quant_maxqparam_quant_minsuper_block_scale_scalesuper_block_min_scalesuper_block_scale_scale_viewsuper_block_min_scale_viewquantized_block_scalequantized_block_mins r9rrsP" KKE+@JJL+' JJ* +EjjEBGjjEBGIIU9#89A=>K$i.C(DDKI  2 &! ++ +:B78*?**,+'""#67K23I/446 #$  +NEzz ).% 05++4-u++0$;#?#?@U#V !6!;!;*G**\* Nr8rrc^ ^ US:Xd SU35eURum m USST 4;d SU35eUS:XaT nUT :ajURSU45nSU-S- nUS-S-nXR5-nX- R5URSS5-nU U 4Sjn U "U5nOUSS U- --nUR5U-nUc!UR[R 5nU$URU5nU$) NrOrrXrrrMrLcJ>URTT45R5nU$rir r s r9r)_dequantize_affine_qqq..reshape_wjrr8rS)rr rr{rjr) rrrrrrrrw_dqrrrs @@r9r'r'Qs( q=>3H:>>=WWNFF "c6* *S.Fzl,SS *R F IIr:& 'xK!O !m) NN,,$$&Q)?? q8|!45 vvx)#wwu}}% Kww|$ Kr8rfbetalp_normc US:XaZ[R"U5[RRR [R "U5SU- - 5-$[R"U5[RRR [R "U5SU- [R "[R "U5US- 5-- 5-$)NrLr)rjsignnn functionalrelurpow)rfr:r;s r9 _shrink_lp_oprBs!|zz!}uxx2277 ! sTz8QRRRzz!}uxx2277 IIaLC$J%))EIIaL'A+*NN N   r8Fgffffff?g$@g)\(?)r;r:kappaiters early_stoprzeromin_maxaxisrrverbose opt_paramsc :USUSUSUSUS4uppn Uc URO[R"U5nUc0URS:Xa[RO[RnUR XVS9nUR XVS9nUR XVS9nSn[ U 5Hn[R"X-U-5RUS US 5nUU- U- n[UU- X5n[R"UUU- U-- US S 9nX-n [[R"UU- 5R55nU(a'[S [US -5-S[U5-5 U (dMUU:aUnM O UR UR5nUR UR5nAAAA[RR!5 [R"X-U-5RUS US 5nUX4$)Nr;r:rDrErFcudarg@rrLTrIrzIter z | Error: )rrjtyperrr{rrkrrBmeanrrprintstrrM empty_cache)rrrGrHrIrrrJrKr;r:rDrErFW_f best_errorrW_qW_rW_e current_errors r9 optimize_weights_proximal_legacyrZs& 9677< /+G5 &~V]]ELL4HF }"(++"7 emm ))%) /C HH5H 0E 777 .DJ 5\kk#+,-33GAJ KTzU"C#It5zz#se 33$M eiic 2779:  'CAJ& s=7I(I J :z)*  HHV]] #E 776== !D S#s JJ ++fnt+ , 2 271:wqz JC  r8val1val2cP[U[R"X- 5-5U:H$ri)rmathceil)r[r\s r9 _is_divisibler`s" tdii ,, - 55r8rVnbitsrcSnSU-S- nXe-S-S- nXrR5- UR5-RUR5nUn URU5n XU4$)NrrMrL)rr{rr) rVrrGrarrrrzero_aoscale_aoW_q_aos r9"_convert_to_affinequantized_formatrfsmI51 I&*a/IJJL(EKKM9==djjIGH XXe_F W $$r8@rMoptimize compute_dtype raw_outputoptimize_weightsc US;dS5eUbK[UR5U5(d,S[UR5-S-[U5-5eUR U[ R S9n U Rn Ub,US:XaU RSU/5OU RUS/5n U RUSS 9S n U RUSS 9S n [S U-S- 5nS nX/nXU - - RS S 9nU *U-nUS;a[RU5nU(aU "U UUUUUUS9unnnOSUR U5nUR U5n[RU U-U-5RUS US5nSU- nUSLa[UUUX5unnnOlURU 5nUS:Xa+URU S S5nURU S S5nO*URSU S5nURSU S5nUR [ RUS9nUR XVS9nUR XVS9nA A A [ R R#5 UUUU 4$)a@Choose quantization parameters and quantize tensor using HQQ (Half-Quadratic Quantization). Performs quantization using HQQ method with optional weight optimization via proximal solver. Args: tensor: Input tensor to quantize (float32, float16, or bfloat16) nbits: Number of bits for quantization (default: 4) group_size: Size of quantization groups (default: 64) optimize: Whether to optimize weights using proximal solver (default: True) axis: Axis along which to perform quantization (0 or 1, default: 1) compute_dtype: Target compute dtype (default: torch.float16) device: Target device for computation (default: "cuda") verbose: Whether to print optimization error information (default: False) raw_output: If True, return params in HQQ library format (default: False) optimize_weights: Weight optimization function (default: optimize_weights_proximal_legacy) Returns: Tuple of (quantized_weights, scale, zero_point, original_shape) Note: Uses proximal solver to minimize ||W - dequantize(quantize(W))||_p^p for weight optimization. r_zaxis should be either 0 or 1zEgroup_size should be divisble by the total tensor dimensions. shape: z, group_size: )rrrLrXTrNrrMg@)r)rO)rrrGrHrIrrJrFr)r`rrRrr{rjrr rrrkrrdrrfrrMrS)rrarrhrIrirrJrjrkWr_min_maxmax_vmin_vrHrrGrVs r9rrsD 6>999>V\\^Z88 S&,,  *o  8  u}} 5A GGE,0AIAIIr:& 'AIIzSUFV)rE stochasticearly_stop_tol hp_tensorqminqmaxrErrrscURS:XdS5e[U[[45(a[ U5S:XdS5eUSS:Xa USS:dS5eX#:dS5e[ R n[ R"U5RnURup[US5n X-S:Xd SU S U 35eS [ RS [ R4S jn S [ RS [ R4S jn U(aU OU nURU5R5nX-nURU UU 5n[[!U5[!U55=(d SnUR!5R#SS9U- R%U5nUR'5n[)[SU55HnU"UUR+S5- 5R-X#5nUU-R/S[ R S9nUU-R/S[ R S9n[ R0"US:UU- U5nUR%U5R!5nUU- R!5UR%U5- R5nUU:a OUnM U"UUR+S5- 5R-X#5nURX5R5R[ R25nUR4nURU5nUU4$)a Half-Quadratic Quantization (scale-only, symmetric) for 2D weights with row-wise blocks. - hp_tensor: [out, in] (bf16/fp16/fp32 accepted; promoted to fp32 internally) - block_size: must be [1, group_size]; groups along the last dim - qmin, qmax: integer range (e.g., -8, 7 for signed 4-bit) Returns: qdata: int32, same shape as hp_tensor scale: hp_tensor.dtype, shape [out, in // group_size] (one scale per row-wise block) rMzhp_tensor must be 2D [out, in]z)block_size must be a 2-element list/tuplerrLz7block_size must be [1, group_size] with group_size >= 1zqmin must be < qmaxz in_features=z! must be divisible by group_size=rfrgc"UR5$ri)rkrfs r9 round_det>_choose_qparams_and_quantize_scale_only_hqq..round_detyswwyr8c\[R"U[R"U5-5$ri)rjfloor rand_likerys r9 round_stoch@_choose_qparams_and_quantize_scale_only_hqq..round_stoch}s{{1uq1122r8rrX)rr)ndim isinstancelisttuplerrjrrrrrrur{r rrrr clamp_mincloner unsqueezersumrrr)rtrrurvrErrrsri compute_epsnkrrzr_rrmn_groupsWgqabsr prev_scalerQgnumdenrelqdata out_dtypes r9rrQs* >>Q @ @@  j4- 0 0S_5I3 I a=A *Q-1"4A 4 ;---;MMM++m,00K ??DAZ]#J >Q  qc::,G U\\ell3u||3 3# B  ]#..0AH 8Z (B s4y#d) $ )D VVX]]q] !D ( 3 3K @EJ3q%= !U__R(( ) / / ;Bwmmm7Bwmmm7 C!GS3Y ;  #%  "'')J,@,@,MMRRT    #"( B$ $ % + +D 7B GGAM $ $ & ) )%++ 6EI HHY E %<r8ebitsmbitsc[US- nS[UU- -[US-SU-- -nURnUR5nUR5R S5R SS9U- nUR U5$)aChoose quantization parameters for floatx quantization. Calculates scale parameter for quantizing to custom floating point format. Args: tensor: Input tensor to quantize (float32, float16, or bfloat16) ebits: Number of exponent bits in target floatx format mbits: Number of mantissa bits in target floatx format Returns: Scale tensor for floatx quantization Note: Uses global lookup table as workaround for torch.compile() compatibility since _n_ones() is not compatible due to << operator. rLrMg-q=r) _ONES_TABLErrrrrr{)rrrexp_bias max_normalrrs r9rrs2519%H{5)H45EAI!U(+J LLE \\^F JJL  a & &5 & 1J >E 88E?r8cbUR5n[XRSS5- X#5nU$)a Quantizes the float32 high precision floating point tensor to low precision floating point number and converts the result to unpacked floating point format with the format of 00SEEEMM (for fp6_e3m2) where S means sign bit, e means exponent bit and m means mantissa bit rXrL)rr r)rrrr tensor_floatxs r9r!r!s/ \\^F+FZZA5F,FUM r8c[XU5nXR5RSS5-nURUS9nU$)NrXrLr)r rrr{)rrrrrs r9r&r&s@%VE :F kkm((Q/ /F YY\Y *F Mr8ry hp_value_lb hp_value_ubc>[R"U5Rn[U5S:Xa>UR 5R5nUcUb[R "XtUS9nXv- nO[ XR5upURU 5n U R 5RU SS9nUcUb[R "XtUS9nXv- n[UR5V V s/sH upXU -PM nn n URU5nU[RLaWU[RLdS5e[R"[R![R""U555nUR%[RS9$s sn n f)a Calculates float8 scaling factor for the given high precision tensor. Args: tensor (torch.Tensor): Input tensor to be quantized. float8_dtype (torch.dtype): Data type of the quantized tensor (e.g., torch.float8_e4m3fn, torch.float8_e5m2). scale_dtype (torch.dtype): Data type of the scaling factor (e.g., torch.float32). block_size (Optional[Tuple[int, ...]]): Block size for block-wise quantization. If None, tensorwise quantization is used. hp_value_lb (Optional[float]): the lower bound for high precision floating point value for calculating scale hp_value_ub (Optional[float]): the upper bound for high precision floating point value for calculating scale rrrTrz!Only float8_e8m0fnuz is supportedr)rjrrrrrrrrr enumerater rfloat8_e8m0fnuexp2rdrlog2r{)rrryrrrrmax_absrrrtensor_reshapedrr output_shapes r9rrsd( L)--I :!**,""$  "k&=kk' LG#.C  / +!++&9:!%%',,,N  "k&=kk' LG#>Gv||=T =TMAJQ- '=T   l+%--'e222W4WW2 6<< 5(9:; 88%--8 (( s+F target_shapec ^^TRT:XaT$TR5S:XaT$[S[TRT555(aT$[ TR5[ T5:wa-[ S[ TR5S[ T535e[ UU4Sj[[ T5555n[[TTRU55H+unupEnXEU-:wdM[ SUSUSUS XE- S 3 5e Tn[U5Hup6US:dM URXcS 9nM U$) a~ Expand a scale tensor to match the target tensor shape for block-wise quantization. If this is rowwise quantization, however, just return the scale as is. Args: scale (torch.Tensor): Scale tensor with shape corresponding to block structure target_shape (torch.Size): Target tensor shape to expand to Returns: torch.Tensor: Scale tensor expanded to match target_shape rLc3F# UHupX:H=(d US:Hv M g7f)rLNr.).0abs r9 6_maybe_expand_scale_to_tensor_shape..4 s" G(F16 Q!V (Fs!zScale tensor has z dimensions but target has c3N># UHnTUTRU-v M g7fri)r)rrrrs r9rr= s&3Ka Q5;;q>)3Ks"%z Dimension z: target size z' is not evenly divisible by scale size z (block size would be )r) rrallziprrrrrrepeat_interleave)rr block_sizesr target_dim scale_dimrexpanded_scales`` r9#_maybe_expand_scale_to_tensor_shaper s` {{l"  {{}  GEKK(F GGG  5;;3|,,EKK 011LSQ]M^L_ `  38\9J3KK 3< L%++{33. .J: Z/ /QC~j\:!!* +A*BXAYYZ\  3N";/  >+==j=PN0 r8cUR[R5n[XR5nX4- n[R "U5R nURU*US9n[RXr5$)h Quantizes the high precision floating point tensor to a float8 tensor, using the given scaling factor. r) r{rjrrrrrrrwr)rrry tensor_fp32scale_expanded tensor_scaled max_valuetensor_clampeds r9r"r"T sj))EMM*K9 MN0M L)--I"((iZY(GN    ==r8cUR[R5n[XR5nX4-nURU5$)9 Dequantizes the float8 tensor to high precision tensor. )r{rjrrr)rrr fp8_tensorrrts r9r(r(g s=5==)J9 MN+I << %%r8c[UUUS9$)rrrry)r"rs r9&_quantize_affine_float8_non_decomposedrx s #! r8%quantize_affine_float8_non_decomposedc*[R"XS9$Nrrj empty_likers r9_quantize_affine_float8_metar    F 77r8c[UUUS9$)rrrr)r(rs r9(_dequantize_affine_float8_non_decomposedr s %! r8'dequantize_affine_float8_non_decomposedc*[R"XS9$rrrs r9_dequantize_affine_float8_metar rr8)NN)NNNNNri)r^enumrrtypingrrrrr r rj!torchao.prototype.custom_fp_utilsr r r torchao.utilsrr__all__rrr serializationadd_safe_globals float8_e4m3fn float8_e5m2float8_e4m3fnuzfloat8_e5m2fnuzrrrint16rrKrr__annotations__rArBrCrDrErFrGrVrWr^uint1uint2uint3r/uint5uint6uint7updateint1int2int3int4int5int6int7keysrrrlibraryLibrary quant_libregister_custom_opautogradFunctionrdrwrrno_gradrurrboolrrr rrrrrrrrr$rr%r<r*r+rrrrrrRrrrr#r)r'rBinference_moderdictrrZr`Sizerfrrrrr!r&rrr"r(rrrr)rs0r9rs ??    @$0 d  4  $$k?%CD         KK JJ  KK& KK& TeEKK$=>c3hOPqqqqqqq KK JJ KK KK PT% \ 9:E#s(OKL RTtE%++|"; LL> <<>++> \\ >,!& & LL& <<&++& \\ &"Y&!& 3 3  LL  << ++  \\ ' 9EF!& 3 38 LL8 <<8++8 \\ 8G8Y&!&   LL  << ++  \\ ' 9GH!& 8 LL8 <<8++8 \\ 8I8qF-sAQ