hP!<SrSSKJrJrJrJrJr SSKrSSKJ r J r SSK J r SSK JrJr SSKJrJr \R&r"SS \5rS \S \S \S \\\44SjrS\R&S\\S44SjrS(S \S\S \S\S\R0S\\S\\S\S \4SjjrS\R&S\\S\S\S\S\S \R&4S jrS!\R&S \4S"jrS!\R&S \4S#jrS$\\\ \\ \ 4\\ 4S \\ \ 44S%jrS&\\ \ 4S S4S'jr g))z; Defines an nn module designed to be used during inference )List NamedTupleOptionalTupleUnionN) is_row_majorpad_tensor_for_matmul)FP8Granularity)PerRow PerTensor)is_MI300is_sm_at_least_89cD\rSrSr%SrSr\\S'Sr\\S'Sr \\S'Sr g) Float8MMConfiga{ Configuration for the scaled_mm in the forward and backward pass. Attributes: emulate (bool): Whether to emulate the matmuls in fp32. use_fast_accum (bool): Whether to use the fast-accumulation option for scaled_mm. pad_inner_dim (bool): Whether to pad the inner dimension of a and b with 0s. This is needed for matmuls not aligned to 16. Femulateuse_fast_accum pad_inner_dimN) __name__ __module__ __qualname____firstlineno____doc__rbool__annotations__rr__static_attributes__rR/home/james-whalen/.local/lib/python3.13/site-packages/torchao/float8/inference.pyrrs'GT ND M4rra_datab_datascaled_mm_configreturncUR(acURS5URS5:Xd+SURS5SURS535e[USS9n[USS9n[UR 55(dUR 5n[UR 55(a,UR 5R 5R 5nX4$)zPreprocess the inner fp8 data tensors for admmm Args: a_data: Input tensor A. b_data: Input tensor B. scaled_mm_config: Configuration for _scaled_mm. Returns: Preprocessed tensors A and B in the format for _scaled_mm. rz"Inner dims must match for mm, got z and )dims)rsizer rstride contiguoust)r r!r"s rpreprocess_datar+,s%%{{1~Q/ 0Q0@fkkRSnEU V /'vA6&vA6   ( (""$FMMO$$&&(**, >r input_scale input_shape.cUR5S:XaURSS5$URS5nUR5S:aURSURS5nU$)z:Ensures input tensor is correctly formatted for _scaled_mmr%)numelreshape unsqueezedimshape)r,r-s rpreprocess_scaler6Fsna""1a((''+K1!))"k.?.?.CD ra_scaleb_scale output_dtype output_scalebiasrc U[R:Xa!Ub[R"UUUUUUUS9nX-$[R"UUUUUUUUS9$)z This is the unwrapped version of addmm_float8, which does not take in Float8TrainingTensors as inputs. This is used to standardize the logic between subclassed and non subclassed versions of the linear module. )scale_ascale_b scale_result out_dtyper)r=r>r;r?r@r)torchfloat32 _scaled_mm) r r7r!r8r9r:r;routputs r addmm_float8_unwrapped_inferencerEXso u}}$)9!!  %") }    !%  rscale data_shaper4startendstepc^^[RRnTRT:XaURR TX#XE5$[ UU4Sj[[T5555nU[U5:aT$XrnUS:XaURR TX#XE5$UbX8-OSn Ub XH-S- U-OSn US:a [S5eURR TX)U S5$)z Slice the scale tensor appropriately based on the data tensor slicing. This function calculates how the scale should be sliced when the data tensor is sliced along a given dimension, taking into account the block structure. c3N># UHnTUTRU-v M g7f)N)r5).0irGrFs r -_slice_scale_for_dimension..s$XAWA 1 Q7AWs"%r%Nz;Slicing with step > 1 is not implemented for scale tensors.) rAopsatenr5sliceTensortuplerangelenNotImplementedError) rFrGr4rHrIrJrR block_sizesblock_size_for_dim scale_start scale_ends `` r_slice_scale_for_dimensionr]s 99>>D {{j zz  C>>Xs:AWXXK c+ $)Qzz  C>>6;5Fe1D  % ).@ @  !8%M zz  )QGGrxc[US5(dS5e[UR5SUR5S- -URS4-:H$)rChecks if a quantized tensor is rowwise scaled Args: x: quantized tensor (should have `block_size` attribute) block_size.Expecting input to have `block_size` attribute)r%r%r/)hasattrrUrar4r5r^s r_is_rowwise_scaledresO 1l # #U%UU #  $!%%'A+"6!''"+"G GGrc^[TS5(dS5e[U4Sj[TR555$)r`rarbc3># UH=nTRUS:H=(d TRUTRU:Hv M? g7f)r/N)rar5)rMrNr^s rrO(_is_tensorwise_scaled..s>HU1 Q2>aAGGAJ!>> sAA)rcallrVndimrds`r_is_tensorwise_scaledrks@ 1l # #U%UU # HMaff  r granularitycSnUc[5[54nU$[U[[45(aX4nU$[U[[45(a[ U5S:Xa[US[[45(a[US[[45(d[ SUS35e[US[US55(d[ SUS35e[U5nU$[ SUS35e)Nr0rr%zInvalid granularity types: ), only PerTensor or PerRow are supported.zEDifferent granularities for activation and weight are not supported: z#Invalid granularity specification: )r isinstancer rUlistrW ValueErrortype)rlprocessed_granularitys r_normalize_granularityrts!!!*ik :( ! ' K)V!4 5 5!, :$ ! # K% / /C 4D4I {1~ 6': ; ;;q>Iv+>??-k]:cd +a.${1~*>??WXcWdeNO !&k 2 ! 1+>g h  r granularitiescUHTn[U[[45(d[SUS35e[ 5(aM>[ 5(aMOS5e g)a Validate that the hardware supports the requested granularities. Args: granularities: Tuple of (activation_granularity, weight_granularity) Raises: AssertionError: If hardware doesn't support the requested granularity ValueError: If invalid granularity type is provided zInvalid granularity type: rnuNFloat8 dynamic quantization requires CUDA compute capability ≥8.9 or MI300+.N)ror r rqrr )ru _granularitys r_check_hardware_supportrxs[& ,F(;<<,\N:cd !""hjj \ 0 &r)NNF)!rtypingrrrrrrAtorchao.float8.float8_utilsrr torchao.float8.typesr torchao.quantization.granularityr r torchao.utilsr rrTrr+intr6dtyperrEr]rerkrprtrxrrrrs <; K/   Z    % 66> 4%,,U38_0&*! % % % % % ++ % 6" % 6 %% %P0H <<0HS 0H 0H  0H 0H  0H \\0HfH%,,H4HU\\d!  ..0 1   " ! >> )*!D 78   r