h9tSSKrSSKJrJrJr SSKrSSKJr SSKJ r \RRr "SS\5r "SS\5r "S S \R5rS \S \ S \S\ 4Sjr\R"R$"SS\R&R(55r\R"R$"SS\R&R(55rS\R.S4S\R0S\R0S\R2S\\ S\\S\\4 Sjjr"SS\R05rg)N)Dict NamedTupleOptional)DTensor)to_fp8_saturatedcR\rSrSr%SrSr\\S'Sr\\S'Sr \\S'Sr \\S'Sr g ) ScaledMMConfig.a Configuration for the scaled_mm in the forward and backward pass. Attributes: emulate (bool): Whether to emulate the matmuls in fp32. use_fast_accum (bool): Whether to use the fast-accumulation option for scaled_mm. fp8_output (bool): Whether to output the result of the scaled_mm in fp8. pad_inner_dim (bool): Whether to pad the inner dimension of a and b with 0s. This is needed for matmuls not aligned to 16. Femulateuse_fast_accum fp8_output pad_inner_dimN) __name__ __module__ __qualname____firstlineno____doc__r bool__annotations__r r r__static_attributes__r_/home/james-whalen/.local/lib/python3.13/site-packages/torchao/float8/float8_training_tensor.pyr r .s1 GT ND JM4rr cz\rSrSr%Sr\"SSSS5r\\S'\"SSSS5r\\S'\"SSSS5r \\S'Sr g ) LinearMMConfig@a Configuration for different gemm operations in LinearMM. This configuration is not user-facing and exists for convenience, allowing Float8TrainingTensor to use the right config based on which gemm from gemms with outputs `output`, `grad_input`, `grad_weight` is being called. Attributes: output (ScaledMMConfig): Configuration for the output gemm. grad_input (ScaledMMConfig): Configuration for the grad_input gemm. grad_weight (ScaledMMConfig): Configuration for the grad_weight gemm. FToutput grad_input grad_weightrN) rrrrrr rrrrrrrrrr@sH ,E4FFNF!/ueU!KJK"0ue"LKLrrc$\rSrSrSrSrSrSrSrg) GemmInputRoleSz Given a Float8TrainingTensor, the enum below describes the expected role of this tensor in the three gemms present in the fw + bw pass of a Linear layer. This is used to choose the right config for a float8 gemm when the gemm is performed. inputweight grad_outputrN) rrrrrINPUTWEIGHT GRAD_OUTPUTrrrrr!r!Ss E FKrr!a_rolea_linear_mm_configb_roleb_linear_mm_configcU[RLaZU[RLaGURUR:Xd!SURSUR35eUR$U[RLaZU[RLaGUR UR :Xd!SUR SUR 35eUR $U[RLaZU[RLaGUR UR :Xd!SUR SUR 35eUR $[SUSU35e)Nz"linear_mm_config.output mismatch: z vs z&linear_mm_config.grad_input mismatch: z'linear_mm_config.grad_weight mismatch: zunexpected a_role z and b_role )r!r&r'rr(rrAssertionError)r)r*r+r,s rchoose_scaled_mm_configr/as` $$$=3G3G)G!((,>,E,EE 01C1J1J0K4PbPiPiOj k E"((( =,, ,=;O;O1O!,,0B0M0MM 45G5R5R4SSWXjXuXuWv w M",,, =,, ,=;N;N1N!--1C1O1OO 56H6T6T5UUYZlZxZxYy z O"---1&fXNOOrc\rSrSrSr\S\RS4S\RS\RS\RS\ \ S\ \S \ \ 4 S jj5r\S 5rS rg) _ToFloat8ConstrFunczz A differentiable conversion to fp8. * forward: convert from high precision to float8 * backward: pass the gradient without changes Ntensorscale float8_dtypelinear_mm_configgemm_input_role axiswise_dimc UR[R5U-n[Xs5n[ U[ 5(a[ U[ 5(dS5eUR n URn UR5n UR5n [U U URUUUS9n [ R"U U U SUR5UR5S9$[UUURUUUS9$)a This function will apply the scaling, and then convert to a Float8TrainingTensor Note: We will call this function with a DTensor subclass. Ideally this would be an aten OP that DTensor could overload to ensure proper semantics. There are some techincal issues with that composing with FakeTensor, so we special case here. DTensor Invariant: DTensor must always be the outer most tensor subclass z>Expected Float8 scale to be a DTensor if bits_fp8 is a DTensor)r6r7r8F) run_checkshapestride)totorchfloat32r isinstancer device_mesh placementsto_localFloat8TrainingTensordtype from_localsizer<)ctxr3r4r5r6r7r8 tensor_scaledbits_fp8 bits_meshbits_placements local_bits local_scaleinner_float8_tensors rforward_ToFloat8ConstrFunc.forwards. %--058 #M@ h ( (eW-- P -!,,I&11O!**,J..*K"6 !1 /) # %%#mmo(  $   LL-+%   rcUSSSSS4$NrrHgs rbackward_ToFloat8ConstrFunc.backwards$dD$..rr)rrrrr staticmethodr!r&r>TensorrErrintrPrVrrrrr1r1zs  6:3@3F3F&*9  9 ||9 kk 9 #>2 9 "-0 9 sm9 9 v//rr1c8\rSrSrSr\S5r\S5rSrg)_FromFloat8ConstrFuncz A differentiable conversion from fp8. * forward: convert from float8 to high precision * backward: pass the gradient without changes cfURRUR5UR- $rS)_datar= _orig_dtype_scale)rHr3s rrP_FromFloat8ConstrFunc.forwards$||v112V]]BBrc USS4$rSrrTs rrV_FromFloat8ConstrFunc.backwards$}rrN) rrrrrrXrPrVrrrrr\r\s1 CCrr\ hp_tensorsr5r6r7r8c0[RXX#XE5$)a Given a high precision tensor `hp_tensor` and a precalculated scale `s`, scales `hp_tensor` by `s` and returns a `Float8TrainingTensor` of the result. Autograd-aware, the derivative is pass-through. DTensor-aware, if the input is a DTensor the output will be DTensor(Float8TrainingTensor). Args: hp_tensor: the tensor to convert s: the scale to use to convert the tensor float8_dtype: the float8 dtype to use linear_mm_config: Defines the configuration for the scaled_mm for the 3 fwd/bwd gemms of linear gemm_input_role: Defines the role of this tensor (input, weight or grad_output) in the 3 fwd/bwd gemms of linear axiswise_dim: for rowwise scaling, contains the axis scaled across )r1apply)rerfr5r6r7r8s rhp_tensor_and_scale_to_float8ris2  $ $lo rc\rSrSr%Sr\R \S'\R \S'\R\S'\ \S'\ \S'\ \ \S'/S Qr \ RS 4S \R S \R S \RS\ \ S\ \ S\ \ 4 SjjrSrSr\S\4Sj5rSr\SSj5r\R0R2rSrg )rDa Note: this is **not** a public API and is only intended to be used inside of this repository. Please file an issue if you would benefit from this being a public API. A Python-only Float8 tensor subclass. Contains: * `_data`: the underlying e4m3 or e5m2 data * `_scale`: the scale used to scale the original fp32 tensor. We multiply by scale to go from fp32 range to fp8 range, and divide by scale to go from fp8 range to fp32 range. Scale is guaranteed to have a shape compatible with `_data`. For example: - if scaling is tensorwise, `_scale` is a scalar tensor - if scaling is axiswise and _data.shape is [3, 5], `_scale` could have shape [1, 5] or [3, 1]. `axiswise_dim` defines the scaling axis. - if scaling is axiswise and _data.shape is [2, 3, 5], `_scale` could have shape [1, 1, 5] or [2, 1, 1]. `axiswise_dim` defines the scaling axis. Non-one entries which are not the first or last element are not supported. * `_orig_dtype`: the original dtype of the tensor used to create this tensor. * `_axiswise_dim`: for axiswise scaling only, contains the axis scales across. Only values of 0 or -1 are supported. Intended usage of this abstraction: 1. to bundle raw data + fp8 metadata together for easy passing through Python PyTorch systems. 2. Float8-aware user code can use the private fields on these tensors to call into float8 operations. 3. Float8-agnostic user code can use these tensors as is - they will convert to original precision in `__torch_dispatch__`. r_rar`_linear_mm_config_gemm_input_role _axiswise_dim)r_rar`rlrmrnNdatar4 orig_dtyper6r7r8c b[RRUUR5UR 5UR 5UUR URURS9nXl X'l X7l UbUO [5Ul XWlUS;d SU35eXglU$)N)stridesstorage_offsetrElayout requires_graddevice)Nrzunsupported axiswise_dim )r>rY_make_wrapper_subclassrGr<rsrtrurvr_rar`rrlrmrn)clsror4rpr6r7r8selfs r__new__Float8TrainingTensor.__new__"s||22  IIKKKM..0;;,,;;3    % 0 < .BR !0},X0I,.XX,) rc SURRSURSURSURSUR SUR 53 $)NzFloat8TrainingTensor(lp_dtype=z, scale=z, linear_mm_config=z, axiswise_dim=z gemm_input_role=z as_orig_prec=)r_rErarlrnrmto_original_precisionrzs r__repr__Float8TrainingTensor.__repr__As/ 0@0@/A$++VijnkAkAjBBQRVRdRdQeewx|xMxMwNN]^b^x^x^z]{| |rclURURURURS.nSS/U4$)Nr`rlrmrnr_rar)rzrHs r__tensor_flatten__'Float8TrainingTensor.__tensor_flatten__Ds?++!%!7!7 $ 5 5!//   "C''r inner_tensorsc h[U5S:Xde[USUSUSUSUSUS5$)Nr_rar`rlrmrn)lenrD)rmetadata outer_size outer_strides r__tensor_unflatten__)Float8TrainingTensor.__tensor_unflatten__MsT=!Q&&&# ' " ( # ] # ( ) ' ( _ %   rc,[RU5$rS)r\rhrs rr~*Float8TrainingTensor.to_original_precisionYs$**400rc^^SSKJn U4Sjm[U4SjU55(d[$X;a XQ"XU5$[ SUS35e)Nr)FLOAT8_OPS_TABLEc>[TU5=(dc [[RRRU5=(d. [[RR R U5$rS) issubclassr> _subclasses fake_tensor FakeTensorfunctional_tensorFunctionalTensor)typerys rallowed_subclassesCFloat8TrainingTensor.__torch_dispatch__..allowed_subclassesisW3%e//;;FFM%%77HH$ rc34># UH nT"U5v M g7frSr).0trs r :Float8TrainingTensor.__torch_dispatch__..rs8%Q%a((%szattempting to run z, this is not supported)torchao.float8.float8_opsrallNotImplementedNotImplementedError)ryfunctypesargskwargsrrs` @r__torch_dispatch__'Float8TrainingTensor.__torch_dispatch__\sR ?  8%888! !  ##)$f= =!$6tfrYrrErr!rrZ __slots__r&r{rrrXrrr~ classmethodr_C_disabled_torch_function_impl__torch_function__rrrrrDrDs@ << LL%%##C= I4A3F3F&*ll||KK  #>2  "-0 sm>|(  D    1VV:??rrD)enumtypingrrrr>torch.distributed._tensorrtorchao.float8.float8_utilsropsatenr rEnumr!r/_dynamoallow_in_graphautogradFunctionr1r\r&rYrErZrirDrrrrs`  -- - yy~~< Z $MZM&  DII  P P&P P' P2E/%..11E/E/P ENN33  (26/