h(SSKrSSKJr SSKJr SSKJr SSKJrJ r J r SSK J r J r SSKJr SSKJrJr SSKJr S r"S S \5r"S S \ 5r"SS\ 5rg)N)DTensor) DeviceMesh)ColwiseParallelPrepareModuleInputRowwiseParallel) ScalingType e4m3_dtype)tensor_already_casted_to_fp8)NoopFwToFloat8BwDynamichp_tensor_to_float8_dynamic) GemmInputRolecUR[R:H=(a UR[R:H$)N)scaling_type_inputrDYNAMICscaling_type_grad_output)ms _/home/james-whalen/.local/lib/python3.13/site-packages/torchao/float8/float8_tensor_parallel.py(_float8_linear_supports_float8_allgatherr"s6  3 33 > & &+*=*= =c^\rSrSrSr\S5r\S5rS\RS\ S\R4U4Sjjr S r U=r $) Float8ColwiseParallel*zc Like `ColwiseParallel`, but with all-gather in float8. This currently assumes tensorwise scaling. c4USn[U[5(d[R"XTUSS9n[U5(dB[ UUR R RUR[RS9nX:waURUSS9nU$NrF run_checkgemm_input_roleT placementsasync_op isinstancer from_localr r configcast_config_input target_dtypelinear_mm_configr INPUT redistribute input_layoutsdesired_input_layoutsmodinputs device_mesh input_tensors r_prepare_input_fn'Float8ColwiseParallel._prepare_input_fn0s ay ,00"--=EL,L996 ,,99$$ - 3 3 L  1'44045LrcURU:waURUSS9n[R"UURUR R R5nU(aUR5$U$NTr r r*r applyr(r%cast_config_grad_outputr'to_localoutput_layoutsuse_local_outputr.outputsr0s r_prepare_output_fn(Float8ColwiseParallel._prepare_output_fnJsu    /**)D+G *//   JJ . . ; ; &6w!B7Brmoduler0returnc>SSKJn [X5(d[S[ U535e[X5(a[ U5(d [ S5e[TU]!X5$Nr Float8Linearz.Expecting module to be Float8Linear but found unsupported torchao.float8.float8_linearrEr# ValueErrortyperAssertionErrorsuper_applyselfr@r0rE __class__s rrMFloat8ColwiseParallel._apply\c=&//@fO    :6BB / /w~f22r__name__ __module__ __qualname____firstlineno____doc__ staticmethodr2r>nnModulerrM__static_attributes__ __classcell__rPs@rrr*sY 2CC" 3RYY 3Z 3BII 3 3rrc^\rSrSrSr\S5r\S5rS\RS\ S\R4U4Sjjr S r U=r $) Float8RowwiseParallelkzc Like `RowwiseParallel`, but with all-gather in float8. This currently assumes tensorwise scaling. c4USn[U[5(d[R"XTUSS9n[U5(dB[ UUR R RUR[RS9nX:waURUSS9nU$rr"r+s rr2'Float8RowwiseParallel._prepare_input_fnqsay ,00"--=EL,L996 ,,99$$ - 3 3 L  1'44045LrcURU:waURUSS9n[R"UURUR R R5nU(aUR5$U$r5r6r:s rr>(Float8RowwiseParallel._prepare_output_fnsp    /**nt*TG*//   JJ . . ; ; &6w!B7Brr@r0rAc>SSKJn [X5(d[S[ U535e[X5(a[ U5(d [ S5e[TU]!X5$rCrGrNs rrMFloat8RowwiseParallel._applyrRrrSrTr_s@rraraksY .CC" 3RYY 3Z 3BII 3 3rrac^\rSrSrSrSSSSS\R SS.U4SjjrSrS\ RS \ S \ R4U4S jjr S r U=r$) PrepareFloat8ModuleInputa Like `PrepareModuleInput`, but with all-gather in float8. This currently assumes tensorwise scaling. The only difference from `PrepareModuleInput` is that after we prepare the input DTensor, we cast the input to DTensor(Float8TrainingTensor) This is to ensure the float8 cast happens before the all-gather (i.e. Shard -> Replicate) so that if there are multiple float8 users of the input activation, we perform fp8 allgather only once. FP8 Args: float8_dtype (torch.dtype, optional): control what float8 dtype to cast to when prepare the module input, we currently only support torch.float8_e4m3fn. default: torch.float8_e4m3fn fwd_config_submodule_fqn (str, optional): the fqn of the submodule that contains the forward config used for the float8 cast. If not specified, we will search for the Float8Linear in the submodules and use the forward config from that module, in this case all module's forward config must be the same. NF)r,r-input_kwarg_layoutsdesired_input_kwarg_layoutsr< float8_dtypefwd_config_submodule_fqnc>[TU]UUUUUS9 X`lSUlXplUR[ R :wa [S5eg)N)r,r-rlrmr<zFPrepareFloat8ModuleInput only support casting to float8_e4m3fn for now)rL__init__rnr(rotorch float8_e4m3fnNotImplementedError) rOr,r-rlrmr<rnrorPs rrq!PrepareFloat8ModuleInput.__init__se '"7 3(C-  ) $(@%    3 3 3%X  4rc|Ub[U[5(aUnO=[U[R5(dS5e[R"XU4SS9n[ U[ UR[RS9nUbX4:waURU4S9nUR(aUR5$U$U$)Nz%expecting input to be a torch.Tensor!Frr)r ) r#rrrTensorr$r r r(r r)r*r<r9)rOinputmesh input_layoutdesired_layoutdt_inps r_prepare_input_arg+PrepareFloat8ModuleInput._prepare_input_args  #%))!%66;6!++,E1%% - 3 3 F )l.L,,8I,J(,(=(=6??$ I6 ILrr@r0rAc>SSKJn URb?URUR5n[ XC5(deUR UlOjUR 5HVn[ XS5(dMUR cUR UlM5UR UR :XaMQS5e UR ce[TU]!X5 U$)NrrDz?All the Float8Linear modules should have same linear_mm_config!) rHrEro get_submoduler#r(modulesrLrM)rOr@r0rE fwd_linearr.rPs rrMPrepareFloat8ModuleInput._applys=  ( ( 4--d.K.KLJj77 77$.$?$?D !~~'c00,,4030D0D-#448L8LL]L ($$000 v+ r)rnror()rUrVrWrXrYrrrsrqr}r[r\rrMr]r^r_s@rrjrjs[*" $(((!%:6RYYZBIIrrj)rrtorch.nnr[torch.distributed._tensorrtorch.distributed.device_meshr!torch.distributed.tensor.parallelrrrtorchao.float8.configrr torchao.float8.distributed_utilsr #torchao.float8.float8_scaling_utilsr r %torchao.float8.float8_training_tensorr rrrarjrSrrrs^  -4 :I@>3O>3B<3O<3~_1_r