ȅiB SSKrSSKrSSKrSSKrSSKJr SSKJrJr SSK r SSK r SSK J s J r SSKJr SSKJr SSKJr SSK JrJrJr SS KJr \R4"\5r"S S \5r"S S \5r\R>S\4Sj5r S\!S\4Sjr"S\RFS\4Sjr$S6S\ RJS\&S\&4Sjjr'S6S\ RJS\&S\&4Sjjr(S\RFS\&4Sjr)S\RFS\&4Sjr*"SS\5r+"SS\5r,"SS\5r-S /S!//r.S"/S"//S#/S#//S$/S%///r//S&Q/S'Q/S(Q/S)Q/r0S\\14S*jr2S+\&S,\&S-\S\14S.jr3S\RFS\14S/jr4S0\ RjRlS\&4S1jr7S0\ RjRlS\&4S2jr8S7S0\ RjRlS3\\&S4\9S\14S5jjr:g)8N)IntEnum)AnyOptional) size_hint)normalize_function)ir)get_dtype_sizesnode_args_kwargs sympy_product)Vc(\rSrSrSrSrSrSrSrSr g) NCCL_COLLrrN) __name__ __module__ __qualname____firstlineno__ ALL_REDUCE ALL_GATHERREDUCE_SCATTER ALL_TO_ALL UNSUPPORTED__static_attributes__rW/home/james-whalen/.local/lib/python3.13/site-packages/torch/_inductor/comm_analysis.pyrrsJJNJKrrc$\rSrSrSrSrSrSrSrg)NVIDIA_GPU_TYPErrrrrN) rrrrVOLTAAMPEREHOPPER BLACKWELLrrrr r"r"s E F FIrr"returnc^[RRR[RRR5=(d SmST;a[ R $ST;a[ R$ST;a[ R$[U4SjS55(a[ R$[ R$)NV100A100H100c3,># UH oT;v M g7fNr).0gpugpu_infos r get_gpu_type../s A(@H_(@)B100B200B300) torchutils collect_env get_gpu_inforunr"r$r%r&anyr')r2s@r get_gpu_typer?&s{{&&33EKK4K4K4O4OPVTVH $$$ 8 %%% 8 %%% A(@ A A A(((%%%r kernel_namec^TceST;a[R$ST;a[R$ST;a[R$[ U4SjS55(a[R $[R $)N all_reduce all_gatherreduce_scatterc3,># UH oT;v M g7fr/r)r0commr@s r r37get_collective_type_from_kernel_name..>s H-GT[ -Gr5) all_to_allalltoall)rrrrr>rr)r@s`r $get_collective_type_from_kernel_namerJ6su  "" "{"###  $### [ (''' H-G H H H###$$$rnodec[U[R5(d[SU35eURnUce[ U5$)Nz!node is not a collective kernel: ) isinstancer _CollectiveKernel ValueErrorpython_kernel_namerJ)rKnames r get_collective_typerRDsJ dB00 1 1>7v>??rc \rSrSrSrSrSrSrg)NCCL_HWprrrrN)rrrrNVLINKPCINETrrrr rvrvps F C Crrvc\rSrSrSrSrSrg) NCCL_ALGOvrrrN)rrrrTREERINGrrrr r|r|vs D Drr|c\rSrSrSrSrg) NCCL_PROTO{rrN)rrrrLLrrrr rr{s  Brrg333333@gffffff@g333333??g@g@)C@rgffffff4@)gU@g6@g3@)ga@gF@gA@)gq@gV@gQ@cRURnUce[USS5nURSnSSKJn U"U5n[ R RU5n[ R"SU35n[U5n[U5upSU;a U SSU S-n [ R RXWS 9n U"U 0U D6n [ RRRRU 5 SSS5 W R n U S:agU S - nU$!,(df  N(=f) NrPr*ror)_resolve_process_groupzcuda:all_gather_into_tensor_outr)groupdevice@@)rKgetattrrrrqrr9 distributedget_rankrevalr _time_estimatorops_c10d_functional wait_tensordefaultestimated_time)snodekernelpy_kernel_namepg_namerpgrankrfnargskwargstime_estimatorw est_time_us est_time_mss r /estimate_nccl_collective_runtime_nccl_estimatorrs" ZZF   V%92>N""2&GI  (B!!**2.D\\E$. )F n B$U+LD$~5ABx$q'!    * * * C~    ""..66q9 D!//KQ#K  D Cs :L!Q;VaZAF#q[\aFvv&GaKWBI NE  !ty/C/C'C9) U E  y###fqj! %% %fqj! )**I,@,@A A!6\V #E I#c/nnG y### A:f*KK )**I,@,@)BVBVW Wqj i ,GW~i(4HW[[!),Z8HK z 8[)H $0;3IIIG3J*,??L  "B cB Irc\[U5n[U5n[U5n[XU5$) Returns estimated NCCL collective runtime in nanoseconds (ms). The following heuristics are copied from https://github.com/NVIDIA/nccl/blob/master/src/graph/tuning.cc. We aim to estimate the runtime as accurately as possible. Assumptions: - only ring algorithm (NCCL_ALGO_RING) is used - only Low-Latency protocol (NCCL_PROTO_LL) is used, i.e. Simple or LL128 is not used - 8 gpus per node # TODO: Need to find a way to get accurate "gpus per node" and "# nodes" info. - collective is one of: allreduce, reducescatter, allgather )rkrtrRr)rKrrrs r estimate_nccl_collective_runtimerRs6!@ E*40J t $D 0!t rfx_nodec^^SmURURp![U5nURSS5 S[R S[ 4SjmS[RR4UU4Sjjn[R"[RRUX45 URRSS5nTb[U[R 5(dg T"U5nTU-$) zSEstimate the size of a collective operation in bytes, including inputs and outputs.Nouttr(c`[UR55[UR5-$r/)rdrSr rh)rs r tensor_bytes1estimate_fx_collective_size..tensor_bytesqs!%affh/.2IIIrrjc>URRSS5n[U[R5(dgTcSmTT"U5- mg)Nvalr)metagetrMr9Tensor)rjinp_val input_bytesrs r add_inp_bytes2estimate_fx_collective_size..add_inp_bytestsG((,,ud+'5<<00   K|G,, rrr)rrdictpopr9rrYfxNodepytree tree_map_onlyrrrM)rrrr output_val output_bytesrrs @@r estimate_fx_collective_sizergsK<<& &\F JJudJ JJ-588==--     !!%.J*Z"F"F +L  %%rcLSSKJn [U5nU"U5(dU$US-$)zEstimate the memory footprint of a collective operation in bytes. This returns the total bytes that need to be live concurrently in memory. For all_reduce, we divide by 2 since it can be done in-place. r)is_all_reduce_tensorr)#torch._inductor.fx_passes.bucketingrr)r is_all_reducerSs r 'estimate_fx_collective_memory_footprintrs,  'w /D$W--4<419^^SSKJnJn U"T5n[RR R U5S:XagU"U5nURU5nUR(dg[R"TT45upVS[R4U4SjjmS[RS[4SjnS[S[4UU4S jjmUVs/sH nT"U5PM nn[R "XV5upTR"n [%U [R&R(5(de[RR+US 9n U "U 0U D6n [R,R.R0R3U 5 SSS5 W R4nUS:agUS - nU$s snf!,(df  N-=f) Nr)_get_pg_default_devicerfaker(c<>[R"TcUOT/UUS9$)N)rhr)r9empty)rSrhrrs r _tensorVestimate_nccl_collective_runtime_from_fx_node.._nccl_estimate.._tensors&;;%-M? rscR[RRRUSS9$)NrrV)r rZr[r)rs r try_size_hint\estimate_nccl_collective_runtime_from_fx_node.._nccl_estimate..try_size_hints"77##--a!-< [U[RR5(aT"URS5$[U[R 5(a6T"[ UR55/URUR5$U$)Nr) rMr9rrrrrdrSrhr)rrto_real_tensors r r]estimate_nccl_collective_runtime_from_fx_node.._nccl_estimate..to_real_tensorsg!UXX]]++%affUm44!U\\** 6qvvx @A177AHHUUHr)rr)rqrrr9rdistributed_c10d get_backend _get_backendsupports_time_estimater tree_flattenrrWExprrYrtree_unflattentargetrM_ops OpOverloadrrrrrr)rrrrbackend flat_argsflat_args_pytree_specra real_args real_kwargsrrrrrrrrrrrrs @@r _nccl_estimateEestimate_nccl_collective_runtime_from_fx_node.._nccl_estimates $J /    - - 9 9" = G'+//&)--+1+>+>f~+N(  ELL  =UZZ =C = c c  1:: 1^A& :!'!6!6y!X ^^"ejj334444    . .R . 8NI--A II & & 2 2 : :1 =9%33  ?!C' ; 9 8sF1r+s{  $$;8CC!g  &o & & %c %i %6bii6I6@@s@S@sS "))@BII@#@ g               ,   ) 8huoBe"e03e;De eZ299*$&$&3$&N =UXX]] =s = $(#b XX]]bC=bb br