ȅiGfSSKrSSKrSSKJr SSKJr SSKrSSKrSSKJ s J r SSK J r SSK Jr SSKJrJrJrJrJrJrJr SSKJr SSKJr SS KJr SS KJr \R>"\ 5r!S \RDRFS \$\S \%S\4Sjr&"SS\ RN5r("SS\5r)g)N)Callable)Any)ir)KernelTemplate)Buffer FixedLayoutget_free_symbolsget_symbolic_inputsgm_original_output_stridesir_node_to_tensorLayout) benchmarker)do_bench_using_profilingV) OrderedSetgminputsnamereturncSSKJn [RRnU[RlU"X5U[Rl$!U[Rlf=f)aInline a subgraph by converting its FX operations to individual IR nodes. This converts a subgraph to multiple ComputedBuffer nodes (fusable), enabling epilogue fusion with subsequent operations. Returns: TensorBox containing the final operation result as individual IR nodes r)process_subgraph_nodes)torch._inductor.loweringrrgraphmodule)rrrroriginal_modules Z/home/james-whalen/.local/lib/python3.13/site-packages/torch/_inductor/codegen/subgraph.pyinline_subgraph_to_ir_nodesrsF@ggnnO)%b1(s AA+c^\rSrSrSrSS\S\\S\S\S\ S \ 4S \ \ \ \ /\ R44S-S S4U4S jjjrS \\ 4S jrS\ S \ 4S\ \\ 4S S4SjrS \4SjrS \ 4SjrS\\ S\ RS \4SjrS\\ S\ RS S4SjrS \4SjrS \R24SjrS \ \\ 44SjrS \4SjrSrU=r$)SubgraphChoiceCaller4z Represents a Subgraph Autotuning choice, and the subgraph can be any arbitrary GraphModule. Compiles the Subgraph down to a module for benchmarking. Nr input_nodeslayout description make_fx_graph. input_gen_fnsrc >[T U]XX45 /Ul[R [ UR 5Hupx[[UR5SS95S:Xde[[UR5SS95S:XdeURR5 Ub*Xv;a%URRXg"U55 MURR[U55 M SSS5 U"UR6Ul[!UR5 [#UR 5UlUR'5UlSUl0UlSUlg!,(df  N|=f)NT) unbacked_onlyr)super__init__example_inputsr fake_mode enumerater"lenr get_size get_stridedata freeze_layoutappendr rr r sym_inputs_compute_sym_input_valuessym_input_values decompositiondecomposition_kwargs_compiled_module) selfrr"r#r$r%r&iinp __class__s rr*SubgraphChoiceCaller.__init__:s< F@  [[#D$4$45+CLLN$OPTUUUU+CNN,.>? $ > > @9=46!%)3[s CE77 Fc [URVs/sH"n[US5(dMURPM$ sn5n0n[ UR UR 5HupE[U[R5(dM&[ UR5UR5Heupg[U[R5(a[U5X6R'M=[U5U;dMN[U5U[U5'Mg M /nURHn [U [R5(a/U RU;aUR!X9R5 MQ["R$R&R(R+U 5n UR!U b [U 5OS5 M U$s snf)aExtract concrete dimension values for sym_inputs from example_inputs. The compiled module expects symbolic dimension values as runtime arguments. This maps each symbolic variable to its concrete value from the example tensors. Used for range based autotuning. r)rr4hasattrrzipr"r+ isinstancetorchTensorr/shapesympySymbolintstrr3rrsizevars shape_env size_hint) r:ssym_input_namessym_name_to_valueinp_node example_inpsym_dim actual_dimresultsym_varhints rr5.SubgraphChoiceCaller._compute_sym_input_valuesas\%!__ C_60BVQVV_ C  -/%()9)94;N;N%O !H+u||44+.x/@/@/BKDUDU+V'G!'5<<88:=j/),,7W8:=j/)#g,7 ,W&PG'5<<00W\\EV5V / =>ww''11;;GD 4+;c$iC '  ) Ds GGr7kwargscXlX lg)zHCache decomposition function and kwargs for range-based dispatch lookup.N)r7r8)r:r7rYs rcache_decomposition(SubgraphChoiceCaller.cache_decompositions+$*!c"SURS3$)NzSubgraphCaller()rr:s r__str__SubgraphChoiceCaller.__str__s  1--r]c SSKJn URRSS5RSS5nU"URUR [ RR[ RR[ RR[ RR[ RR[ RRSU3S9 nURH@nXCRUR'UR R#UR5 MB [ R$"U5 [&R("SSS S 9 UR*"UR 6 UR-5sS S S 5 sS S S 5 $!,(df  O=fS S S 5 g !,(df  g =f) zCCompile the subgraph for benchmarking, returns the compiled module.r) GraphLoweringz::_. benchmark_) rr+rL cpp_wrapperaot_modeextern_node_serializer is_inference is_backwardrFATEN) max_autotunemax_autotune_gemmmax_autotune_gemm_backendsN)torch._inductor.graphrerreplacerr+rr _shape_envrirjrkrlrmr4 graph_inputsgraph_input_namesr3set_graph_handlerconfigpatchruncompile_to_module)r:re safe_namebm_graph_loweringsym_inps r_compile_for_benchmarking.SubgraphChoiceCaller._compile_for_benchmarkingsK7II%%dC088cB )ww..gg((++WW%%#$77#A#A--++i[)  G;B * *7<< 8  / / 6 6w|| D' !2 3""'+1 "%%t':':;(::< 4 34 3 3s$F<%)F! F<! F/ +F<< G argsoutc>^^^URcUR5UlURRmURm[R (a[ UUU4Sj5$[R"UUU4Sj[R"/TQTQ76S9$)zBRegular benchmarking: compile and use benchmarker with warmup/rep.c>T"/TQTQ5$Nrbm_funcr4sr0SubgraphChoiceCaller.benchmark..sGT"/TQTQ5$rrrsrrrsG0j0401r])device) r9rcallr6rx/profile_bandwidth_with_do_bench_using_profilingrr benchmark infer_device)r:rrrr4s `@@rrSubgraphChoiceCaller.benchmarks}  ($($B$B$DD !'',,**  A A+,QR R$$ 1++?Z?$?  r]cURcUR5UlURR/URQUQ5 g)zFRun once for collective benchmarking (barrier sync handled by caller).N)r9rrr6)r:rrs rbenchmark_collective)SubgraphChoiceCaller.benchmark_collectivesD  ($($B$B$DD ! ""#BT%:%:#BT#BCr]c SRURRSS5S/URVs/sHn[ UR 55PM snQURVs/sHn[ UR 55PM snQ[ URR5P5$s snfs snf)N-rfr@r) joinrrsplitr"rJr/r0rr)r:r<s rhash_keySubgraphChoiceCaller.hash_keysxx   a(+ 151A1AB1A##clln%1AB 483C3CD3CC#cnn&'3CD DGGMM"    CDs #B7 -#B< c [RR[R"URUR UR URURS95$)N)r#r"rr+ subgraph_name) r TensorBoxcreateSubgraphBufferr#r"rr+rras r output_node SubgraphChoiceCaller.output_nodesN||""   {{ ,,77#22"ii    r]c SURS.$)zRInformation returned here is logged to the autotune log file when that is enabled.subgraph)backend kernel_namer`ras r info_dictSubgraphChoiceCaller.info_dicts"99  r]c SUR3$)N subgraph_r`ras rautoheuristic_id%SubgraphChoiceCaller.autoheuristic_ids499+&&r])r9r7r8r+rr6r4r)__name__ __module__ __qualname____firstlineno____doc__rJlistrr rrdictrIrDrEr*r5r[rbrfloatrrrrrrrr__static_attributes__ __classcell__r=s@rr r 4sjJN%*%*&\%* %*  %*  S) %*C3%*=!>>?$F%* %*%*N49<+%c3h/+9=c3h+ +..=3=@  tCy  u||    D$s)D%,,D4D #   R\\   4S> '#''r]r c\^\rSrSrSr\R "5rS\4U4Sjjr SS\S\ \ S\ S\ S \4S \S \\\ \/\R$44S-S \S \4SjjrSS\S\ \ S \4S\ \ S\ \\\4S\ S \4S-S \\\ \/\R$44S-S \ \4SjjrS\ S \4S \\\4S \4SjrS \\\4S S4SjrS\S\ \ S \4S\ \ S S4SjrSS\ \ S\ S \4S \\\4S\ S \4S-S \\\ \/\R$44S-S \ 4 SjjrSrU=r$)SubgraphTemplatez A template for subgraph evaluation to be used in autotuning. This class allows creating customized subgraphs that can be appended as choices during the autotuning process, enabling the selection of optimal implementations for complex operations. rc >[TU]US9 g)zd Initialize a subgraph template. Args: name: The name of this template graph: The FX graph r`N)r)r*)r:rr=s rr*SubgraphTemplate.__init__s d#r]Nr"r#r%.r$r&rYrc V[US[[R53UUUUUS9$)a! Generate a SubgraphChoiceCaller instance for autotuning. Args: name: The name for this subgraph choice input_nodes: List of input nodes to the subgraph layout: Memory layout information for the output make_fx_graph: Callable that creates the FX graph for this subgraph description: Optional description of this choice input_gen_fns: Optional dict mapping input indices to tensor generators **kwargs: Additional keyword arguments Returns: SubgraphChoiceCaller: A callable object that can be used for autotuning rf)rr"r#r$r%r&)r nextr index_counter)r:rr"r#r%r$r&rYs rgenerateSubgraphTemplate.generates;4$64 0 > >?@A##''   r]decompositionsnon_tensor_args default_implc ^U(d/$[U5[U5:Xd S[U5S[U5S35e[X$5VVs/sHupxURX7XU5PM n nnURXU 5 U Sn /n [X$5Hup|SSKmUU S.S[ S[ S [ 4S [[[ 4S [ 4U4S jjjn URX|5nURUS U3UU U SUR3US9nURX|5 U RU5 M U $s snnf)a Generate multiple SubgraphChoiceCaller instances for custom op autotuning. This method extends SubgraphTemplate to support custom op decompositions, allowing multiple implementations to compete in autotuning. Args: name: Base name for the choices decompositions: List of decomposition functions to compete in autotuning input_nodes: List of tensor inputs. All tensor arguments must be passed here. non_tensor_args: List of non-tensor kwargs only, one dict per corresponding decomposition. default_impl: Default implementation for layout inference input_gen_fns: Optional dict mapping input indices to tensor generators Returns: List of SubgraphChoiceCaller instances for autotuning z>decompositions and non_tensor_args must have same length, got z decompositions and z kwargsrN)decomp decomp_kwargsrr.rrc`>SSKJn SSKJn U"5nU"TR"U40UD6US9"U6$)Nr)make_fx)select_decomp_table)decomposition_table)"torch.fx.experimental.proxy_tensorrr7rpartial)rrrrrr functoolss rr%BSubgraphTemplate.generate_custom_op_choices..make_fx_graphMsB G?&9&;#%%f> >(;r]rfz CustomOp )rr"r#r%r$r&)r.rB_infer_custom_op_layout_validate_layout_equivalencerrrrrJ_generate_variant_namerrr[r3)r:rrr"rrr&rrYlayoutsr#choicesrr% variant_namechoicers @rgenerate_custom_op_choices+SubgraphTemplate.generate_custom_op_choicess4I>"c/&:: ~&'';C:SubgraphTemplate._generate_variant_name..vsN7Mtq1#Qqc 7Ms)rrsorteditems)r:rrY base_name param_suffixs rr'SubgraphTemplate._generate_variant_nameosFOO  xxNvflln7MNN Al^,,r]cUR5HAup#[U[R[45(dM,SUS[ U5S35e g)z8Validate that kwargs contains only non-tensor arguments.zkwargs['z'] contains tensor zo. Tensor arguments should be in input_nodes, not kwargs. Only scalar/non-tensor parameters should be in kwargs.N)rrCrDrErtype)r:rYkeyvalues r_validate_non_tensor_kwargs,SubgraphTemplate._validate_non_tensor_kwargsysV ,,.JC!%%,,)?@@ 3%24;-@IJ @)r]op_namerc2U(dgUSn[USSSS9HupVURURURUR4URURURUR4:wdMe[ SUSX%R SURSURSURSURS USR SURSURSURSURS 35e g) zXEnsure all layouts have consistent stride, device, dtype, and sizes for fair autotuning.Nrr@)startzLayout mismatch in custom op 'z': decomposition 'z ' produces (z, z) but 'r_)r-rdtypesizestrideAssertionErrorr)r:rrr referencer;r#s rr-SubgraphTemplate._validate_layout_equivalences AJ "712;a8IA v||V[[&--H    M %4WI>&&4&7&@&@%AB bb R W*1-6678!(()IOO+NbQZQaQaPbbc e9r]function_decompositionc vSSKnSSKJn URU5 UR /n[ U5HupU(aX;a XY"U 5n OU R 5n URRRU [RS9n U R5nURRRU[RS9n[R"U UU R5U R!5S9n UR#U 5 M UR$"U40UD6nU"U6n['U[R(5(dS[+U5S35e[-UR.UR0UR2UR55S9sSSS5 $!,(df  g=f) zInfer output layout for custom ops using the default implementation when available. Note that the Subgraph assumes custom ops return exactly one tensor output. TODO: Add support for multiple output custom ops. rNr)fallback)rrz#Expected single tensor output, got z:. Multi-output custom ops not yet supported in autotuning.)rrrr)rtorch._inductor.virtualizedrrr,r-r/rrK size_hintsrxunbacked_symint_fallbackr0rD empty_strided get_dtype get_devicer3rrCrErrrrrFr)r:r"rrYrr&rrr+r;r< fake_tensor raw_shapeconcrete_shape raw_strideconcrete_stridefnoutputs rr(SubgraphTemplate._infer_custom_op_layouts 1 ((0 [[N#K0 Q%7"/"23"7K # I%&WW%5%5%@%@!F,K,K&A&N"%!1J&'gg&6&6&A&A"V-L-L'B'O#("5"5&'!mmo"~~/ #K %%k2%1(""#9DVDB(Ffell33 5d6l^DKL 3 }}ll\\}} ?[[s E8F** F8r)N)NN)rrrrr itertoolscountrrJr*rrr rrrrIrDrEr rrrrrrrrrs@rrrskOO%M $ $&IM! ! &\!  !  S) !  ! C3%*=!>>?$F! !  ! R37IMSSXc3h/0S&\ S d38n- S sCx(4/ SC3%*=!>>?$FS " #Sj-sCx(-26sCx.- -$sCx.TXc3h/0f   >37IM 7&\7!)c 27S#X 7 sCx(4/ 7 C3%*=!>>?$F 7 77r]r)*rloggingcollections.abcrtypingrrGrDtorch._inductor.config _inductorrxtorch._inductorrtorch._inductor.codegen.commonrtorch._inductor.irrrr r r r r $torch._inductor.runtime.benchmarkingrtorch._inductor.utilsrrrtorch.utils._ordered_setr getLoggerrlogfx GraphModulerrJr ChoiceCallerr rrr]rrs$ ''9=:)/!) )&*3i)7:)),h'2??h'Vt~tr]