ȅi J%SSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK r SSK r SSKJrJr SSKJrJr SSKJr SSKJrJrJrJrJrJrJrJr SSKJrJr SSK r SSK!r!SSK"r!SSK#J$r$ SS K%J&r' SS K(J)r) SS K*J+r+ SS K,J-r- SS K.J/r0 SSK1J2r2J3r3J4r4 SSK5J6r6J7r7 SSK8J9r9J:r: SSK;Jr>J?r? SSK@JArA SSKBJCrCJDrDJErEJFrFJGrGJHrHJIrIJJrJJKrKJLrLJMrMJNrN SSKOJPrPJQrQJRrRJSrSJTrTJUrUJVrV \(akSSKWJXrXJYrYJZrZJ[r[ SSK"J\r\ SSK]J^r^ SSK_J`r`JaraJbrbJcrc SSKdJere SSKfJgrgJhrhJiri SSK@Jjrj SSKkJlrl \"S 5rm\X\\h/\g4rn\o\lrp\\q\ R4rs\qrt\!RR\wS!5rx\R"\w5rzGS S"jr{\R"S#S$55r}"S%S&\R$5r~"S'S(\5r\H"S)S*9"S+S,\55r"S-S.5r\R"S/S055r\R"S1S255r\R"S3S455r\R"S5S655r\R"S7S855r\\\\\\4r0rS9\S:'"S;S<5r0rS=\S>'0rS?\S@'0rSA\SB'GS GS SCjjr"SDSE\5rGSSFjrGSSGjrGSSHjrGSGSSIjjrGSSJjrGSSKjr\GR,GSSLj5rGSSMjrGSSNjrGSSOjr\!GR6\!GR8\!GR:\!GR80\!GR<\!GR>\!GR@\!GRB\!GRD\!GRF\!GRH\!GRJ\!GRL\!GRN\!GRP4 Vs0sHnX_M snErSP\SQ'GSSRjrGSSTjrGSSUjrGSSVjr"SWSX5r"SYSZ\05r/"S[S\5r\ GR`"S]\ GRbS^9rGSS_jr"S`Sa\>\\R\5r\R"SbSc55r\"GS0Sd\"\$GRnSeSfSg9_Sh\"\$GRnSiSjSkSl9_Sm\"\$GRnSnSoSpSl9_Sq\"\$GRnSrSsStSl9_Su\"\$GRnSvSwSxSl9_Sy\"\$GRnSzS{SyS|9_S}\"\$GRnS~SSSl9_S\"\$GRnSSSSS9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSSS9_S\"\$GRnSSSS|9_S\"\$GRnSSSSl9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSSl9_S\"\$GRnSSSSl9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_6rS\S'GSSjr"SS\D5r"SS\G5r"SS\5r\R"SS55r"SS5r\"5r"SS5r"SS5r\"S\qS9r\"SS\\S9r\(a\\!GR\T\\\\S444r"SS\\\45r"GSGS5r"GSGS\\\5r\R"GSGS55r\GR,GS GSj5r"GSGS5r"GS GS \?5rgs snf(!) annotationsN)ABCabstractmethod)autoEnum)chain)AnycastClassVarGeneric NamedTupleOptional TYPE_CHECKINGUnion)SelfTypeVar)ELEMENTWISE_TYPE_PROMOTION_KIND)_pytree) ConfigModule) OrderedSet)int_oo) PythonPrinter)free_symbol_is_typesymbol_is_typeSymT) bound_sympy ValueRanges)configmetrics)DtypePropagationOpsHandler)BasicMathOpsMixinDefaultHandler)ShapePropagationOpsHandler) boolean_opsDeferredLineBasegenerate_assertget_current_backendIndentedBuffer ir_dataclass ScopedDict sympy_dotsympy_index_symbol sympy_subs triton_typeunique) NullHandlerops OpsHandlerOpsValue ReductionType StoreModeV)CallableIteratorMutableMappingSequence) GraphModule)CustomGraphModulePass)Buffer ChoiceCaller FixedLayoutIRNodeLoopBody)BaseScheduling Scheduler SchedulerNode)BlockShapeTypePythonWrapperCodegen_Tschedulec[R[R5(a[R SU5 gg)NzData type propagation: %s) schedule_log isEnabledForloggingDEBUGdebug)msgs X/home/james-whalen/.local/lib/python3.13/site-packages/torch/_inductor/codegen/common.pydata_type_loggerrUas-  //6<0cj\rSrSr%SrS\S'S\S'S Sjr\SSj5rSS jr \SS j5r S r g )FileBackedGraphModulefz Output of FX wrapper codegen. Exposes the same methods as ModuleType, but these map back to a GraphModule instead of Python source. r<gmzCallable[..., Any] compiled_fnc.[R"SSSS9Ul[R"[R URR 5 URnURUR5 SSS5 g!,(df  g=f)Nzw+z.pyF)modesuffixdelete) tempfileNamedTemporaryFileatexitregisterosremovenamewritevalue)selffs rT __post_init__#FileBackedGraphModule.__post_init__ps^!33eE    4==#5#56 ]]a GGDJJ ]]s !B Bc.URR$N)r`rfris rT__file__FileBackedGraphModule.__file__zs}}!!!rVc UR"U6$rn)r[riargss rTcallFileBackedGraphModule.call~s&&rVc.URR$rn)rZcoderos rTrhFileBackedGraphModule.valuesww||rV)r`NreturnNoner{str)rt list[Any]r{r ) __name__ __module__ __qualname____firstlineno____doc____annotations__rkpropertyrprurh__static_attributes__rVrTrXrXfsF O## ""'rVrXcH\rSrSrSrSrSr\S Sj5r\S Sj5r Sr g) WorkspaceZeroModerrHrcX:XdU[R:XaU$U[R:XaU$[SU<SU<S35e)NzWorkspaceZeroMode.combine(, ))r UNINITIALIZEDNotImplementedErrorabs rTcombineWorkspaceZeroMode.combinesI 6Q+999H !// /H!$>qe2aU!"LMMrVcPU(a[R$[R$rn)r ZERO_ON_CALLr) zero_fills rT from_boolWorkspaceZeroMode.from_bools $11 1 ...rVrN)rrrrr{r)rboolr{r) rrrrrrZERO_PER_GRAPH staticmethodrrrrrVrTrrs9MLNNN//rVrc@\rSrSrSr\SSj5r\SSj5rSrg) CodegenSymbolzH An IR object possibly corresponding to a variable in the wrapper code. cgrnrros rTget_nameCodegenSymbol.get_name rVcgrnrros rT get_exampleCodegenSymbol.get_examplerrVrNr}r{z!Union[torch.Tensor, sympy.Symbol]) rrrrrrrrrrrVrTrrs/    rVrT)frozencN\rSrSr%SrS\S'S\S'S\S'S \S 'S rS \S '\Rr S \S'\ S S!Sjj5r \ S"Sj5r \ S#Sj5r \ S#Sj5rS$Sjr\rS%SjrS&SjrS'Sjr\S'Sj5r\r\r\rS(SjrS)SjrS)SjrS*SjrS+SjrS,SjrSrg)- WorkspaceArgaA temporary buffer used for a single kernel, then discarded. Not registered as a traditional buffer since there are no users, so it would be dead code eliminated. Args: nbytes: The size of the buffer in bytes. zero_fill: Whether the buffer should be initialized to zero. sympy.Exprcountr zero_mode torch.devicedevicer~ outer_namews_ptr inner_name torch.dtypedtypecPU[[RR53$rn)nextr7graph workspace_id)prefixs rT unique_nameWorkspaceArg.unique_names!$qww334566rVcURUR:H=(a9 URUR:H=(a URUR:H$rn)rrrrs rTcan_joinWorkspaceArg.can_joins@ LLALL ( XQWW-? XAHHPQPXPXDX rVc [URUR-[RURUR5UR UR URURS9$N)rrrrrr) rrrrrrrrrrs rTjoinWorkspaceArg.joinsS''AGG#'// Q[[I''88||||   rVc URUR:Xa4URUR:XaURUR:Xde[[R "UR UR 5[RURUR5URURURURS9$r) rrrrsympyMaxrrrrrrs rTmaximumWorkspaceArg.maximums GGqww 188qxx#7ALLALLUR5R5$rn) get_layoutrros rTrWorkspaceArg.get_examples ,,..rVc`SSKJn U"URURUR/S/S9$)Nr)r@rH)rrsizestride)irr@rrr)rir@s rTrWorkspaceArg.get_layouts.$;;****3   rVc"UR5$rn)rros rTlayoutWorkspaceArg.layouts  rVc6[RR$rn)rSZeroros rT get_offsetWorkspaceArg.get_offsetsww||rVcUR/$rn)rros rTget_sizeWorkspaceArg.get_sizes |rVc8[RR/$rn)rrOneros rT get_strideWorkspaceArg.get_strides }rVcUR$rn)rros rTrWorkspaceArg.get_names rVcg)NFrros rT get_is_pinnedWorkspaceArg.get_is_pinned srVc/$rnrros rTget_inputs_that_alias_output)WorkspaceArg.get_inputs_that_alias_output s rVrN) workspace_)rr~r{r~)rrrrr{r)rrrrr{r)r{r)r{rr)r{r@)r{r)r{list[sympy.Expr]r})r{r)r{ list[str]) rrrrrrrtorchuint8rrrrrrrget_device_or_errorrrrrrget_output_specmaybe_get_output_specmaybe_get_layoutrrrrrrrrrVrTrrs    OJE;$77        %/ !!!O&!rVrc(\rSrSrSSjrSSjrSrg)TritonScratchWorkspaceicXlX lgrn)r_generate_dtype_str)rirgenerate_dtype_strs rT__init__TritonScratchWorkspace.__init__s  #5 rVc"UR5$rn)rros rTr)TritonScratchWorkspace.generate_dtype_strs''))rV)rrN)rintrCallable[..., str]r})rrrrrrrrrVrTrrs 6*rVrcx\rSrSr%S\S'S\S'S\S'\R RrS\S'S r S \S 'S r g ) TensorArgir~rfbufferrrroffsetN Optional[str]alias_ofr) rrrrrrrrrr rrrVrTrrs. I K FJ%"Hm"rVrc>\rSrSr%S\S'S\S'\S Sj5rSrg) SizeArgi"r~rfrexprcgrnrros rTr SizeArg.alias_of'srVrNr{r )rrrrrrr rrrVrTr r "s I  rVr c \rSrSr%S\S'Srg) ConstexprArgi,r~rfrNrrrrrrrrVrTrr,s IrVrc>\rSrSr%S\S'S\S'S\S'S\S'S rg ) TMADescriptorArgi1r~rfapi_typezOptional[list[sympy.Expr]] block_shapeOptional[torch.dtype]rrNrrrVrTrr1s IM++  rVrcF\rSrSr%S\S'S\S'SrS\S'SrS\S 'S rg) DeviceCodegeni9SchedulingConstructor schedulingWrapperConstructorwrapper_codegenNOptional[WrapperConstructor]cpp_wrapper_codegenfx_wrapper_codegenr)rrrrrr r!rrrVrTrr9s&%%''8<5<7;4;rVrzdict[str, DeviceCodegen]device_codegensc\rSrSrSSjrSSjrSSjrSSjrSSjrSSjr SSjr SS jr SS jr SS jr SS jrSS jrSSjrSSjrSSjrSSjrSSSjjrSrg)DeviceOpOverridesiFc[ernrrirfs rTimport_get_raw_stream_as*DeviceOpOverrides.import_get_raw_stream_asG!!rVc[ernr&ri device_idxs rT set_deviceDeviceOpOverrides.set_deviceJr*rVc[ernr&ros rT synchronizeDeviceOpOverrides.synchronizeMr*rVc[ernr&r,s rT device_guardDeviceOpOverrides.device_guardPr*rVc[ernr&ros rTcpp_device_guard"DeviceOpOverrides.cpp_device_guardSr*rVc[ernr&ros rTcpp_aoti_device_guard'DeviceOpOverrides.cpp_aoti_device_guardVr*rVc[ernr&ros rTcpp_stream_guard"DeviceOpOverrides.cpp_stream_guardYr*rVc[ernr&ros rTcpp_aoti_stream_guard'DeviceOpOverrides.cpp_aoti_stream_guard\r*rVc[ernr&ros rTcpp_getStreamFromExternal+DeviceOpOverrides.cpp_getStreamFromExternal_r*rVc[ernr&ros rT kernel_headerDeviceOpOverrides.kernel_headerbr*rVc[ernr&ros rT kernel_driverDeviceOpOverrides.kernel_driverer*rVc[ernr&ros rTcpp_stream_type!DeviceOpOverrides.cpp_stream_typehr*rVc[ernr&ros rTaoti_get_stream!DeviceOpOverrides.aoti_get_streamkr*rVc[ernr&ros rTcpp_kernel_type!DeviceOpOverrides.cpp_kernel_typenr*rVc[ernr&ros rTcpp_device_ptr DeviceOpOverrides.cpp_device_ptrqr*rVc[ernr&ros rTtma_descriptor_helpers(DeviceOpOverrides.tma_descriptor_helperstr*rVNc[ernr&)riidx workspacers rT cpp_scratchDeviceOpOverrides.cpp_scratchws "!rVrrfr~r{r~)r-rr{r~r}rn)r[rr\rrr r{zOptional[tuple[list[str], str]])rrrrr(r.r1r4r7r:r=r@rCrFrIrLrOrRrUrXr]rrrVrTr$r$Fs""""""""""""""""TX""#9"CP" (""rVr$zdict[str, DeviceOpOverrides]device_op_overrides_dictz*dict[str, Optional[CustomGraphModulePass]]custom_backend_passesz!dict[str, Optional[ConfigModule]]custom_backend_codegen_configsc[UUUU5[U'U[U'U(a1[U[5(a U[ LdSU<S[ <35eU[ U'g)Nzdevice_custom_config=z: cannot be the same as the default inductor config config=)rr"ra isinstancerrrb)rdevice_schedulingdevice_wrapper_codegendevice_cpp_wrapper_codegendevice_fx_wrapper_codegendevice_custom_passdevice_custom_configs rTregister_backend_for_devicerksy,"! OF %7&! +\ : :$F2 %#%%`Y_Xa b   3 .B"6*rVc\rSrSr\"5r\"5r\"5r\"5r\"5r \"5r \"5r \"5r \"5r \"5rSrg)BackendFeatureirN)rrrrrFOREACH BUCKETIZEINPLACE_BUFFERSMASKED_SCATTER_WITH_INDEXSCANSORTTUPLE_REDUCTIONPREFER_STORE_LOOP_ORDERTRITON_TEMPLATESREDUCE_TO_SINGLE_ELEMENTrrrVrTrmrmsLfGIfO $ 6D 6DfO"fv#vrVrmc\Uc [5$[5 [U[R5(a UR nO=[U[ 5(d[ U55eUn[R"U5n[U5nU(deU"S5nURU5$rn) rinit_backend_registrationrdrrtyper~get_scheduling_for_deviceget_backend_features)r device_typescheduling_ctorrs rTr|r|s~|&%,,''kk &#&&4V 4& k*/ ^^SSKJn SSKJn SSKJn SSKJn SSKJ n SSK J n SSK J n SS KJn SS KJn SS KJn SS KJn SS KJn SSKJn [5S5c;UUU US.m[7SU4SjU [8R:R<(aUOUU 5 [5S5cUUUS.m[7SU4SjU UU 5 [5S5c[7SU U UU 5 [5S5c[7SUU UU 5 [5S5c[7SU U UU 5 [>R@RC5n U S:wa\[5U 5cOSSK"J#n U"S5nU"S5nU"S5nU"S5nU(a U(aU(a[7U UUUU5 gggggg![Ha gf=f) z Register the backend for different devices, including the scheduling for kernel code generation and the host side wrapper code generation. rH) CppScheduling) CppWrapperCpu)CppWrapperCpuArrayRef) CppWrapperGpu) CppWrapperMps)CUDACombinedScheduling)HalideScheduling)MetalScheduling)PallasScheduling)PythonWrapperMtia)TritonSchedulingrI)WrapperFxCodegencpuN)cpphalidetritonpallasc6>T[R"U5$rn)r cpu_backend)r cpu_backendss rT+init_backend_registration..s|F,>,>? KrVcuda)rrrc6>T[R"U5$rn)r cuda_backend)r cuda_backendss rTrr!s}V-@-@A*MrVxpumpsmtia privateuseoner)_get_custom_mod_func SchedulingrJCppWrapperCodegenr)%rrcpp_wrapper_cpurcpp_wrapper_cpu_array_refrcpp_wrapper_gpurcpp_wrapper_mpsrcuda_combined_schedulingrrrrrrrpython_wrapper_mtiarrrwrapperrJ wrapper_fxirrr{rkr aot_inductorallow_stack_allocationr_C_get_privateuse1_backend_name torch.utils.backend_registrationr RuntimeError)rrrrrrrrrrrrJrprivate_backendrrerr r!rrs @@rTryrys #.@..@($(6(-. '/ &&&  $  K ""99 "  !(0-&& $  M    !'/#      !'/#      !(0#       hh<<>O?* %o 6 >I  4\ B 23IJO"67J"K !56H!I  _9L+#%#'& :M_  ? +$   sAF FFcJSSKJn /UQ[XRU55P$)Nr)FlexibleLayout)rrr,contiguous_strides)index index_varssizesrs rTindex_prevent_reorderingrZs* $ UU TIj*K*KE*RS TTrVcU[U'grn)r`)rdevice_op_overridess rTregister_device_op_overridesres(;V$rVc[U[5(d[U55e[(dSSKJnJn SSKJn SSK Jn SSK Jn [U$)NrH)cpu_device_op_overridesmps_device_op_overrides)r) rdr~rzr`rrrrrr)rrrrmtia_op_overridesxpu_op_overridess rTget_device_op_overridesrks; fc " "0DL0 " # #F-B@ #F ++rVzdict[torch.dtype, torch.dtype]DTYPE_TO_COMPUTATION_DTYPEcU[5;a[R$US;aSU;aUS$US$US;a[R$US;a[R$US:XaSU;aUS$US$US:XaSU;aUS$US$US ;a$USn[ R RU5$US :XaSU;aUS$US $g ) zC Given op name and a list of input dtypes, deduce the output dtype )to_dtype index_exprr)randrandn) get_index randint64 load_seed reductionrHconstant)loadstorestore_reductionto_dtype_bitcastN)r%rrfloatint64r7rr)op_namertkwargsbuf_names rTdeduce_output_dtype_by_namers +-zz  #*V"3vgAbA  {{   {{ K ")V"3vg@a@ J ")V"3vgAbA   7ww  ** & &")V"3vgAbA rVCSEVariableTypec*[5n[RR(a(US:Xa"UR SUS[ U5S35 g[RR (aUS:XaSSKJnJ n [X5(d[U55eU[R:Xa"UR(aSUS 3nO2S US US 3nO(S US3nUR(aSUS3nSUSXRS3nUR SUS35 ggg)Nrtl.static_assert(z .dtype == rrrH)CppCSEVariable DTYPE_TO_CPPzIsVecMaskType::valuezstd::is_same_v || std::is_same_vz decltype(z typename z ::value_typezstd::is_same_vzstatic_assert(z);)r(r test_configsruntime_triton_dtype_assert writeliner/static_cpp_dtype_assert cpp_utilsrrrdrzrris_vec)rvarrbackendrr is_same_dt c_var_types rT check_dtypers"#G 667h;N,SEK*R89!:J 4rVc[5nUce[RR(aOUS:XaH[ U5S:waSR SU55OUSS3nUR SUSUS 35 ggg) NrrHrc38# UHn[U5v M g7frnr~).0ds rT check_shape..s,ec!ffesr,rz .shape == ()))r(rrruntime_triton_shape_assertlenrr)rrshaper shape_strs rT check_shapers"#G    667h;N03E aDII,e, ,azQR^  ,SEYKrJK .sR) ! # #qvv - B*../55TA B sAA)all_input_nodesrdrfxNodeoprall functoolsreduce promote_typesrrrr)rinodeinputsr input_nodesall_input_nodes_propagateds rTdeduce_node_dtype_by_inputs/DataTypePropagation.deduce_node_dtype_by_inputss%% !Auxx}}!=A!$$-BWAv   { q %() !) & " *   T$++->>+ ;; ! !"3 4 455d; ;8 ++ L    //55rVcUR(deSnURHn[RUR;aUR[RnO [5nUR U5UlXCR[R'UR S:XdMUR nM U$)Nr2)nodesrrrr6rr,)rir graph_dtyper%opt_ctxs rTr-#DataTypePropagation.propagate_graph&s{{{-1 KKD"&&$))3))$7$;$;<-/ 2248GM18II)-- .{{h&%mm  rVc>URURS5$r )r-rros rT propagateDataTypePropagation.propagate8s##DKK$788rVc.U"U5R5$rn)rA)clsr s rTpropagate_loopbody&DataTypePropagation.propagate_loopbody;s4y""$$rVcSSKJn SSKJn [ X5(d[ U55e[ UR U5(d[ UR 55e[RUR 5$)NrrB)rF) loop_bodyrC schedulerrFrdrz_bodyr rE)rDr%rCrFs rTpropagate_scheduler_node,DataTypePropagation.propagate_scheduler_node?s\(-$..:T :.$**h//Adjj1AA/"55djjAArV)r rN)r rCr{r|)r% torch.fx.Noder{r)r%rMr{r)rztorch.fx.Graphr{r)r{r)r rCr{r)r%rFr{r)rrrrrr)r/r6r-rA classmethodrErKrrrVrTr r sJ% * 6:$9%%BBrVr cX^\rSrSrSSS.SU4SjjjrSS U4SjjjrSrU=r$) riIT)simplifypc>U(ag[U[R5(aH[[R S5(a)[R R RU5n[TU]%U5$)Nsizevars) rdrExprhasattrr7rrSrPsuperdoprint)rir rPrQ __class__s rTrWPythonPrinter.doprintJsQ  444*9U9U77##,,T2Dwt$$rVc>[U[R5(aSURU5S3$[TU]XU5$N(r)rdrMod_printrV parenthesize)riitemlevelstrictrXs rTr_PythonPrinter.parenthesizeRsA dEII & &t{{4()+ +7'V< ?@@rVc [R"S[R5n[R"U[R "U[R "[R"U5555$rl)r2rrrnrmrr~neg)rpones rTsigmoidOpDecompositions.sigmoidsCll1ekk*{{3SWWSWWQZ-@ ABBrVcv[R"U[R"S[R55$Nr)r2rrrrnros rTreluOpDecompositions.relus"{{1cll1ekk:;;rVcX[R"[R"X5U5$rn)r2rrtrpyzs rTfmaOpDecompositions.fmaswwswwq}a((rVcX[R"[R"U5U5$rn)r2rfloorrrs rT floor_to_intOpDecompositions.floor_to_int||CIIaL%00rVcX[R"[R"U5U5$rn)r2rceilrs rT ceil_to_intOpDecompositions.ceil_to_ints||CHHQK//rVcX[R"[R"U5U5$rn)r2rtruncrs rT trunc_to_intOpDecompositions.trunc_to_intrrVc [R"X5n[R"[R"U[R"S[ R 55[R"[R"U5[R"U555n[R"U[R"X!5U5$r) r2modand_nerrrnsignbitwherer)rrrconds rT remainderOpDecompositions.remaindersu GGAMxx FF1cll1ekk2 3 FF3;;q>3;;q> 2 yyswwq}a00rVcX[R"[R"U5U5$rn)r2rroundrs rT round_to_intOpDecompositions.round_to_intrrVrN)rhOpVarTr{rrprr{r)rprrrrrr{r)rrrrr{rrrrrr{r)rrrrrrrirqrur{rrrrrrrrrrrrrrrrrVrTrgrg[s}<<CC<<CCRRQQMMAACC<<))1100111111rVrgz[a-z0-9_.]+|\([^)]*\)|)flagscUSS:wd[U5S:agSn[USS5H8up#US:XaUS- nO US:XaUS-nUS:XdM$U[U5S- :wdM8 g US:Xdeg)Nrr\rFrHrT)r enumerate)stringrichars rT_all_in_parensrs ayC3v;? EVABZ( 3; QJE S[ QJE A:!s6{Q. ) A:: rVc\rSrSr\S$Sj5r\S%Sj5r\S&Sj5r\S'Sj5r\S(Sj5r \S(Sj5r \S(Sj5r \S(S j5r \S(S j5r \S)S j5r\S*S j5rS+S,SjjrS-SjrS.SjrS/S0SjjrS1SjrS2SjrS3SjrS4SjrS5SjrS6S7SjjrS8SjrS(SjrS\R:S SS.S9SjjrS:SjrS;Sjr \SS"j5r$S#r%g)? OpOverridesic[U[5(d*[RU5(d[ U5(aU$SUS3$r[)rd CSEVariable_RE_PAREN_NOT_NEEDED fullmatchr)rs rTparenOpOverrides.parensB v{ + +#--f55f%%M6(!}rVc[U5$rn)repr)rhrs rTrOpOverrides.constants E{rVc2S[RU53$)N~rrros rT bitwise_notOpOverrides.bitwise_nots;$$Q'())rVc2[RU5S3$)Nz == 0r)rs rT logical_notOpOverrides.logical_nots##A&'u--rVc\[RU5S[RU53$)Nz & rrprs rT bitwise_andOpOverrides.bitwise_and+##A&'s;+<+> rrs rTbitwise_right_shiftOpOverrides.bitwise_right_shiftrrVc.[R"X5$rn)r2rmrs rT int_truedivOpOverrides.int_truedivs {{1  rVcX[R"U[R"U55$rn)r2rrInteger)rfrs rTrOpOverrides.load_seedsxxemmF344rVTc*[[U55$rn)r-r~)rirrcheckwrap_negs rTindirect_indexingOpOverrides.indirect_indexings"#c(++rVcD[[U5RS35e)Nz,: check_bounds should be handled by CSEProxyrrzrrir rloweruppers rT check_boundsOpOverrides.check_boundss'"Dz""##O P  rVcD[[U5RS35e)Nz$: load should be handled by CSEProxyrrirfrs rTrOpOverrides.load s%!Dz""##G H  rVNcD[[U5RS35e)Nz%: store should be handled by CSEProxyrrirfrrhr]s rTrOpOverrides.stores'"Dz""##H I  rVcD[[U5RS35eNz3: device_assert_async should be handled by CSEProxyrrirrSs rTdevice_assert_asyncOpOverrides.device_assert_async%!Dz""##V W  rVcD[[U5RS35e)Nz/: store_reduction should be handled by CSEProxyrrirfrrhs rTrOpOverrides.store_reductions%!Dz""##R S  rVcD[[U5RS35e)Nz): reduction should be handled by CSEProxyrrir src_dtypereduction_typerhs rTrOpOverrides.reduction"s'"Dz""##L M  rVcD[[U5RS35e)Nz$: scan should be handled by CSEProxyrridtypes combine_fnvaluess rTscanOpOverrides.scan-s'"Dz""##G H  rVcD[[U5RS35e)Nz$: sort should be handled by CSEProxyrrirrstable descendings rTsortOpOverrides.sort:s'"Dz""##G H  rVcD[[U5RS35e)Nz): bucketize should be handled by CSEProxyrrir boundariesboundary_indicesindexing_dtyperightsortersorter_indicess rT bucketizeOpOverrides.bucketizeEs'"Dz""##L M  rVcD[[U5RS35e)Nz2: halide_clamp only implemented for Halide backendr)rirhrrs rT halide_clampOpOverrides.halide_clampSs%!Dz""##U V  rVcD[[U5RS35e)Nz): dot only implemented for Triton backendr)rirprs rTdotOpOverrides.dotXs%!Dz""##L M  rVrH) constraintsris_purepackcD[[U5RS35e)Nz<: inline_asm_elementwise only implemented for Triton backendr)riasmr2rr3r4r&s rTinline_asm_elementwise"OpOverrides.inline_asm_elementwise]s'"Dz""##_ `  rVcD[[U5RS35e)Nz.: ops.output should not appear at codegen timeAssertionErrorrzrrss rTr2OpOverrides.outputjs%Dz""##Q R  rVcD[[U5RS35e)Nz3: ops.placeholder should not appear at codegen timer:rirs rTrOpOverrides.placeholderos%Dz""##V W  rVc4^SU4SjjnTUlSUlU$)NcJ>[[U5RST35e)Nz does not implement ops.r)rirtrrfs rT unimplemented1OpOverrides._unimplemented..unimplementedvs*%:&&''?vF rVT)rirrtr rr r{r)ris_unimplemented)rfrBs` rT_unimplementedOpOverrides._unimplementedts   "& )- &rVc[XS5n[[US5nU(+=(d X#:H=(d [USS5$)NrDF)getattrr3)rDrffn default_fns rT_is_unimplementedOpOverrides._is_unimplementeds; S %Zt4 vS)SWR9KU-SSrVc`US;dU5e[R5Hup#[X15nUc5URU5(a[ XUR U55 MFMHX R ;dSUSUR35eX$l[ X[U55 M g)N)rrcppvecrrzmultiple definitions of z on ) pointwise_overrides_datarrHrKsetattrrE__dict__rr)rDr,funcnamedataimpls rT_initialize_pointwise_overrides+OpOverrides._initialize_pointwise_overridessEEMvME6<<>NH4(D|((22C3+=+=h+GH3 ||3.xjS\\NK3!) |D'9:?rVr)rrr{r)rhzUnion[bool, float, int]rrr{rr)rrr{r)rprrrr{rr)rfr~rrr{rTT) rrrUnion[sympy.Expr, int]rrrrr{ sympy.Symbol r rrrrrrrr{r|)rfr~rrr{rrn) rfr~rrrhrr]r6r{r|rrrSr~r{r|)rfr~rrrhrr{r|) rrrrrr5rh!Union[OpVarT, tuple[OpVarT, ...]]r{r\)rtuple[torch.dtype, ...]rzFCallable[[tuple[OpVarT, ...], tuple[OpVarT, ...]], tuple[OpVarT, ...]]rtuple[OpVarT, ...]r{r^) rr]rr^rrrrr{r^NN)rrr$.tuple[str, sympy.Expr, sympy.Expr, sympy.Expr]r%rr&rr'rr( Optional[tuple[str, sympy.Expr]]r)zOptional[OpVarT]r{r)rhrrrrrr{r)r&rr6r~r2r rrr3rr4rr{r)rtrr{r|)rrr{r)rfr~r{zCallable[..., OpVarT]rfr~r{r)r,r~r{r|)&rrrrrrrrrrrrrrrrrrrrr rrrr r*r-r0rryr7r2rrErNrKrUrrrVrTrrs1  **..BBBBBBCCCC!! 55 , ,%, ,  ,  ,  &0 9= FJ   NR  * 39 AJ         &  1  +    '     #       '  #        $48+/    C  !  $    1  )       &*"]]      #             TT  ; ;rVrc\rSrSr%S\S'S\S'SrS\S'SrS\S '\Rr S \S 'Sr S\S 'Sr S\S 'Sr g) OverridesDatair~rfrrNzOptional[Callable[..., str]]rrNrtype_promotion_kindrrr) rrrrrrrNrDEFAULTrerrrrrVrTrdrdsQ I +/F (/+/F (/'//8,0F (/(,C %,rVrdairy_aicSUS3$)Nzairy_ai_forward(rrros rTrrs(1-rVspecial_airy_ai)rerrf bessel_j0cSUS3$)Nzbessel_j0_forward(rrros rTrr*1#Q/rVcSUS3$)Nz libdevice.j0(rrros rTrr =1-rVspecial_bessel_j0)rerrrf bessel_j1cSUS3$)Nzbessel_j1_forward(rrros rTrrrlrVcSUS3$)Nz libdevice.j1(rrros rTrrrnrVspecial_bessel_j1 bessel_y0cSUS3$)Nzbessel_y0_forward(rrros rTrrrlrVcSUS3$)Nz libdevice.y0(rrros rTrrrnrVspecial_bessel_y0 bessel_y1cSUS3$)Nzbessel_y1_forward(rrros rTrrrlrVcSUS3$)Nz libdevice.y1(rrros rTrrrnrVspecial_bessel_y1digammacSUS3$)Nz calc_digamma(rrros rTrrs aS*rVc US3$)Nz .digamma()rros rTrrs A3j)rV)rerrNrfrcSUS3$)Nz calc_erfcx(rrros rTrr A3a(rVcSUS3$)Nzlibdevice.erfcx(rrros rTrrs+A3a0rV special_erfcxrcSUSUSUS3$)Nz std::fma(rrrrs rTrrsis"QCr!A6rVcSUSUSUS3$)Nzfmadd(rrrrs rTrrss"QCr!A6rVcSUSUSUS3$)Nzlibdevice.fma(rrrrs rTrrss"QCr!A>rV)rerrNrrfigammacSUSUS3$Nz calc_igamma(rrrrs rTrr<s"QCq1rVigammaccSUSUS3$Nz calc_igammac(rrrrs rTrr=2aS2rVgammainccSUSUS3$rrrs rTrrrrVspecial_gammainc gammaincccSUSUS3$rrrs rTrrrrVspecial_gammaincci0cSUS3$)Nzcalc_i0(rrros rTrr 1orVcSUS3$Nzlibdevice.cyl_bessel_i0(rrros rTrr3A3a8rVc US3$)Nz.i0()rros rTrrs A3erV)rerrrNrfi0ecSUS3$)Nz calc_i0e(rrros rTrr !A&rVc US3$)Nz.i0e()rros rTrrs A3frV special_i0ei1cSUS3$)Nzcalc_i1(rrros rTrrrrVcSUS3$Nzlibdevice.cyl_bessel_i1(rrros rTrrrrV special_i1i1ecSUS3$)Nz calc_i1e(rrros rTrrrrV special_i1elog_ndtrcSUS3$)Nzcalc_log_ndtr(rrros rTrr s qc+rVspecial_log_ndtrmodified_bessel_i0cSUS3$)Nzmodified_bessel_i0_forward(rrros rTrr3A3a8rVcSUS3$rrros rTrrrrVspecial_modified_bessel_i0modified_bessel_i1cSUS3$)Nzmodified_bessel_i1_forward(rrros rTrrrrVcSUS3$rrros rTrrrrVspecial_modified_bessel_i1modified_bessel_k0cSUS3$)Nzmodified_bessel_k0_forward(rrros rTrrrrVspecial_modified_bessel_k0modified_bessel_k1cSUS3$)Nzmodified_bessel_k1_forward(rrros rTrr rrVspecial_modified_bessel_k1ndtrcSUS3$)Nz calc_ndtr(rrros rTrr&s 1#Q'rV special_ndtrndtricSUS3$)Nz calc_ndtri(rrros rTrr+rrV special_ndtri polygammac *USUSUSUSUSUS3 $)Nz == 0 ? calc_digamma(z) : (z == 1 ? trigamma(z) : calc_polygamma(rrrrs rTrr0s3S%aSaS0A!DWXYWZZ\]^\__a brVscaled_modified_bessel_k0cSUS3$)Nz"scaled_modified_bessel_k0_forward(rrros rTrr8:1#Q?rV!special_scaled_modified_bessel_k0scaled_modified_bessel_k1cSUS3$)Nz"scaled_modified_bessel_k1_forward(rrros rTrr=rrV!special_scaled_modified_bessel_k1spherical_bessel_j0cSUS3$)Nzspherical_bessel_j0_forward(rrros rTrrCs4QCq9rVspecial_spherical_bessel_j0zetacSUSUS3$)Nzzeta(rrrrs rTrrHs52aS*rV special_zetachebyshev_polynomial_tcSUSUS3$)Nzchebyshev_polynomial_t_forward(rrrrs rTrrM:1#Rs!DrVspecial_chebyshev_polynomial_tchebyshev_polynomial_ucSUSUS3$)Nzchebyshev_polynomial_u_forward(rrrrs rTrrRrrVspecial_chebyshev_polynomial_uchebyshev_polynomial_vcSUSUS3$)Nzchebyshev_polynomial_v_forward(rrrrs rTrrWrrVspecial_chebyshev_polynomial_vchebyshev_polynomial_wcSUSUS3$)Nzchebyshev_polynomial_w_forward(rrrrs rTrr\rrVspecial_chebyshev_polynomial_wlegendre_polynomial_pcSUSUS3$)Nzlegendre_polynomial_p_forward(rrrrs rTrra9!BqcCrVspecial_legendre_polynomial_pshifted_chebyshev_polynomial_tcSUSUS3$)Nz'shifted_chebyshev_polynomial_t_forward(rrrrs rTrrfB1#Rs!LrV&special_shifted_chebyshev_polynomial_tshifted_chebyshev_polynomial_ucSUSUS3$)Nz'shifted_chebyshev_polynomial_u_forward(rrrrs rTrrkrrV&special_shifted_chebyshev_polynomial_ushifted_chebyshev_polynomial_vcSUSUS3$)Nz'shifted_chebyshev_polynomial_v_forward(rrrrs rTrrprrV&special_shifted_chebyshev_polynomial_vshifted_chebyshev_polynomial_wcSUSUS3$)Nz'shifted_chebyshev_polynomial_w_forward(rrrrs rTrrurrV&special_shifted_chebyshev_polynomial_whermite_polynomial_hcSUSUS3$)Nzhermite_polynomial_h_forward(rrrrs rTrrzs82aSBrVspecial_hermite_polynomial_hhermite_polynomial_hecSUSUS3$)Nzhermite_polynomial_he_forward(rrrrs rTrrrrVspecial_hermite_polynomial_helaguerre_polynomial_lcSUSUS3$)Nzlaguerre_polynomial_l_forward(rrrrs rTrrrrVspecial_laguerre_polynomial_lzdict[str, OverridesData]rOc^[U4Sj[RR[RR[RR [RR 455$)Nc3.># UH nTU;v M g7frnr)rrprfs rTr$is_buffer_removed..s  A   s)anyr7rremoved_bufferskernelinplaced_to_removerfs`rTis_buffer_removedrsU  GG # # HH $ $ GG & & HH ' '   rVcD^\rSrSrSrSU4SjjrSSjrS SjrSrU=r $) DeferredLineizHA line that can be 'unwritten' by adding name to V.graph.removed_buffersc^>[TU]U5 Xl[U[5(aegrn)rVrrfrdr&)rirflinerXs rTrDeferredLine.__init__s,  d$455555rVcP[UR5(d UR$grn)rrfrros rT__call__DeferredLine.__call__s ++99 rVc.[URU5$rn)rrf)rirs rT _new_lineDeferredLine._new_linesDIIt,,rVr)rfr~rr~r)rr~r{r) rrrrrrrrrrdres@rTrrsR6  --rVrc"\rSrSrSSSjjrSrg) BracesBuffericL^^[RSUU4Sjj5nU"5$)Nc3># [T5H)nTRS5 T=RS- slM+ [T*5H)nT=RS-slTRS5 M+ Sv [T*5H)nTRS5 T=RS- slM+ [T5H)nT=RS-slTRS5 M+ g7f)N{rH})ranger_indent)_rris rTctx BracesBuffer.indent..ctxs6]s# ! #F7^ ! s#$ F7^s# ! $6] ! s##sC(C+)r{Iterator[None]) contextlibcontextmanager)rirrs`` rTindentBracesBuffer.indents$  " " $ # $u rVrN)rH)rrr{z'contextlib.AbstractContextManager[None])rrrrrrrrVrTr r s rVr c*\rSrSr%S\S'S\S'Srg)InplacedBufferir~rr other_namesrNrrrVrTrrs OrVrc8\rSrSr%S\S'SrS\S'S SjrSrg ) ArgNameir~rfFr is_constexprcLURUR(aS3$S3$)Nz : tl.constexprr)rfrros rT full_nameArgName.full_names*))$2C2C.LMMLMMrVrNr})rrrrrrr!rrrVrTrrs IL$NrVrc\rSrSrSSjrSrg) RemovedArgicg)NREMOVEDrros rT__str__RemovedArg.__str__srVrNr})rrrrr'rrrVrTr$r$srVr$c8\rSrSr\SSj5rSSjrSSjr\SSj5rSSjr SSjr SSjr \ R4SS jjrSS jrS S jrS!S jrS"S jrS#SjrS$SjrS%SjrS&S'SjjrS(SjrS)SjrS*SjrS+SjrSrg), KernelArgsicURU[5n[U[5(aU[ U53=X'nU$U$rn)rr&rdr$r)rodictrfresult new_results rT_lookupKernelArgs._lookupsE */4)A fj ) )*0#e*'> >EK*  rVcJ0Ul0Ul0Ul0Ul/Ulgrn) input_buffersoutput_buffersinplace_buffersrSworkspace_argsros rTrKernelArgs.__init__s)-/ACMO/1 24rVc SRSR[[URUR UR UR/555$)NzKernelArgs({})r)formatrmaprr2r3r4rSros rT__repr__KernelArgs.__repr__sS&& II**++,,    rVc"[U[5$rn)rdr$rs rT_buffer_is_marked_removed$KernelArgs._buffer_is_marked_removeds$ ++rVcT[RR(a3[RRRR X5nU[RR ;dU5eXR ;a[[UR U5$XR;a'[[URU5R$URS5(aURSURU5$URSURU5$)Nseedin_ptr)r7rrImutation_real_namerrr3r r~r4rrr7r/r2r's rTinputKernelArgs.inputs 77  77$$77;;DGD1772228D82 && &T0067 7 '' '(<(R!R  /01)C03  ,03  -! sF 6F F7Fc~[U[RU5[RR 5[R 5US9n[UR5HupV[RXd5(aIURn[RXd5URU'URURU4s $URUR:waURUR:waMU5e URRU5 URURS4$)a Allocate or extend a workspace buffer of nelem elements. This function manages the allocation of a workspace buffer. It either creates a new WorkspaceArg or extends an existing one. Note: - Calling this function will in-place mutate the args by adding or updating a WorkspaceArg. - The codegen for generating the Python argdefs and call_defs will check this field and allocate the buffer accordingly. - A new argument "ws_ptr" will be present in the generated code. Args: nelem (sympy.Expr): The number of elements to allocate. zero_fill (bool): Whether to initialize the buffer to zero. dtype (torch.dtype): the dtype of the workspace tensor Returns: Tuple[str, str, int]: A tuple containing: - "ws_ptr": A string identifier for the workspace pointer. - "workspace_{i}": agraph level unique identifier for the workspace tensor. - offset: An integer representing the item offset in the workspace. )rrrrrr)rrrr7rget_current_device_or_throwrrr5rrrrrrK)rinelemrrargr existing_argrs rTr\KernelArgs.workspace+s 8'11)<77668#//1   ))<)<=OA$$\77%++)5):):<)M##A&#.. 0G0GOO''3>>9 ++s~~=  > > ""3'~~s~~q00rVc [RR5n[U[R [ RSSURSUR3US9nURH,nURUR:XdMX4:XaM&X445e URRU5 UR$)a Lazily allocate a graph-wide semaphores buffer with at least min_size. This is a single buffer shared by all kernels and zero initialized once at graph start. Each kernel must leave the buffer zeroed on exit. Warning: multiple calls to this function will return the same buffer. Args: min_size: the number of int32 semaphores required Returns: name of the semaphores buffer sem_ptr semaphores_r)rrrrrr) r7rrUrrrruint32rzrr5rrK)rimin_sizecurrent_devicerWrXs rT semaphoresKernelArgs.semaphoresZs<<>'66,, $^%8%8$9>;O;O:PQ!  !//L&&#..8*?S,??*0 ""3'~~rVcz^[U[5(d[U5U45e[R"U5nX R ;aUR U$TUR R 5;a1T[U4SjUR R 5553mTUR U'T$)Nc3V># UHoRT5(dMSv M g7f)rHN)r7)rrrfs rTr)KernelArgs.seed_offset..~sU(>1,,tBTQQ(>s) ))rdrrzrrrSrsum)rirfrhs ` rT seed_offsetKernelArgs.seed_offsetvs%%%;U U';;% e$ MM !==' ' 4=='') )&U (<(<(>UUVW  $ e rVc[U[R5(d[U5U45eURS:XaSUR U'gUR SUR U5$)Nr@ks)rdrSymbolrzrfrSr/r's rTrKernelArgs.sizes[$ --AT D/AA- 99 "(DMM$ ||D$--66rVc[URR5URR5URR55$rn)rr2keysr3rSros rT call_namesKernelArgs.call_namessA    # # %t':':'?'?'A4==CUCUCW  rVc&URRUS5nUb![U[5(d UR$UR RUS5nUb[U[5(dU$UR RUS5$)z+ Returns inner name of a given outer name. N)r4rrdr$rr3r2)rirfinplacedrMs rTarg_nameKernelArgs.arg_names''++D$7   8Z(H(H&& &))--dD9  ":k:+N+N !!%%dD11rVcU$rnr)rirNrs rT wrap_ptr_argKernelArgs.wrap_ptr_args rVc[U5$rnr)rirs rT wrap_size_argKernelArgs.wrap_size_args 4yrVNcSSKJn UcSSKJn Un/n/n/n[URR 55Hn[ U[5(aMURSnURn [RRU5n Xn URU SU 35 URURX55 URU S35 M URR!5HupXR;aM[RRU5n Xn URSU SU 35 URURX55 URSU S35 M UR"R!5HupXR;d[ U [5(aM+[RRU5n Xn URU SU 35 URURX55 URU S35 M UR$R!5Hup[ U[&R(5(aE[+U[,R.5(a&URSU 35 URS 5 O+URSUS U 35 URSU35 URUR1U55 [RR2(dM[RR2R5U5 GM UR6(aS 5eXTU4$) NrH) INDEX_TYPE)rrz* *zconst z const float z const float zWorkspace not supported on CPU )rr{rr0r4rrdr$rrr7rrrKrur2rr3rSrrjrrUNBACKED_FLOATrx wrapper_codeensure_size_computedr5) ridtype_to_cpp_typer{r call_argsarg_defs arg_typesrqouterinnerr cpp_dtype maybe_inners rT cpp_argdefsKernelArgs.cpp_argdefss *  $ / ,   t33::<=H(J//((,E''EGG%%e,E)0I OOykE73 4   T..u< =    {!_ ->!..446LE,,,GG%%e,E)0I OOfYKr%9 :   T..u< =   vi[2 37#'"5"5";";"= E,,, ; 0S0SGG%%e,E)0I OOykK=9 :   T..u< =    {!_ -#>!MM//1LE%..>++44,ug 67  /& AeW =>  6*!67   T//6 7ww###$$99%@2&&I(II&I--rVc /n/n/n/n[URR55Hn[U[5(aMUR [ UR55 UR URS5 UR [RRURS55 UR [URURS[RRURS5S95 M [URR5UR R55HupgX`R;d[U[5(aM+UR [ U55 UR U5 UR [RRU55 UR [UU[RRU5S95 M UR"R5HupgUR [ U55 UR U5 UR [%U55 UR ['Xv55 [RR((dM[RR(R+U5 M UR,HnnUR [ UR55 UR UR.5 UR U5 UR UR05 Mp XXC4$)Nr)rfrr)r0r4rrdr$rKrrrr7rrrrr2rr3rSrzr rrr5rr) rirrrprecompile_argsrqrrrWs rTpython_argdefsKernelArgs.python_argdefss#%! ! /1t33::<=H(J// OOGH$7$78 9   X11"5 6   QWW..x/C/CB/GH I  " "!,,#//3''++H,@,@,DE  >"    $ $ &    % % ' LE ,,, 5*0M0M OOGEN +   U #   QWW..u5 6  " " ''++E2  "!MM//1LE OOGEN +   U #   T%[ )  " "75#8 9ww###$$99%@ 2&&C OOGCNN3 4   S^^ ,  " "3 '   SYY ' ' O>>rVc## [URR55Hn[U[5(aMUR HnU[ RR;dU[ RR;aMAX R;aURUUR4v X R;dM~[[URU5UR4v M M g7frn)r0r4rrdr$rr7rrrr2rr3r r~)rirqothers rTaliasesKernelArgs.aliases st33::<=H(J//!--QWW777 ; ;;...,,U3X5H5HHH///sD$7$7$>?ATATTT.>s CC> 4C>c[URRU[5[5=(a. [UR RU[5[5$rn)rdr3rr&r$r4r's rT is_removedKernelArgs.is_removedsK    # #D' 2J N--11$@*M NrVc[5n[URR55H8n[ U[ 5(aMUR URS5 M: URR5H<up4X0R;d[ U[ 5(aM+UR U5 M> U$)Nr) rr0r4rrdr$rrr3r)ri live_outsrqrrs rTlive_output_buffersKernelArgs.live_output_buffers!s%/\ t33::<=H(J// MM(..r2 3>!//557LE,,, 5*0M0M MM% 8rV)r4r2r3rSr5)rr~r,z6Union[dict[_T, Union[str, RemovedArg]], dict[_T, str]]rfrKr{r~rzr})rfr r{rr_)rLr~rMr~r{r|)rVrrrrrr{ztuple[str, str, int])r^rr{r~)rfr~rhrr{r~)rfrYr{r~)r{z Iterator[str])rfr~r{r )rNr~rrr{r~)r SymbolLiker{r~rn)rz Optional[dict[torch.dtype, str]]r{z&tuple[list[str], list[str], list[str]])r{z?tuple[list[ArgName], list[str], list[KernelArgType], list[Any]])r{zIterator[tuple[str, str]]rb)r{zOrderedSet[str])rrrrrr/rr:r=rCr2rRrrr\r`rfrrnrrrurxrrrrrrrrVrTr*r*s  E    5  ,, @B4:HM{{-1-1,0-19D-1 -1^8 7 2EI4.!A4. /4.l1? H1?f UN rVr*cx^\rSrSrSrS S U4SjjjrS SjrS SjrSSjrSSjr S Sjr S r U=r $)ri.a4A CSEVariable is just a name for an expression but it is useful to be able to annotate them on a backend dependent basis. To do so, the backends can simply overload `Kernel.create_cse_var` The "CSEVariable.update_on_args" method gives you a hook for annotations See example of TritonCSEVariable in triton.py c>[TU]5 [U[5(d[ U55eXlX lSUlX0lX@l grl) rVrrdrrzrfbounds use_countrr)rirfrrrrXs rTrCSEVariable.__init__5sG &+..<V <.    rVcUR$rnrros rTr'CSEVariable.__str__Ds yyrVc,[UR5$rn)hashrfros rT__hash__CSEVariable.__hash__GsDIIrVcb[U[5=(a URUR:H$rn)rdrrf)rirs rT__eq__CSEVariable.__eq__Js!%-I%** 2IIrVcgrnr)rirfrtrs rTupdate_on_argsCSEVariable.update_on_argsMs rVcPURRSUR<S3$r[)rXrrfros rTr:CSEVariable.__repr__Ps$..))*!DII=::rV)rrrfrrr_)rfr~rValueRanges[Any]rrrrGr})r{r)robjectr{r)rfr~rtr rr r{r|) rrrrrrr'rrrr:rrdres@rTrr.s_(, $  ! %    J ;;rVr AugmentedKeyT)default)boundr.cx\rSrSrSrSSSjjrSSjrSSjrSSjrSSjr SS jr SS jr SS jr SS jr \R"5S S SSS.SSjjr\R"5SS4SSjjr\R"5SS4SSjjrSrg)CSEi_z Common subexpression eliminationNcXlX l0UlX0lU=(d 0UlU=(d 0UlU=(d [ R"5Ul[5Ul U=(d 0Ul grn) rr^_cache name_prefix store_cachereduction_cache itertoolsriter_buffer_idsrinvalidated_stores varname_map)rirr^r iter_buffersrrrs rTr CSE.__init__bsg  FH &ALARPR  !r 6B5VY__EV3=<7B7HbrVcL/URR5QH4up#X1;dM URU URRU5 M6 U(a<URR5VVs0sHupEXQ;dM XE_M snnUlg0Ulgs snnfrn)rrrrr)ri keep_varsrftmprrs rT invalidateCSE.invalidatezs44++1134ID#$$T*''++D15 ,0KK,=,=,?R,?DA1>414,?RDKDKSs 9 B B c [U5"URURURURUR UR URS9$)N)rr^rrrrr)rzrr^rrrrrros rTclone CSE.clonesNDz;;;;((--(((( 00  rVcUR5n[UR5Ul[UR5Ul[UR5UlU$)zNReturn a copy of using ScopedDict so changes to *_cache aren't visible in self)rr+rrr)rinew_cses rT scoped_copyCSE.scoped_copysH**,#DKK0",T-A-A"B()9)9:rVc"[[U5$)z@Override this method to augment cache key with backend specifics)r rri cache_keys rT augment_keyCSE.augment_keysM9--rVc>X RURU5'grnrr)rirrOs rTputCSE.puts36 D$$Y/0rVc>URU5UR;$rn)rrrs rTcontains CSE.containss *dkk99rVcXURRURU5S5$rn)rrrrs rTtry_get CSE.try_gets"{{t// :DAArVc>URURU5$rnrrs rTrCSE.gets{{4++I677rVT)rrg assignmentrrc p[U[5(a URnU(d U(de[U[5(aEURR U5UlU=R S- sl[[U5$[U[5(aUR5nO;[U[5(a URnO[U[5(deUnURU5n Uc U(dSnU (GdURX6U5n UR!X5 U(Ga["R$R&(a(["R$R&R)USS9 [U[5(aUU(a UR+UR,U S35 UR/U5 UR+UR05 U $[U[5(aRU(deUR+UR3UR,U SURUR0355 U $U(a UR,U SUUR03n OUUR03n UR+U 5 U(a[[4R6R8(d[4R6R:(aUb[=5S:wa [?XU5 U $U RR U5U lU =R S- slU $)NrHrT) only_oncez =z = r) rdr4rhrrtightenrr rr)getvaluer&rr~rnewvarrr7r current_nodecodegen_originating_inforrsplicer^rrrrrr(r) rirr rrgrrrrrrs rTgenerate CSE.generates{ dH % %::D "" dK ( (++--f5DK NNa N. . n - - I . / / IdC(( ((Ill9% =E++fU3C HHY $88((HH))BB$CdN33!((DKK=R)@AMM$'$$T[[1: 9 &677%%:$$$++se3tyyk$++'WX4 -""&++se3tfT[[MJ"& }5$$T*#"//KK%22JJ!-/1U:#F7  ++F3CJ MMQ M rVcUR[UR53n[RR XAX#5nXPR U'U$rn)rrrr7rcreate_cse_varr)rirrrvar_namers rTr CSE.newvarsN &&'T-A-A(B'CDhh%%hE%(" rVc^[R"TUR;U4Sj5 [RR TX#U5nXPRT'U$)Nc>ST3$)Nzduplicate name: rrsrTrCSE.namedvar..s 4DTF2KrV)r _check_valuerr7rr)rirfrrrrs ` rTnamedvar CSE.namedvarsQ  (( (*K hh%%dF5A!$ rV) rrrrrrrr^r)rrrNNNN)rr~r^r~rr~rzOptional[itertools.count[int]]rz.Optional[MutableMapping[str, CSEVariableType]]rz[TU]5 [R"5Ulgrn)rVrr ExitStack exit_stackrirXs rTrCodeGen.__init__s $..0rVc:URR5 U$rn)r __enter__ros rTrCodeGen.__enter__s !!# rVc<URRXU5 grn)r__exit__)riexc_typeexc_valexc_tbs rTrCodeGen.__exit__s   F;rV)rrzrrr rr rr r{r|) rrrrrrrrrdres@rTrrs1<S"jr$S#r%U=r&$)?Kernelirr~ newvar_prefixr^Nz'Optional[Callable[[], OpsHandler[Any]]] overridescH>[TU]5 U(a[=RS- slU=(d [ 5Ul[ 5Ul[ 5Ul[ 5Ul SUl SUl SUl SUl [URUR 5Ul[%5Ul[%5UlSUlSUlSUlSUl[%5Ul[%5Ul0UlSUlSUlg)NrHFr)rVrr generated_kernel_countr*rtr)loadscomputestoresatomic_add_foundnum_load num_store num_reductionrrr^csermust_keep_buffersstore_buffer_names _load_mask _load_otherrnode_to_boundsrrinplace_update_buffersmin_elem_per_thread kernel_name)rirtincrease_kernel_countrXs rTrKernel.__init__ s   * *a / *(JL #% %' $& % .1$2D2Ddkk.R2<,3=<)-4859OS0: 3=< 79##$ *.rVc## URnXlURR5R5UlSv X lg!X lf=f7frn)rrJr get_boundsr)rir%priors rTset_current_nodeKernel.set_current_nodeEsH!! "jj//1<<> &  %  sAAAAAAc## UcUnUSL=n(a [5nURnURnURnURnXlX lX0lUR 5UlSv XPlX`lXplXlU(aU(aS5egg!XPlX`lXplXlU(aU(aS5eff=f7f)Nz$unexpected store inside swap_buffers)r)r r r rr) rilbcbsbdisallow_storesr r r rs rT swap_buffersKernel.swap_buffersOs :B Dj (? (!B ,,hh   ??$ F J"L KHEEEv2 J"L KHEEEv2sA2C5B(9/C(0CCc[ernr&rs rTr Kernel.loadmr*rVcURnURUlURX5X0l$!X0lf=f)z+A load the depends on an index we have read)r r r)rirfrrs rT indirect_loadKernel.indirect_loadps2  DJ99T)JJs!6>c[ernr&rs rTrKernel.store_reductionzr*rVc[ernr&rs rTr Kernel.store} "!rVcD[[U5RS35errr s rTr Kernel.device_assert_asyncr rVc[ernr&rs rTrKernel.reduction "!rVc[ernr&)rirfrrh extra_metas rTpartial_accumulateKernel.partial_accumulater7rVc[ernr&rs rTr Kernel.scans "!rVc[ernr&rs rTr  Kernel.sortr7rVc[ernr&ros rT var_rangesKernel.var_rangesr*rVc[e)z# See [Note: Inductor bucketize op] r&r#s rTr*Kernel.bucketizes "!rVc[ernr&ros rTassert_functionKernel.assert_functions!!rVc [U[5(a [U5n[U[5(d[U55eUb[U[5(deUb[U[5(deU(a!U(aSUSUSUSUS3 nUSUSU3nO#U(a USU3nUnOU(deUSU3nUnU(a SUSUS3nURSUSUS3$) Nr\z <= rz < rz) | ~(z, "index out of bounds: z"))rdrr~rzrF)rirrrmaskr cond_prints rTindirect_assertKernel.indirect_asserts c; ' 'c(C#s##.T#Y.#} 5# 6 666} 5# 6 666 UugT#eC5E7!s rT index_to_strKernel.index_to_strr*rVc 4>[TU]5 UR(deURR [ R "[XR5555 URR [ R"U55 U$rn) rVrrr enter_contextr7set_ops_handlerCSEProxyset_kernel_handlerrs rTrKernel.__enter__sj ~~~ %%  ht^^-=> ?  %%a&:&:4&@A rVcF>UR5 [TU] XU5 grn)remove_kernel_local_buffersrVr)rirrrrXs rTrKernel.__exit__s ((* F3rVc^^[RRmT(dg[U4SjUR55n[5mURHmnX R ;dMX R R;dM/TRX!5(dMGU=RS-sl TRU5 Mo THnX R R;aUR RUn[U[5(aML[U4SjUR55nU(aUR!U5 UR"RU5 MUR%U5 M g)z Any buffers that are both created and have a last use in the same kernel can be removed. Note that V.graph.scheduler can be None when codegening triton template kernels. Nc3># UH4nUTR;dMTRUR5v M6 g7frn) name_to_bufdefining_op_name)rrNrIs rTr5Kernel.remove_kernel_local_buffers.. s?& .i+++ :I ! !# & 7 7 9 9.s?$?rHc3,># UH oT;v M g7frnr)rrnames_to_removes rTrr_ sK?a/1?s)r7rrIrrrrtr2$can_buffer_be_removed_through_fusionrrr4rdr$r!rremove_inplace_bufferr remove_buffer)rifused_node_namesrfrNrerarIs @@rTrY"Kernel.remove_kernel_local_bufferss(GG%%  %& ..&   ,6<++D222 7 77BB!###D),$Dyy000ii//5c:..K3??KK..t4''++D1""4($rVc[RSU5 [URRU'UR R U5 g)Nzremove_buffer(%r))rrRr&rtr3rrr's rTrdKernel.remove_buffer s; %t,)0   &   &rVc[RSU5 [URRU'UR R U5 g)Nzremoving_inplace_buffer(%r))rrRr&rtr4rrr's rTrcKernel.remove_inplace_buffer% s9 /6*1 !!$'   &rVc [U[[45(a!UVs/sHo RU5PM sn$[R R RU5n[URSS9nUVs0sHmn[U[R[R[R[R45(dMRX R R#U5_Mo nn[%X5$s snfs snf)NcUR$rnr)ss rTr(Kernel.rename_indexing..2 s!&&rV)r)rdlisttuplerename_indexingr7rrSrPsorted free_symbolsrr UNBACKED_INTSIZEPRECOMPUTED_SIZEr~rtrr.)rirrpsorted_symbols replacementss rTrqKernel.rename_indexing* s edE] + +5:;U((+U; ;  ))%0 2 28HI$  #%%II))''  !Ayy~~a #   %..!<  sD?ADDc[U0UD6$rn)r)rirtrs rTrKernel.create_cse_varB sD+F++rVc\UcgURRUR55$)z3 Returns arg name of a given input or output node. N)rtrrr)rir%s rTrrKernel.arg_nameE s' <yy!!$--/22rV)rrrtrr rrrrrr rrrrrrrrr )NT)rtzOptional[KernelArgs]rrr{r|)r%rFr{rr_)r#r)r$Optional[IndentedBuffer]r%r~r{rrfr~rrr{rrfr~rrrhrr{r|rn rfr~rrrhrr]r6r{r|r[ rrrrrr5rh+Union[CSEVariable, tuple[CSEVariable, ...]]r{r) rfr~rr5rhrr9dict[str, Any]r{r|rr]rzUCallable[[tuple[CSEVariable, ...], tuple[CSEVariable, ...]], tuple[CSEVariable, ...]]rtuple[CSEVariable, ...]r{r rr]rrrrrrr{r)r{zdict[sympy.Symbol, sympy.Expr]rrr$r`r%rr&rr'rr(rar)zOptional[CSEVariable]r{rr}) rzUnion[CSEVariable, str]rr rr rIz!Optional[Union[CSEVariable, str]]r{r~rZ)rrr{r~rrrz)rfr~r{r|)rz;Union[list[sympy.Expr], tuple[sympy.Expr, ...], sympy.Expr]r{r)rtr rr r{r)r%rAr{r )'rrrrrrr^rrrrr r'rr,rrr rr:rr rAr*rrFrKrrPrrrYrdrcrqrrrrrdres@rTrrsM3FC9=I6=PT#/(#/HL#/ #/#/J&&(,'+ F F %F % F  FF:""SW"" *"3>"FO" "  """& " ; " 5 """&" " # "  ""'" " ( " !""'"(" "  " ! ""4804 " "C "& " $ "  "1 ". "  """37 V $VV V 0 V V<""&0"9="FJ" " "4&)P'' /P/ /0,33rVrc@\rSrSr%SrS\S'SrS\S'SrS \S 'S rg) riN r>z ClassVar[str]rNrrrr~ops_namer) rrrrrrrrrrrVrTrrN s!"C"#'E 'HcrVrc^SSKnURURS9$![a gf=f)Nr) undefined)jinja2 EnvironmentStrictUndefined ImportError)rs rT jinja2_envrV s?!!,,"   s  ,,c\rSrSrSr\SSSjj5r\SSj5r\SSj5rSSSjjr \ SSj5r \ SS j5r SS jr SS jrSS jrS rg)KernelTemplateib z[ Base class for defining kernel templates. Children classes: TritonTemplate, CUDATemplate cURS5n[U5S:a"USSVs/sHnSU-U-U-PM snUSS&SRU5$s snf)NTrHr}r) splitlinesrr)source num_indentsindents_spacinglinesrs rTindent_except_first"KernelTemplate.indent_except_firsti sh!!$' u:>INqrIR&4<E!"Iwwu~sAc[5nUcg[RURS'SSKJn UR U5$!Uan"SSU5nU"U5UeSnAff=f)Nrr)TemplateSyntaxErrorc6^\rSrSrSU4SjjrSSjrSrU=r$)IKernelTemplate._template_from_string..DetailedTemplateSyntaxErrori c>[TU]URURURUR 5 Xlgrn)rVrmessagelinenorffilenameoriginal_error)rirrXs rTrRKernelTemplate._template_from_string..DetailedTemplateSyntaxError.__init__ s<G$&..&--&++&//  +9'rVcXSURS3nUSURS3- n[URS5(aURRR S5nUS- n[ SURS- 5n[[U5URS-5n[X45HwnXPRS- :XaSXS-S X%S3- n[URS 5(a'US S URRS- --S -- nMfMhXS-SX%S3- nMy U$)NzError in template at line  zError message: rz Context: rrrHz: --> columnz r}z^ z: ) rrrUrrsplitmaxminrrr)ri error_inforstartendrs rTr'QKernelTemplate._template_from_string..DetailedTemplateSyntaxError.__str__ s:#=dkk]"!MJODLL>"DDJt22H== $ 3 3 : : @ @ F"l2 #At{{Q 7!#e*dkkAo>!&u!2A KK!O3 *QvehZr.J J #*4+>+>#I#I$.(/*-1D1D1K1Ka1O*P)Q*/)0%&J$J!+QvehZr.J J "3&%rV)r)rrr{r|r})rrrrrr'rrdres@rTDetailedTemplateSyntaxErrorr s9&&rVr)rrrfiltersrr from_string)renvrers rT_template_from_string$KernelTemplate._template_from_stringt sjl ;-;-O-O )*.# 8??6* *"! 8 &.A &>.a0a 7C! 8sAA# AA#c0^^[RRm[U[[ 45(a0UVs0sH!oR 5UR5_M# snmO UR 5UR50mSUU4SjjnU$s snf)Nc@>TRU5nUbU$T"U5$rn)r)rfr-_get_dtype_reallookups rTr1KernelTemplate._fake_get_dtype..get_dtype s'ZZ%F! "4( (rV)rfr~r{r)r7rrrdrorpr) fake_outsrNrrrs @@rT_fake_get_dtypeKernelTemplate._fake_get_dtype s|''++ i$ / /AJK#llncmmo5KF((*I,?,?,ABF ) ) Ls(BNcXlX lgrn)rf_hash)rirfrs rTrKernelTemplate.__init__ s   rVcUR$)z entry point to override for templates to ensure a uid e.g. through a prefix the purpose of this is that every KernelTemplate/ExternKernelChoice is unique in the system, but reproducible e.g. restarting pytorch should yield the same id rros rTuidKernelTemplate.uid syyrVcUR$)z source hash for a Template. Templates can optionally provide a src hash to make it easier to cache/validate that a template has not changed from one version to another. Override this if that detection is different for your specific Template )rros rTsrc_hashKernelTemplate.src_hash szzrVc \/nUR"U40UD6nUc[U5S:XaUS$g)z Maybe generates a new ChoiceCaller and returns it, or None if generation fails. kwargs: Additional kwargs to be passed to self.generate() to generate a new ChoiceCaller. NrHr)maybe_append_choicer)rir temp_choicesr-s rTchoice_or_noneKernelTemplate.choice_or_none s< #% )),A&A >c,/14? "rVc URUR"S0UD65 g![aNn[R SU[ U5[R 5[R:S9 UsSnA$SnAff=f)z Maybe generates a new ChoiceCaller and appends it into existing choices. Returns None if success, otherwise returns the error. choices: A list of ChoiceCallers. kwargs: Additional kwargs to be passed to self.generate() to generate a new ChoiceCaller. Nz3Cannot Append Choice: %s. KernelTemplate type is %s) stack_infor) rKrrrinforzgetEffectiveLevelrPINFO)richoicesrrs rTr"KernelTemplate.maybe_append_choice sl  NN4==262 3"  HHET 002W\\A   H s!$ A<AA71A<7A<c [e)z= Generates a ChoiceCaller instance from the given arguments. r&)rirs rTrKernelTemplate.generate s "!rV)rrf))rr~rrrrr{r~)rr~r{r )rzUnion[list[Buffer], Buffer]r{zCallable[[str], torch.dtype]rn)rfr~rr r{r|r})r{zUnion[str, None])rr r{zOptional[ChoiceCaller])rrrr r{zOptional[NotImplementedError])rr r{r?)rrrrrrrrrrrrrrrrrrrVrTrrb s >?"%8; *8*8X. %"  ,/ &."rVrcd^\rSrSrSrSrSU4SjjrSSjrSSjrSSSjjr SSjr SSjr SS jr SSS jjr SS jrSS jrSS jrS SjrS!SjrS"SjrS#S$SjjrSrU=r$)%rUi zvA ops handler that proxies calls to `kernel` and its handler and returns `CSEVariable`s with correct shape and dtype. c^>[TU]5 SSKJn U"5UlXlX lg)NrValueRangeAnalysis)rVrrr vr_analysisrparent_handler)rirrrrXs rTrCSEProxy.__init__ s' /-/ ,rVc z^^^^ ^ ^ ^ ^UR"T/TQ70TD6m [URT5"T0TD6n[5n[ 5n[ 5m [UT5nSm SmTS:XaT S:XaUR m URmO}TS:XaUT S:XaO[RRRR[RS5R m SmO"T S;a[UT5nU"T0TD6m U"T0TD6mT S;aT ceSm SUU U UUU U U4Sjjn [R "X5$) Nmaskedrr)rrr)rrrc>[T [[45(aT T OT n[T [[45(a2[T 5S:a#[T S[[45(aT T OT nT S- m [U[5(a,TS:XaUR cXlUR cX l[RRR[RRUTT T S9nURTTT5 [RR(d[RR (a)Uce[#[RRX15 [RR$(a*T ce['[RRUT 5 [R((a$[+[RRU5 U$)NrrHrrrr)rdrorprrrrr7rrrr rrrrrrrrruntime_triton_nan_assertsr) r var_dtype var_shapecsevarrtrrrrfr9 output_idx output_shapes rTdo_cse!CSEProxy._default..do_cse( s lT5M::Z(! lT5M:: %)|Au >>Z("  !OJ![))e#'G77?'GXX\\**  "" +F  ! !$f 5##??&&>> ,,,AHH,,f@"">>#///AHH,,flC00!((**F3MrV)rzUnion[str, CSEVariable]r{r)_bound_variablerHrr!r$r(rrr7 interpreterrrrrrpytreetree_map)rirfrtrrh dtype_handler shape_handlershape_opdtype_oprrrr9rrs ``` @@@@@rT_defaultCSEProxy._default s:%%d>#''e  L 0 0}d3H#T4V4L#T4V4L ' '+ ++ 0 0 dv--rVc>^ SSKJn SSKJn SSKJn [ [RU5(a[R"5$[ [RU5(a[R"5$[ [R[5(a[R"5$[RRm T RU:XaURRb[ URR[ 5(d$[#URR55eURRR%T [R"55$[&R((az[+XA5(aj[-U 4SjS55(a[R"5$U(aeS Sjn[/[1Xr55n[3UR4U5"U6$[R"5$) z If the variable comes from an FX node, we forward the bound we have already computed Else, if the variable when codegen'ing another op, we try to compute its bounds rr)TritonTemplateKernelrH)CUDATemplateKernelc3@># UHoTR;v M g7frn)r,)rrmfx_nodes rTr+CSEProxy._bound_variable..w sV0U1&0Us) set_indirectrrc[U[5(a UR$[U[R5(a [ U5$U$rn)rdrrrrTrros rT arg_to_bound.CSEProxy._bound_variable..arg_to_bound s8a--88O5::..&q>)HrV)rpr r{r )rrselect_algorithmrcuda.cuda_kernelrrdr7rrrrr1rr,rdictrzrrcompute_all_boundsrUrror9rHr) rirfrtrrrrr arg_boundsrs @rTrCSEProxy._bound_variable\ sy 0;8 ahh 4 5 5&&( ( ahh 2 3 3&&( ( amm[ 1 1&&( (--,, >>T !dkk&@&@&Ldkk88$??  **B ?;;--11';;N;N;PQ Q  & &73E+L+LV0UVVV"**,,  : c,56J4++T2J? ?""$$rVc[U[5(a[R"U5n[U[R5(d[ U5U45eUR RS:GaU(a[R"U[R"U[R55nUR RS:a.[R"US5n[R"XeU5nOUn[ R""5nUR [ R""5:wa[U[R$5(aUR [![&*S5-n[!URU-URU-5nUR RS:a!UR [!S[&5-n Xy-nUR(R*R-UR(R.UUUR0UR2S9nUR4R7XU5n [9U5(avUR RS:+n [U[R$5(+=(d UR RU:+n UR(R;XX5 U $)Nrrr)rdrrrrTrzrrr2rrrlongrltrrrNumberrrrrr rrrrr'r) rirrrrstmr new_bounds neg_boundspos sympy_var assert_lower assert_uppers rTrCSEProxy.indirect_indexing s dC ==&D$ ++?d4j$-??+ ::  a ggc3>>$ #CD::##q(QB))BS1C%,,.Jzz[0022z$ 7U7U!ZZ+vgr*BB ($$t+Z-=-=-D ::##q(**{1f'==C!+!1J++//** ##!iiii +C''99#UK 5 ! ! # 0 0A6L)$ ==   4BL KK $ $Yl QrVc:URRXX45$rn)rrrs rTrCSEProxy.check_bounds s{{''EAArVcXRRR;a)[RRR U5 [ U[R5(aURRX5$URRRnX;aX1$URRX5nURS:XaUR=RS- sl U$rl)rrrr7rrrrTMPr,rrrr)rirfrrouts rTr CSEProxy.load s ;;??55 5 HH & & * *4 0 udhh / /;;,,T9 9kkoo11  $ $kkt+ ==A  KK A %  rVczX RRRU'URR(a~U[R R ;a_URRRU5nUR5H%nX RRRU'M' gggrn) rrrrr7rname_to_buffer get_output get_mutations)rirfrhrN other_names rT_update_store_cacheCSEProxy._update_store_cache s,1 ##D) ;; # #0F0F(F++**55d;C!//1 :? ++J72)G #rVc&URRRU5 UcURX5 U[R R ;a:URRXX4S9 UR=RS- slgg)N)r]rH) rrrrr7rrrrrs rTrCSEProxy.store sr &&**40 <  $ $T 1 qww.. . KK  d5  < KK ! !Q & ! /rVc:URRX5 grn)rr r s rTr CSEProxy.device_assert_async s ''2rVc6URR"U6 grn)rr:rss rTr:CSEProxy.partial_accumulate s &&-rVc"URRRU5 URX5 U[R R ;a;UR=RS- slURRXU5$grl) rrrrr7rrrrrs rTrCSEProxy.store_reduction sk &&**40   - qww.. . KK ! !Q & !;;..tEB B /rVcxUR=RS- slURRXX45$rl)rrrrs rTrCSEProxy.reduction s0 !!Q&!{{$$U~MMrVc:URRXU5$rn)rrrs rTr CSEProxy.scan s{{F;;rVc:URRXX45$rn)rr rs rTr  CSEProxy.sort s{{CCrVc DURRUUUUUUU5$)a [Note: Inductor bucketize op] Inputs: ------- values: the values to be bucketized. boundaries: a tuple containing (a) the name of the boundaries tensor (which must be sorted, unless the sorting tensor is present), (b) the length of the tensor in the last dimension (i.e. the length of one set of boundaries), (c) the number of elements in the underlying storage (i.e. the length of the flattened tensor, ignoring striding), and (d) the stride of the tensor in the last dimension. boundary_indices: indices into a flattened version of the boundaries tensor, of the same size and shape as "values". Each index points to the first element in the set of boundaries to be used for the corresponding value. indexing_dtype: the dtype to use when indexing into the boundaries tensor. This must be int64 or int32. This additionally specifies the dtype of the return value. right: see "Details" below. sorter: an optional tuple containing (a) the name of an optional sorting tensor, used to access unsorted boundaries without reordering the boundaries tensor, and (b) the stride of the tensor in the last dimension. The values in the sorting tensor are used as indices into the *last* dimension of the boundaries tensor, with all other indices matching. The size of the sorting and boundaries tensors must be equivalent. sorter_indices: must be present if the sorting array is present; see "boundary_indices" for the equivalent definition for the boundaries tensor. Output: ------- The buckets each value belongs in, within a given set of boundaries. 0 indicates a position before the first boundary, and len(boundaries_set) represents a position after the last boundary. Details: -------- Given a value and a set of boundaries, calculate the bucket that each value belongs to. This works differently in 1-D and N-D cases. for values [[-1, 0, 1, 2], [3, 4, 5, 9]], boundaries [0, 4, 4, 8], right=True return = [[ 0, 1, 1, 1], [1, 3, 3, 4]]. for values [[-1, 0, 1, 2], [3, 4, 5, 9]], boundaries [[0, 4], [4, 8]], right=True return = [[ 0, 1, 1, 1], [0, 1, 1, 2]] Note that in the N-D boundaries case, the shape of "values" and "boundaries" must match in every dimension _except_ the last. When right == False, bucket i refers to range (boundaries[i], boundaries[i+1]]. When right == True, bucket i refers to range [boundaries[i], boundaries[i+1]). Boundaries must be non-decreasing, or a sorter must be provided which would re-index offsets in a non-decreasing order (e.g. the second output of torch.sort(offsets)). Otherwise, the result is undefined. )rr*r#s rTr*CSEProxy.bucketize s1L{{$$         rV)rrr)rz Kernel[Any]rzOpsHandler[Any])rfr~rtztuple[Any, ...]rrr{r )rfr~rtr rr r{rrW) rrrrXrrrrr{rYrZr)rfr~rhrr{r|rnrr[)rtr r{r|rrrrr_r)rrrrrrfrrrrrrrrr r:rrrr r*rrdres@rTrUrU s D-S.j.%h 5 5%5 5  5  5nBB&0B9=BFJB B "@SW'' *'3>'FO' '3.CNNN& N ; N 5 N <' <  <( < ! <D'D(D D  D ! D 4804N N CN & N $ N  N 1N .N  N N rVrU)rSr~r{r|)NNNN)rr~rerrfrrgrrhrriOptional[CustomGraphModulePass]rjOptional[ConfigModule]r{r|)rUnion[torch.device, str, None]r{zOrderedSet[BackendFeature])rr/rrmr{r)rr~r{zOptional[SchedulingConstructor])FF)rr~rrrrr{r)rr~r{r-)rr~r{r.rz)rSequence[sympy.Expr]rr0rr0r{r)rr~rr$r{r|)rr~r{r$)rr~rtr rr r{r)rr)rrrrr{r|)rr)rrrrGr{r|)rr)rrr{r|)rr~r{rrrb)r{r ) __future__rrbr dataclassesenumr"rrPrr4rdrer`abcrrrrrtypingr r r r r rrrtyping_extensionsrrrrtorch.fxtorch._prims_commonr torch.utilsrrtorch.utils._config_modulertorch.utils._ordered_setrtorch.utils._sympy.numbersrtorch.utils._sympy.printersr_PythonPrintertorch.utils._sympy.symbolrrrtorch.utils._sympy.value_rangesrrrrr dtype_propagationr! ops_handlerr"r#shape_propagationr$utilsr%r&r'r(r)r*r+r,r-r.r/r0 virtualizedr1r2r3r4r5r6r7collections.abcr8r9r:r;r<custom_graph_passr=rr>r?r@rArHrCrIrDrErFrGrrJrKrrzrr~rjrr_logginggetArtifactLoggerrrN getLoggerrrU dataclassrXrrrrrr rrr KernelArgTyper"rr$r`rarbrkrmr|rr{rrrcacheryrrrbfloat16rfloat16rryfloat64int8int16rnrruint16r]uint64rrrrrr rgcompile IGNORECASErrrrdr INT_TO_FLOATrOrrr rrr$r&r*rrrrprReductionCacheKeyrrrrrrrUrs0rTrZs1 "    #   , ?)3/-GOOD:;:    LL$9>>$DD2- B$hy&9%:N%JK23sELL()JF~~//*E  !=  >/ /(  C  Td=ddN** ###   !!! <<<lIw8H,VW ,.).5"5"p:<6;DFAFDF AF8@D>B:>37B B,B/B!= B < B 8 B1B B4 &T &3 *33$3 *35C3 3U @E   " 8< ! -6 ccLU U$U U U; ;&7; ; , NNEKK MM5;;> JJ MM MM JJ KK KK KK KK LL LL LL    E    >:,' ' '' 'T: :!0:9D: :2 L  L!0 L9G L  L aBaBH=N=$S1S1lzz";2==Q  O;#%5z#O;d  - - - 6:`6 ;HH -  `6 ;HH /-  `6;HH /- `6$;HH /- %`60;HH /- 1`6< ;HH *)   =`6L ;HH (0   M`6X ;HH 66>   Y`6h ;HH 1  i`6r ;HH 2  s`6|;HH 1 }`6F;HH 2 G`6P;HH %8$  Q`6^ ;HH &%   _`6j;HH %8  k`6v ;HH &  w`6@;HH + A`6L%;HH 88 ) M`6X%;HH 88 ) Y`6d%;HH 8 )e`6n%;HH 8 )o`6z ;HH '  {`6D ;HH (  E`6N;HH c  O`6^,;HH ? 0_`6h,;HH ? 0i`6t&;HH 9 *u`6~ ;HH *  `6H);HH D -I`6R);HH D -S`6\);HH D -]`6f);HH D -g`6p(;HH C ,q`6z$1;HH L 5${`6D$1;HH L 5$E`6N$1;HH L 5$O`6X$1;HH L 5$Y`6b';HH B +c`6l(;HH C ,m`6v(;HH C ,w`62`F -#-">*Z  NNN ,XXv #;#;L5 +; T  k5c!1223 5l'/=0 1l^ < <p3Wgo.p3f   S"S"lm ~m };s/ d