ȅi5<SSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK J r SSKJ r SSKJrJrJrJrJr SSKrSSKJr SSKrSSKrSSKrSSKJrJr SSKJr SS KJrJr SS K J!r! SS K"J#r# SS KJ$r$J%r% SS K&J'r'J(r( SSK)J*r* SSK+J,r, SSK-J.r. SSK/J0r0J1r1 SSK2J3r3J4r4J5r5J6r6J7r7J8r8J9r9J:r: SSK;Jr> SSK?J@r@ SSKAJBrB SSKCJDrD SSKEJFrFJGrGJHrH SSKIJJrJJKrKJLrLJMrMJNrNJOrOJPrPJQrQ SSKRJSrSJTrTJUrUJVrV SSKWJXrX SSKGJYrYJZrZJ[r[J\r\J]r]J^r^J_r_J`r`JaraJbrbJcrcJdrdJere SSKfJgrgJhrhJiriJjrjJkrkJlrlJmrmJnrnJoroJprpJqrqJrrr SSKsJtrt SS KuJvrv SS!KwJxrx SS"KyJzrzJ{r{J|r|J}r}J~r~JrJrJrJrJrJrJrJr SS#KJrJr \(a=SS$KJrJrJrJr SS%KJr SS&KJr SS'KJr SS(KJr SS)KJr SS*KJr SS+KJr \\\L4rSS,KJr \GR>"\5r\GRDGRG\S-5r\GRJGRLr\GRN"5r\FGRR"5(aSS.KJr OS:S/jrS;S0jrSS3jrS?S4jrS@S5jr"S6S7\GRdGRf5r"S8S9\5rg)A) annotationsN) defaultdict)contextmanager)AnyNoReturnOptional TYPE_CHECKINGUnion)Expr)deviceTensor)get_decompositions)defake dynamo_timed)FakeScriptObject)get_layout_constraint_tag) LazyStringtrace_structured)compute_required_storage_lengthmake_channels_last_strides_for) FakeTensor)full_aoti_runtime_assert) BackwardState) magic_methodsmethod_to_operator)_get_placeholder_exprfree_unbacked_symbolshas_free_symbolsresolve_unbacked_bindings RuntimeAssertShapeEnv SympyBooleanSymTypes)Node) _is_view_op) no_dispatch) OrderedSet)int_oo)configirmetrics)BackendFeatureDeviceOpOverridesFileBackedGraphModuleget_backend_featuresget_device_op_overridesget_wrapper_codegen_for_deviceinit_backend_registration WorkspaceArg)CppWrapperCodegenErrorLoweringExceptionMissingOperatorWithDecompMissingOperatorWithoutDecomp)count_flops_fx) assign_origin_nodeConstant DonatedBuffer FixedLayoutget_device_typeGraphPartitionSignature InputBuffer Pointwise ReductionShapeAsConstantBuffer StorageBox TensorBoxTorchBindObject) constrain_to_fake_tensorsconstrain_to_fx_stridesFALLBACK_ALLOW_LISTfallback_handler%fallback_node_due_to_unsupported_type lowerings make_fallbackmaybe_layout_constraintsneeds_realized_inputsrequire_contiguoustag_to_layout_constraintunsupported_output_tensor)autotune_cache)AutotuneCacheBundler)SizeVarAllocator) convert_shape_to_inductorgather_origins get_cloned_parameter_buffer_nameget_donated_idxsget_sympy_Expr_dtypeGraphPartitionMapis_same_tensor#maybe_get_suppress_shape_guards_ctxnormalize_nameshould_assume_input_alignedshould_fallback_by_defaultSUPPORTED_MKLDNN_DEVICESValueWithLineMap) NullHandlerV)CallableIterableIteratorSequence) ModuleType) _EffectType) GraphModule)Graph)PythonWrapperCodegen)Dep)BaseSchedulerNode)output_code_log perf_hints)log_module_codecgN)argskwargss O/home/james-whalen/.local/lib/python3.13/site-packages/torch/_inductor/graph.pyrrrrs c[U[R[R[RR R 45(dS5e[U[RR R 5(a[R$[U[R5(a [U5$UR(a[R$UR(a[R$g)Nzgget_constant_buffer_dtype only supports input of sympy.Symbol, sympy.Expr or sympy.core.numbers.Integer) isinstancesympySymbolr corenumbersIntegertorchint64rZ is_integeris_floatfloat32)constant_buffers rxmay_get_constant_buffer_dtypers %,, EJJ4F4F4N4NO   r  /5::#5#5#=#=>>{{/5::..#O44!!{{  ! !}}ryc6[S[55nX;$)Nc38# UHn[U5v M g7frt)r).0ms rx "is_magic_method..sH-Q-a00-s)r'r)op magic_opss rxis_magic_methodrsH-HHI ?ryc URS5nUn[U5H@upE[X55(d [SSR USU535e[ X55nMB U$)N.z#Node referenced nonexistent target )split enumeratehasattr RuntimeErrorjoingetattr)objtarget target_atomsattr_itriatoms rxgetattr_recursiversn<<$LH\*x&&5chh|BQ?O6P5QR 8* + Orycn0nURSS9SnSUR;aU$[URS[R R 5(dURSnO URn[U5H-upEXBRS;dMURSUX'M/ U$)Noutputrruser_visible_output_idxsoriginal_output_strides) find_nodesmetar{rvrfxr$r)gret output_nodeoutput_node_argsidxnodes rxget_user_visible_output_stridesrs')C,,(,+A.K!)9)99 k&&q)588== 9 9&++A.&++/0  ""#=> >#(()BCCHCI1 Jryc0UEn/UR5Qn[/UQ5nU(aUR5n[UR5(aUR (a~[ UR S[RR5(aHUR SnXS;a4URUS5 URU5 URU5 U(aMU$)z[ Extend user_visible_output_strides to include view ops that lead to user-visible outputs. rN) keysr'popr%rrvr{rrr$ setdefaultaddappend)user_visible_outputsresultqueuevisitedcurrentbases rx"extend_user_visible_output_stridesrs ":$8!9F fkkm E5"G ))+  ' ' 7<<?EHHMM::<<?D"!!$- D! T" % Mrycz[R(dg[U5n[[R [R [R/5n[[R[R[R[R[R[R[R[R[R [R"[R$[R&/ 5nSSjn[)UR*5GHn[-UR.[0R2R4R65(aSUR8S'MR[-UR.[0R:R<5(aL[?UR.5[0R@RBRD:XaSUR8S'MU"U5nU(dMXs;aSUR8S'UR8RGSS5(a:URHH*nU"U5n U (dMX;dMSUR8S'M, [RJ(aGMiXb;dGMqSUR8S'GM g)ar Nodes like convolution/convolution_backward want its input to be dense. If we pad their inputs, we result in extra calls to copy kernels! On the other hand, padding usually helps reduction. The pass finds nodes that dislike padding. These are nodes that can be reached from a convolution/convolution_backward in the backward direction without going thru a reduction. NcURS:Xad[UR[RR 5(a1[ URS5(aURR$S$)N call_function_overloadpacket)rr{rr_ops OpOverloadrr)rs rx_get_overload_packet8mark_nodes_dislike_padding.._get_overload_packets[ ww/)4;; (=(=>> %677 KK ' '   ryTdislike_paddingF)r torch.fx.Nodereturnz%Optional[torch._ops.OpOverloadPacket])&r*comprehensive_paddingrr'aten convolutionconvolution_backward _scaled_mmvar_meansummeanprodanyaminamaxminmaxargminargmaxscatter_reducereversednodesr{rr_higher_order_opstriton_kernel_wrapTritonKernelWrapperMutationrrrr_CTagneeds_exact_stridesgetall_input_nodes pad_outputs) ruser_visible_output_stridesextended_user_visible_nodesops_dislike_paddingops_like_paddingrcurrpriorprior_ops rxmark_nodes_dislike_paddingrs  ' '"D##%     % % OO " MM HH II II HH II II HH HH KK KK    "    .     JJ  # # 6 6 R R  +/CHH& '  szz5::#8#8 9 9)#**5xx||//0+/CHH& '  !# &  $*.CHH& ' 88<<)5 1 1,,/6348EJJ01 -!!!c&H*.CHH& 'A!rycx^\rSrSr%S\S'SFSGU4SjjjrSHSjrSISjrSJS jrSKS jr SLS jr SMS jr SNSOS jjr SPSjr \RSQSj5rSRSjr\SSSj5rSTSjrSUSjrSVSjrSWSjrSXSjr\SYSj5rSZSjrS[SjrS\SjrS]SjrS^SjrS_U4Sjjr S`Sjr!SS.SaS jjr"SbS!jr#ScS"jr$SWS#jr%SdS$jr&SeS%jr'SfSgS&jjr(ShS'jr)SiU4S(jjr*SjU4S)jjr+\SkS*j5r,SlS+jr-SmS,jr.SmS-jr/SnU4S.jjr0SHS/jr1\SoS0j5r2\SpS1j5r3SqS2jr4SrU4S3jjr5SsS4jr6SHS5jr7StSuS6jjr8SvS7jr9SwS8jr:SHS9jr;SwS:jrSr?S=\S>'SzS?jr@SzS@jrAS{SAjrBS|SBjrCS}SCjrDS~SDjrESErFU=rG$) GraphLoweringiBlist[ir.IRNode] graph_outputsNFc >[TU]U5 X lUbUOURXS9UlSUlXlXlXlXl Xl Xl UUl SUl Uc[5nSUlOSUlX0lUR"R%5Ul[([*R,"5Ul[1U5Ul/Ul0Ul0UlSUl[)5UlU(a UR>O [)5UlU(a UR@O [)5Ul SUl![E[(5Ul#[E[(5Ul$0Ul%/Ul&/Ul'U (aU O0Ul(U (a[)U RS55O [)5Ul*U(a URVO0Ul+U(a URXO0Ul,URZR]S[)55Ul/U(a UR`O0Ul00Ul10Ul20Ul30Ul4[)5Ul5[)5Ul6[)5Ul7[)5Ul8[)5Ul9[)5Ul:SUl;SUl<SSK=J>n [~R"5(a U(aUOUUlASUlB0UlC[)5UlD/UlE0UlF[E[5UlH0UlI[R"5UlKUUlLXPlMUUlNXPlO0UlPX`lQX@lR[[5UlUSUlVSUlWSUlXSUlYSUlZUR(aUR5O [)5Ul\[)S/5Ul][UR5Ul`[URUR5 S UlbS Ulc/UldSUle0UlfUR5UlhURRY5HunnUURXU'M URRa5HunnUUR`U'M URRZR]S 05UljUb URO0Ulk[5 [R"S5"[5Ulo0Ulp[)5Ulq[)5Ulr[)5Uls[)5Ult[)5Ulu[R"5UlxS Uly[5Ul{0Ul|g) N) is_inferencerFTcpumutated_named_buffers)extern_node_json_serializerzaten.convolution_backward dynamo_flat_name_to_original_fqn)}super__init__example_inputsdecide_layout_opt layout_optnum_channels_last_convr is_backwardis_const_graphconst_wrapper_codeconst_kernel_code const_moduleinputs_to_checkextra_tracebackr!reuse_shape_env _shape_envdeferred_runtime_assertscopy ras_by_symbolr'r|r}bound_unbacked_symbolsrUsizevarsgraph_input_names graph_inputsgraph_inputs_originalpartition_mapszero_dim_cpu_tensor_list device_types device_idxs device_typeradditional_buffer_depsadditional_star_depsbuffer_to_padded_sizebuffers operationsconst_output_indexrfolded_constants constants named_buffersrrrnamed_parameterstorchbind_constantsopaque_value_type_classesseen_subgraphsconstant_reprsremoved_operationsremoved_buffersremoved_inplace_buffersmutated_buffersnever_reuse_buffersinplaced_to_remove device_ops wrapper_code&torch._inductor.extern_node_serializerrr* is_fbcodeextern_node_serializer current_nodelistsmutated_inputsmutated_input_idxsname_to_bufferlist name_to_users name_to_optime creation_timename cpp_wrapper fx_wrapperrecord_multi_kernel_choicemulti_kernel_to_choiceaot_modegraph_idnext_post_grad_graph_counterpost_grad_graph_id schedulerautotuning_inputsautotuning_mappingautotuning_gridscurrent_devicefind_nodes_prefer_channels_lastnodes_prefer_channels_last_warned_fallbackrgraphrr cache_key cache_path cache_linemapdisable_cudagraphs_reasondevice_node_mapping__copy__orig_gmmodulerallocated_constant_namer3 functools lru_cacher0 effectful_opsunaligned_buffersno_fuse_buffer_nameslow_precision_codegen_opsinvoke_quant_opsall_codegen_kernel_names itertoolscount workspace_idplaceholder_idxrYbw_donated_idxsdep_size_hint_cache)selfgmr shape_envr<r7r;rr+rrrrrrrr6rr8rkv __class__s rxrGraphLowering.__init__Es. ,% '''F  '(#(&,"4!2(.$   I#(D #'D #  . . 3 3 5 '1&>&@#(3 ,.VX=?"AE9C%)5L % %:< )5L $ $*, !BM C #ALJ@W!<>"(* .0"4 "  " )..0 1  '3L " " +7L & &B @Bww{{ #Z\@ ".:L ) )r    ;=&68.03=<0: 8B $0: 4>L 3=<-126V!!&< #, # ,0+- /9|-/46@KD@Q35!YY[ &$ +6'68#  "&'?"@>B@DGK:>7;7;ooD 0 0 2:< '!+,G+H I+J288+T("288T-M-MN !  9=&GI -/[[] LL..0DAq$%D  q !1LL113DAq'(D ! !! $404 0@0@0D0D .1 -5A4LL 0 0RT $ "#$-$7$7$=>R$S!;=3=,5?\!:D,&1;:D%&OO- "/1AC ryc8URR5 grt)rfreeze_runtime_assertsr`s rxrh$GraphLowering.freeze_runtime_assertss ..0rycZUR(a2[UR55[UR554$SSKJn U"S[ URR535nURRUU5unnnUVs/sH:n[U[R5(aURROUPM< nnUVs/sH:n[U[R5(aURROUPM< n nX4$s snfs snf)z Support dynamic shapes and dynamic strides by assigning variables to each dimension. We duck-shape tensors, so if two tensors have the same size they get assigned the same symbolic variable. r)ConstantSource__inductor_unknown_tensor_)rrVsizestridetorch._dynamo.sourcerllenr var_to_val,create_symbolic_sizes_strides_storage_offsetr{rSymIntrexpr) r`exrlsourcernro_rr_sizer_strides rxsymbolic_sizes_strides$GraphLowering.symbolic_sizes_stridess   ,RWWY79R :  <$,S1K1K-L,MNFLL   NRRTAu||!#>AFFKKAEvVSVsAD#AD(cUR5Vs/sHn[R"U5PM nnUR5Vs/sHn[R"U5PM nnX44$s snfs snf)z Primarily used to weights )rnr|rro)r`rvrrnros rxstatic_sizes_strides"GraphLowering.static_sizes_strides<sZ +-'')4)Q a )4,.IIK8Kq%--"K8|58s A, A1cr[U[R5(a URn[U[R5(a URn[U[R 5(a3UR UR;aURUR $UR5$rt) r{r+rEdatarDComputedBufferr6rget_size)r`rs rxget_allocation_size!GraphLowering.get_allocation_sizeFs dBLL ) )99D dBMM * *99D tR.. / / T777--dii8 8==? "rycUR5nURU5nURnURn[ X4U5$rt) get_layoutrrooffsetr)r`rlayoutrnrors rxget_allocation_storage_size)GraphLowering.get_allocation_storage_sizeYs?"''-.tVDDrycr[U[5(dU5eX R[U55;$rt)r{r-r0r>)r`r features rx has_featureGraphLowering.has_featurebs5 '>22;G;233OF4KLLLrycX4UR;aOSnUR5(d(U(aUR5nOUR5nX0RX4'URX4$![a N+f=f)zS Get the size hint for a dependency with caching to avoid expensive recomputation. r)r_has_unbacked_symbols numbytes_hint numel_hintKeyError)r`dep count_bytesress rxget_dep_size_hintGraphLowering.get_dep_size_hintjs  T%=%= =C //11"!//1!nn. =A>cBUR=n(aU$[S5e)NzNo current device)rDrr`r s rxget_current_device_or_throw)GraphLowering.get_current_device_or_throw~s$(( (6 (M23 3ryc#\# URnXlSv X lg!X lf=f7frt)rD)r`r rs rxset_current_device GraphLowering.set_current_devices*##$ ( "' % s,!,),cLUR(agUR(agg)N inferencebackwardforward)rrris rxget_training_phase GraphLowering.get_training_phases      ryc[R(dg[R(agURRVs/sHAo"R [ RRRRLdM?UPMC nn[U5nUS:Xag[ RRR(aE[ RRR5(a[!SU55(ag[[#URR55SU-:a[$R'S5 g[)SU55(a[$R'S5 gSS jnSS jnSS jnU(a[+[,5nUHQn [/U 5n U cMU"U 5(aS n O"U"U 5(aS n OU"U 5(aSn OSn X==U - ss'MS [$R'S5 Sn Sn SnSn[1UR355nUS U -US U--USU--USU --nUU:*nU(d[$R'SUU5 U$[)[5XS55(a[$R'S5 g[)[5Xc55(a[$R'S5 g[![5Xs55(a[$R'S5 ggs snf)zT Decide if we should enable layout optimization for this graph based on heuristics. FTrc3# UHFnSH<nURURSRR[;v M> MH g7f)rr)valN)rvrr typerarnrs rxr2GraphLowering.decide_layout_opt..sJ#A!Cs   '..337OO!P#sAAi,z*Skipped layout opt because only a few convc3# UH4nSH*n[URURS5v M, M6 g7fr)rrvrrs rxrrs?  QVVC[--e4 5 5 6s<>zeSee perf regression with dynamic shape. Follow up in https://github.com/pytorch/pytorch/issues/102670cURSRSn[U[R5(deURSS:=(a UR S5S:$)Nr)rr)rvrr{rr rn)rmeta_vals rx is_grouped3GraphLowering.decide_layout_opt..is_groupedsTvvay~~e,Hh 55 5566":>:hmmA&6&: :ryc URSRSRS5S-URSRSRS5:*=(a. URSRSRS5S:$)Nr)rrrvrrnrs rxis_in_out_channel:GraphLowering.decide_layout_opt..is_in_out_channelsvq u%**1-1QVVAY^^E5J5O5OPQ5RR6FF1INN5)..q1A5 rycURSRSRS5S:*=(a. URSRSRS5S:*$)Nr)rr@rrs rxis_small_channel9GraphLowering.decide_layout_opt..is_small_channelsTq u%**1-38FF1INN5)..q1R7 rygroupedsmallin_outdefaultzConv inputs meta not foundg|?5^?gtV?g333333?guV?zhSkipped layout opt in inference because weighted flops indicate slowdown, default: %d, channels last: %dzFSkip layout opt because found grouped convolution with >1 in_channels!zBSkip layout opt because some convolutions have smaller out_channelz>Skip layout opt because all convolution channels are too small)rrrbool)rrrr)r*layout_optimizationforce_layout_optimizationrHrrropsrrrrqbackendsmkldnnenabled is_availableallr1logdebugrrfloatr9rvaluesmap)rarr conv_nodesnconvrrr flop_countsr counted_flops node_typeGROUPED_MULTIPLIERDEFAULT_MULTIPLIERIN_OUT_MULTIPLIERSMALL_MULTIPLIER total_flopsweighted_flops do_layout_opts rxrGraphLowering.decide_layout_opts ))  + +xx~~ %!UYY^^5O5O5W5W)WA~  J A: NN ! ! ) )%%2244#  tBHHNN# $e 3 IIB C     IIw  ;     ,7,>K" .t 4  (d## )I%d++ 'I&t,, (I )I&-7&#  67 "' !&  % $ k0023KI&);;g&)99:h'*;;<i(+==>  +k9M  ~" ! & s:* + + IIX  s$1 2 2 IIT  s#0 1 1 IIV W] s >L L cBURbURSU3$U$)z2Prepend the given name with the graph name if any.rx)r6r`r6s rx qualify_nameGraphLowering.qualify_name0s& 99 ii[$( ( ryc[UUUURURURURUR UR URU5S9 $)a Make a subgraph of the current graph with all inherited parts, except the graph module (`gm`) and `example_inputs`. The subgraphs are lowered separately and lifted into a separate function in the parent output wrapper code. The subgraph name is qualified by the parent graph's name. Note that the lifting of subgraph is supported for python wrapper only. For cpp wrapper, we inline the subgraphs in the parent wrapper. ) parentrarrbr7r;r+rrr6)SubgraphLoweringrr7r;r+rrr)r`rar subgraph_names rx make_subgraphGraphLowering.make_subgraph6sZ )oo((]]#'#>#>**((""=1  rycSn[RRRR/n[ [ "5n[URRR5HnUR[RRRRLaURU5 UcUnMVURU;aMhURHnXS;dM URU5 M M URRRHKnUbXA:Xa U$XC;dMURH&nURU;aMURU5 M( MM U$)a The rule to decide if an node prefer channels last is simple. 1. if it's input/output of a convolution 2. if one of its user prefers channels last We have rule 1 because cudnn runs a faster convolution kernel for channels last inputs; Rule 2 is also important. It makes sure that indirect inputs to convolution also prefers channels last. Consider the scenario: conv -> batch-norm -> relu -> conv Without rule 2, batch-norm output may use a contiguous layout. That will cause 2 extra copies: 1. the output of batch-norm should be channels last initially since its input is a conv's output. Forcing the batch-norm's output to be contiguous results in the first copy 2. The second conv's input is initially contiguous. This layout is propagated from the batch-norm's output. We need convert it to channels last layout which results in the second copy. With rule 2, we makes sure all the tensors in the chain uses channels last layout. So both copies can be saved. N)rrrbmmrr'r$rrPrHrrrrusers)r` last_convnodes_cannot_propagate output_setrusers rxrE-GraphLowering.find_nodes_prefer_channels_lastQs)& "')).."4"4"<">55===q!$ !Ixx11%NN1% 36""((A$ GGD{{&<< NN4($ )rycXR;a2URRU5 [RSU5 gg)NzUsing FallbackKernel: %s)rGr perf_hint_loginfors rx warn_fallbackGraphLowering.warn_fallbacks8 ,, ,  ! ! % %d +   94 @ -ryc`URRUR5 URb%URRUR5 [ R R(a8XR;a([ R RURU'gggrt) rrrindexrrdrHr,rMrs rxadd_device_infoGraphLowering.add_device_infosw fkk* << #     . 77  F2J2J$J/0ww/C/CD $ $V ,%K ryc"[R$rt)rd fake_moderis rxrGraphLowering.fake_modes {{ryc XR;aURU$XR;aURU$XR;ay[RRUn[ R "U[ R"URUR/[RRU5Q76S9$gNr6r) r0r rrdrHr+ConstantBufferr=r dtyper~)r` buffer_namers rxtry_get_bufferGraphLowering.try_get_buffers -- -&&{3 3 ++ +$$[1 1 .. (77$$[1D$$ ~~KK./gg.J.J4.P ryc[S5e)Nz'Should not be called for the main graph)r)r`symbols rxadd_symbol_graph_input$GraphLowering.add_symbol_graph_inputsDEErycJURU5nUbU$[SU35e)Nz$Failed to find buffer matching name )rrr`rbufs rx get_bufferGraphLowering.get_buffers1!!+. ?JA+OPPrycXR;aURUR$[URS5(aXRR;aqURRUnX R ;aUR UR 5$X R;aURUR 5$XR ;aUR UR 5$XR;aURUR 5$[R"SU5nU(a UR URS55$[SU35e)Nmutation_real_namez1(as_strided|reinterpret_tensor)\(([a-zA-Z0-9_]+),r)could not find ) rrrr@r r0 get_dtyper rematchgroupr)r`r mutated_bufrs rxrGraphLowering.get_dtypes. .. (>>+.44 4 DNN$8 9 9~~@@@..;;KHK111**;7AACC///((5??AA -- -&&{3==? ? ++ +$$[1;;= = HHI; W >>!''!*- - 677rycVXR;aURUR5$XR;a5URUnUR5(dgUR 5$XR ;aUR UR 5$[ SU35e)Nr)r)rnumelr0has_tensor_output get_numelr rrs rxrGraphLowering.get_numels .. (>>+.446 6 -- -%%k2C((**==? " ++ +$$[1;;= = 677rycj>[S5 [TU]"U6sSSS5 $!,(df  g=f)NzGraphLowering.run)rrrun)r`rvres rxrGraphLowering.runs$ - .7;%/ . .s $ 2c"URb SU35e[U[R5(deUR S[ UR 535nUR RU5 XRU'X!lU$)NzOperation registered twice: r) operation_namer{r+ Operationrrqrrr3)r`rr6s rxregister_operation GraphLowering.register_operations  (M,H*MM("bll++++  2c$//&:%;!<= r" "  ryset_namecURS[UR535nURRU5 XRU'UR 5nUb_[ U[R5(a/UR5(aU[R"S5:XdURU5 U(aX1l U$)Nr r)rrqrrr0 get_devicer{r+ris_zero_elementsrr rr6)r`bufferr#r6r s rxregister_bufferGraphLowering.register_buffers  3s4<<'8&9!:; F#$*D!""$  62#4#455++--ell511   ( K rychURSSRU5-5nXRU'U$)Nlist_rx)rrr-)r`operation_namesr6s rxregister_operation_list%GraphLowering.register_operation_lists1  388O+D!DE* 4 ryc*^^SUU4SjjmT"U5 g)Nc>[U[[45(aUH nT"U5 M [U[R5(a6UR 5H!nTR URU5 M# ggrt)r{r1tupler+rEget_read_namesr2r)valuex read_nameregisterr`s rxr61GraphLowering.register_users_of..registersg%$//AQK%..!&!5!5!7I&&y188?"8/ry)r3%Union[Iterable[ir.IRNode], ir.IRNode]rNoneru)r` node_outputr6s` @rxregister_users_ofGraphLowering.register_users_ofs @ @ ryc[U[5(deURRU5 XR;agURUHnUR 5 M g)zz When a buffer is mutated we need to make sure all the reads to the old version are realized before the mutation happens. N)r{strr$rr2realize)r`r6rs rxmark_buffer_mutated!GraphLowering.mark_buffer_mutatedsX $$$$$   & )) ) &&t,D LLN-rycXR;aXR;d SU-5e[URU5nX RR;aURRU$URU$)z In AOTI, module buffers may have been mutated during the tracing and compilation. Thus we need to read from previously stored original buffers, to make sure the generated model.so uses correct initial values. z$Can not find the original value for )rQrrXrPr)r`r6 orig_names rxget_original_value_of_constant,GraphLowering.get_original_value_of_constants 3338N 2T 9 N5T5Q5QRV5WX KK,,, KK  Y ' % ryc [RR(d7URR 5Hup4[ X$5(dMUs $ UcS[ UR53nUnUSR5(aSU3nURU5n[U5nUnSnXR;aUSU3nUS- nXR;aMX RU'UR<SUR<S[UR55<S[UR55<S[U5S3 UR U'XPR"U'U$)Nconstantr constant_rxr) r4)r* aot_inductoruse_runtime_constant_foldingritemsr\rqisdigitrr^r rr1rnrohashr rQ)r`r6r constant_namer3rCprefixcnts rxallocate_non_dup_const_name)GraphLowering.allocate_non_dup_const_name.sO""??(,(<(<(>$ !$..(()? <c$..123D 7??  tf%D  & %nn$XQse$D 1HCnn$ $t{{oQtzznATYY[!$AeDKKM&:%=QDz!n  D! .7$$T* ryc URX!5n[R"[R"U[ UR UR/URU5Q76S95$r) rRrEcreater+rr=r rr~)r`rr6new_names rxadd_tensor_constant!GraphLowering.add_tensor_constantMs`33D?   "KK.2.G.G.M   ryc^URTRU:XdUcT$[RRR 5 UR TSURUR=(d S3URTRU55nX0R;d US35e[U4SjUR55(aURUURU'[U4SjUR55(aURUURU'UsSSS5 $!,(df  g=f)z We AOT copy constants to the devices they are needed on. If device_override doesn't match the constant's device, then copy it and return a different name. Nrxrz' should be in V.graph.constants alreadyc3@># UHnT[U5:Hv M g7frtr^)rrr6s rxr.GraphLowering.constant_name..ps"#5K{33#5c3@># UHnT[U5:Hv M g7frtr[)r param_namer6s rxrr\xs""7Jz22"7r]) rr rutils_python_dispatch_disable_current_modesrRrrtorrr)r`r6device_overridenon_dup_const_names ` rxrOGraphLowering.constant_nameZsE >>$  & &/ 9_=TK [[ ) ) @ @ B"&!A!A&/../0E0E0J/KLt$''8"  &7 %&&MN 7 #'#5#5:>&:""#56"&"7"7=ANN&=%%&89&=C B Bs C-E Ec  >U=RS- sl[T U] XU5nURU5n[ U[ 5(av[ RR(d[UR5nOURRnXPRU'URRU5 U$[ U[[ ["45(aA[$R&"U5nXPRU'URRU5 U$[ U[(5(a4[+XS9nX`RU'URRU5 U$UcURRU5 g[ U[,5(aURRU5 g[ U[.R05(Ga[3[ RR4R65S:Xa[9[;[ RR4R655R<[.R>R@RB[.RDRFRH4;de[JRL"XRNS9nXpRU'URRU5 U$[ U[.RP5(dU5eURR(dURUU5upOURWU5upUR(aiURX(aXURURX;a>[ZR\"[_U[aURNURbX5S95n O=[ZR\"[eU[aURNURbX5S95n XRU'URRU5 U RfRfURhU'UR4R6(aURkURN5 [m5 [oU5(dURpRsU5 SSS5 U $!,(df  U $=f)Nr)r6r3)r6r r):r]r placeholderrr{r#rdrHrrrrur r rintrrr|sympifyrrFrr Generatorrqr,rr=iterr_prims rng_primsgraphsafe_run_with_rng_stater higher_orderinvoke_subgraphr+GeneratorStater r _has_symbolic_sizes_stridesr~r{r^rErUr<r=rr@rr rr]r_rUr) r`rrvrwexamplerurgensizesstridestensorres rxriGraphLowering.placeholders !'%fF;""6* gx ( (77&&,W\\:||(((,  f %  " " ) )& 1K #tU!3 4 4==)D(,  f %  " " ) )& 1K !1 2 2!v=C(+  f %  " " ) )& 1J _  " " ) )& 1 g} - -  " " ) )& 1  1 1qww++112a7DQWW))//0=f &&CC &&66=  ##GC(+  f %  " " ) )& 1J'5<<009'90 22!66w?NE7!88ANE   $$$$(<(<<%%&w~~w}}eUF%%&w~~w}}eUF%+&! %%f--3[[-=-=""6*    " "   01 2.w77&&**623 3 2 s ,R:: S c>^T[RLa3[US[[[ 45(a[ TU]TX#5$[T[RR5(d[TS5(aT"U0UD6$T[;Ga_[T[RR5(d TS35eTR5RS5SnU[ ;a [#TSSS9 O[$R&(a[)T/5(a[*O[,n[.R1SUR3TX#55 [5TSS 9nUcF[R6R8R;T5(aUR<(a[>nO[5TSS 9n[AU5n[#TUS 9 O)[)T/5(a [+TX#5e[-TX#5e[.RCS [T5 URDn [GT5n U (aX#pU [HLaS U RJ;apU RJS up[T[RR5(deSU4S jjnU"X5upU"X#5up#U"X5up[IX#X5up#OU "U /UQ70UD6up#SU RJ;a[MTSS9"U0UD6nO[T"U0UD6nU (aUROU W W X#5 U$![Pa,n[SUTX#5RUURV5SeSnAff=f)Nr_inductor_lowering_functionz is not an OpOverloadrFT)warnoverride_decompz"Creating implicit fallback for: %s) with_default)layout_constraintz via %seager_input_valscv>[RRRTX5nUceUSUS4$)Nrr))rroperator_schemasnormalize_function)rvrwrrs rx normalize.GraphLowering.call_function..normalize?sC%*XX%>%>%Q%Q &&F$*#55#5#)!9fQi#77ryshould_fallbackadd_to_fallback_set)rvrrwrrztuple[Any, Any]),operatorgetitemr{r1r1dictrrrrOpOverloadPacketrrLrr6rrIrMr*implicit_fallbacksrr7r8rr operator_strr_libraryr` is_builtinrrPrQrr,rNrGrrJpropagate_mutation Exceptionr6with_traceback __traceback__)r`rrvrw base_nameerrortagdecided_constraint default_tagrlayout_constraintsold_args old_kwargs fake_args fake_kwargsrouteres ` rxrGraphLowering.call_functions* X%% %*T!WtUD>Q*R*R7(> >&%**"="=>>7 1D D 4*6* *  "fejj&;&;<< (/0 < ++C03I//f5$G***6(33.5 9&&vt< /H/K,,77??((+'1JT1K*B+)N&f8JK#VH--0EE264HH5  IIj)F"3 4!!A!9&!A !'+*%)BB *QVV3128J1K.  *&%**2G2GHHHH82;91R. '0'> /8/N,'@ )( $6a#I$#I&#ILD AFF*&v5I# '88!''8ZNJ #Avt<KK  s6D!L M"'M  Mcd[UR5S:H=(a URSS:*$)z= True if this is a small constant attr that will be inlined. r)r)rqshape)ts rxcan_inline_constant!GraphLowering.can_inline_constantcs( 177|q 4QWWQZ1_4ryc[URU5n[U[RR 5(aBXR ;aUR U$[R"XS9nXPR U'U$[U[RR5(a&X@RU'SURU'[XS9$[U[5(a&X@RU'SURU'[XS9$[U[R5(de[ R"R$(d4[ R&(d[)U5(dXR*;aUR-XA5$[/5 UR0S:Xa6[3UR55UR6UR8S9sSSS5 $UR;U5(aX[<R?S[AU55 SSK!J"n U"URG5UR6UR8S 9sSSS5 $SSS5 UR-XA5$!,(df  N=f) N)r6 graph_modulerrhru)r3rr zInlining constant: %s r))ry)rr )$rrPr{rrrkrr+Subgraphr ScriptObjectrr rFrr r*rJrKalways_keep_tensor_constantsrRrrWr&rr;itemrr rrrr>loweringrytolist)r`rrvrwr3rrys rxget_attrGraphLowering.get_attrks"$++v6 eUXX11 2 2,,,**622++6>C*-   'J eUXX22 3 3/4 $ $V ,*,D   '"< < / 0 0/4 $ $V ,*,D   '"< <%....    < <22(//333++E: : ]{{b **,ekk%,,] ''.. 2CK@,ellnEKK U] / ''66]s=I*"A$I** I8c[ertAssertionErrorr`rrvrws rx call_moduleGraphLowering.call_moduleryc[ertrrs rx call_methodGraphLowering.call_methodrryc >[TU]XU5n[U[[45(dU4n[U[[45(d[ U55e[ SU55(dU5e[RRRSn[U[[45(dU4nUVs/sH"n[RRU5PM$ nn/n[U5[U5:Xde[XE5GH\up[U[R [R"45(dUR%U5 MH[UR'5[R(5(a0UR%[RR+U55 M[,R.RR1U5(deU R2SR55V s/sH:n [U [,R65(aU R8R:OU PM< n n UR%[R<"X55 GM_ XplUR@RC5GHup[U [D5(aM[U [ [FRH[,R.RRJ45(dS[ U 535e[U [ 5(dMU RM5 [U [ 5(deU RNn [U [RP5(deU nU RNn [U [R5(aU RU5U :wdGM[RVRYXRZU 5 UR>R]U5nURZU UR>U'GM URa5 [bReSURfURhbURh5 gS5 gs snfs sn f![^a GMf=f)Nc 3># UHn[U[[R[ S5[R [ R[ RRR[[R[R4 5v M g7frt)r{rEr+r;rrr|r logicboolalgBooleanrjEffectfulKernelrC)rr4s rxr'GraphLowering.output..sv  KKJ%%JJKK''//&&,,   sBBrrz'Unsupported inductor graph input type: zGForce channels last inputs for %d conv for the current graph with id %dr)5rrr{r1r1rrrdrHr,rvr+ ExternKernel realize_inputrqziprEBaseViewrget_output_specCommBufferLayout copy_inputr _inductoris_storage_and_layoutrrortrrutry_match_insignificant_stridesrr rLrFr|r rsr?rrDr@get_nameMutationLayoutSHOULDREMOVE realize_intor r ValueErrorfinalizerrrr<)r`rrvrwr fx_node_argsr4result_correct_stridesrfx_nodes meta_stridesr6r3value_storage_boxindres rxrGraphLowering.outputs f5&5$-00YF&5$-00>$v,>0      !  $ww++003 , 66(?L#>AFFKKAE9 '--66qG#4*4,,224KD%11 5::u/A/A/P/PQ G8e F GeY// MMOeY// //JJEeR]]33 33 % JJEe[11U^^5E5M--::55d;,,223DEC.2.H.H.ND&&s+-54  U  ' '!]]6DMM =? mD" B"s)Q+AQ07Q55 RRcJURHnUR5 M grt)r decide_layout)r`r s rxrGraphLowering.finalizes<XLagURbURR(a[U[R5(aU4nU4n[ X5HKup4X4LaM TR [RRRRX4405 MM gggrt) alias_infois_writer{r+IRNoderrrrrcopy_r) schema_argold_argnew_arg old_arg_item new_arg_itemr`s rxmaybe_propagate9GraphLowering.propagate_mutation..maybe_propagateBs!$$0Z5J5J5S5Sgryy11&jG&jG25g2G.L#3 && ,,44|6RTV3H6T0ry)rztorch._C.Argumentr ir.IRNoderrrr9)rqrrrrqtriton_kernel_wrapper_mutationrwr{rrrget_mutated_tensorsrLrr$rrrrrrr_schemarr argumentsr6)r`rrrnew_args new_kwargsrwrcrdmutatedr6rrrschemarrarg schema_kwargskeys` rxr GraphLowering.propagate_mutationsA 8}H ---:#j/111 >>UYY33RR R^^H-Ffd++ ++--@@TT<(./!'  . 1ehhmm(D(Dqvve}!K .45G $X.t4$X.t4%""599>>#7#7#?#?'ASUWX   '..%***?*?@@@@ ) 4= HQ   ('''0X1H'I #C#'))#.J J 9(J392B2BC2B332B CC oG oG&s+J J 9 YTDs AI I!c ">^^^$^%^&S%U4SjjnSSKJn [TR5m%[TR5m&[ T/5nTR S:HnU(a!TRT5upgU[Xg5-n[RRU5 TRT5 [R"T5 TR S:XaTR(a[TR[ R"R$5(a[ R&R(R+TR5(aP[-T5(dUR/SSU4Sj5(a$U"S5 [1TRS S 9"W0WD6nGOXTR S:Xa[TR[ R"R$[ R"R245(a4[5T5(a$U"S5 [1TRS S 9"W0WD6nGOTR S:XaTR[ R6R8R:La[<R>S :waU"S 5 [<R>S :XaWn Wn TR@RCS5=n (aU Sn U Sn [EUUU U 5upgO[GT/UQ70UD6upgTRITRXg5nTRKTXXg5 O[MS[<R>35e[OTR5(aU"S5 [TR@S[ RP[ RR[ RT45(a$TR@SRVRXnO'[ZT'T]T5nOU"S5 [ZT'T]T5n[ R6R^R`Rb[ R6R^RdRb[ R6R^RfRb[ R6R^RhRb[ R6R^RjRb/m$[mSTRn55nTTRp;n[mU$4SjTRn55nTR@RCSS 5(a[U[r5(aURu5 TR@SRw5n[ RxR(Rz"U6nUR}5U:wa=U(d6[R~"U5n[RRUU5nU(aN[U[r5(a9[UR[R5(aURu5 U(dU(Ga[TR@S[ R5(GauU(aTRpRCT5nOTR@SRw5nUGb1[U5S:Ga![<R=(d U(+=(a U(+n[ RRTR@S5n[[U55S:nU(d~U(aw[UR55S:XaZTTR;aJU(dCU(d<[RRUR5[ R5nU(Gd%[U5(GaTR@SR5(d)[UR[R5(a4[RRU[R~"U5US9nO[UR55S:Xa[U5S:a/nOGUVs/sH:n[U[ RP5(aURVRXOUPM< nn[RRUUUS9n[[ TRn55nUS:Ga:[U[r5(Ga$TRnGHnUR[;GaUR5 [ R6R^RRb[ R6R^RRb[ R6R^RRb/n/nTR(d=UR[ R6R^RRb5 [ RR(GaU[ R6RRRb[ R6RRR[ R6R^RRb[ R6RRRb[ R6RRR[ R6RRR[ R6RRR/- nU[ R6RRRb[ R6RRR[ R6RRR[ R6RRRb[ R6RRRb[ R6RRR/- n[ RR(a2U[ R6RRRb/- nURU;aN[RRU[R~"TR@SRw55SS9nURU;afTURSLaT[RRU[R~"[TR@SR555nUR S:XdGM[URR[[45(dGMURu5 GM UR[TRn55 [U[r5(a%UR5(aUR5 [U[r5(an[UR[5(aOURRn[U[5(a$URSS9(aURu5 SSS5 SSS5 SSS5 [WT5 TRU5 [ [R"5nTRT%SHnUUR5-nM TRT&SHnUUR5-nM [RRRn TR S:XaF[U[R5(a'U RU5(aURU5 S&U%U&U4Sjjn![RR(aTR S:XaU$[[RRRTR@RCS055n"U"ce[ S U"55n#UU#:d&S!US"U#S#TR5S$U!"535eTGRTU5 U$s snf!,(df  GN=f!,(df  GN=f!,(df  GN=f)'NcZ>[RS[TR5U5 g)Nzlowering %s %s)rrr format_node)msgrs rxr%GraphLowering.run_node..debugds II& 1==(A3 Gryr)CompilerBisectorrinductorrLc>[T5$rt)reprrsrx(GraphLowering.run_node..s aryrJFrflexible_layout-user_defined_triton_kernel_layout_constraintsneeds_fixed_stride_orderrr)z1Unknown triton_kernel_default_layout_constraint: rrrc3># UHoRS:Hv M g7f)rNr)rrs rxr)GraphLowering.run_node..sDGDGGx/Gsc3@># UHoRT;v M g7frt)r)rras_strided_opss rxrrs*:A$ ~-'r]inductor_realize_to_strides) allow_paddingTrd) thresholdric>TRTSVs/sHnSUR5SUS3PM nnURSTRTS55 SR U5$s snf)Nunbacked_symbol_defs= in:  c3P# UHnSUR5SUS3v M g7f)rrrN)get_unbacked_symbol_defs)rrs rxrBGraphLowering.run_node..format_new_defs..s1?B((C(C(E'FfRDPRS?s$&z*** )rr extendrr)r rbuffer_watermarkoperation_watermarkr`s rxformat_new_defs/GraphLowering.run_node..format_new_defss <<(8(9::C((D(D(F'GvcURTU:  HH//*=*>? <<? "s!A,unbacked_bindingsc3# UH8n[RRRR X5v M: g7frt)rdrrbunbacked_renamingsr)rrs rxrrs5/ & KK ! ! 4 4 8 8 > >&sAAzfailed  >= z (inductor >= fx) fx node is: z new operations are: )rr>rr9rr>)!torch._inductor.compiler_bisectorr rqrrr'rfetch_args_kwargs_from_envrWr+rcurrent_originsrrdrr{rrrrr`rrKdisable_subsystemrJHigherOrderOperatorr`rrqrr*'triton_kernel_default_layout_constraintrrrGrHrrrrrtSymFloatSymBoolrrurrun_noder as_stridedr as_strided_as_strided_scatterresize resize_asrrrrEr?rorany_is_symbolicmaybe_get_strideget_stride_orderrrequire_stride_orderrrr r _prims_common%is_non_overlapping_and_dense_or_falserrrFFlexibleLayout stride_ordered_for_memory_format channels_last_is_viewrequire_exact_stridesrO realize_hintrmm_int_mmrrrr _has_mkldnnr_linear_pointwisebinarymkldnn_rnn_layeronednnqlinear_pointwisery binary_tensor_convolution_pointwise_convolution_pointwise_ _convolution_transpose_pointwiseqconv_pointwiseqconv2d_pointwisehas_mklmkl _mkl_linearrvrrrArB mark_reusehas_exceeded_max_readsrDhas_large_inner_fnr:r;r|r}r rHr rbis_unbacked_symintrrrrcreate_deferred_runtime_asserts)(r`rrr originsis_call_functionrvrwrrrrinp_args inp_kwargs is_outputis_user_visibleis_input_for_as_stridedrx sym_strides stride_orderrdenseunbacked_symbols_in_stridesr num_usersrneed_fixed_layoutneed_fixed_channels_last_layoutcurrnew_unbacked_defsr rrbr%r'renamed_unbacked_bindingsrr#r$res(`` @@@rxr4GraphLowering.run_nodecs  H Gt||,!$//2$.qc?44?2 ::1=LD ~d3 3G II % %g .  ! !! $  q !'HHqxx)>)>??NN((33AHH==9!<<'99"K ())!((N 'HHuzz44ejj6T6TU/q11())!((N 'HH 6 6 U UUBBFWWEFBB12 $H!'J+,66::6H+II'I#3A#6%5a%8 '@ "$& ( f(?q'R4'R6'R !//$GF++AxTR&KFLzLzK{|!**'(FF5MELL%..%--#PVVE]//44F"W-a0Fb )!, ))11 **22 1199 %%-- ((00 NDAGGDDI4#C#CCO&)*:;''*' #vvzz7??J EE &&-..0#oo33CCWM **,7 #%#6#6w#?L__AA&,WFvy11v{{BKK88 4*u u||;;#">>BB1EGffUm224G&3wa3N3NAFFKKTU$U-4!(+"&(__%J%J &}&K&F Jqww/0I1}FI!>!>GGD{{&;;++-"IINN??GG!IINN--55!IINN22::-) ;=7#-44UYY^^5O5O5W5WX 88///- % 0 0 B B J J % 0 0 B B I I %  ? ? G G % 0 0 B B J J % 0 0 B B I I % 0 0 B B I I % 0 0 B B P P2-< % 0 0 G G O O % 0 0 G G N N % 0 0 H H O O % 0 0 Q Q Y Y % 0 0 @ @ H H % 0 0 B B I I @; %xx// 1eiimm6O6O6W6W5X X 1;;*;;%'__%I%I & " 3 3AFF5M4H4H4J K.2&J&F !KK+JJ !TYYq\ 1%'__%I%I & " 3 3$B166%=CVCV$W!"&F ww(*%fkk&6&6I8NOO"NN,y$~!!#agg,/&),,1N1N1P1P##%&),,FKK1T1T{{''dI.....=(C " % /J 61% v&&u||46<< 0 12C !=!=!? ? 3//"5"67B ! > 8GG$$.. DDM !65<<00,,V44  ! !& ) # # 77  144=#8M.6 GG   & & 3F(K !,,,%// &/ % ! !$== '(-F,GH==?+,&&5&7%8 : = ,,Q0AB e+"M " ! % $ / .ss!AE 3AD8 _!AD&+AAD! ,TAD&-AD&3C6AD&)AD81AE D!AD&D& AD5 D0AD8D8 AE EAE E AEcb^S U4Sjjn[5(a|UR[RRR R LaATR(a0TRU5upEUSS:waU"USUSS35 ggT=RU-sl [RRRnUGH[nTRRU/5nUR Un UR#5R%U 5(d{S Sjn U "U R&5(a$U"XyR&:USU R&35 U "U R(5(a$U"XyR(:*USU R(35 UHn [+U R,5n U TR- n U (a;[/U [0S9nTRR3U/5R5U 5 MiU"U R,U R,5 M GM^ g) Nct>[R"X5nTRUSS9 TRU5 g)NTr")r+ AssertScalarr(r )rur assert_opr`s rx make_assertBGraphLowering.create_deferred_runtime_asserts..make_asserts22I  T :  # #I .ryrTz to be Truec`U[[*4;ag[U5 g![a gf=f)NFT)r(rj TypeError)rs rxis_convertibleEGraphLowering.create_deferred_runtime_asserts..is_convertible s5& 11#()F#'()#()s --r*z <= )r)rur"rr>rr9)rr rr)rrrrr_assert_scalarrr;r-r rdrHr rbrr var_to_range _default_unspecified_value_rangeissubsetlowerupperrrurr>rr)r`rrkrr node_argsrxrbi0rasvrrvrafvsmissingi1s` rxr[-GraphLowering.create_deferred_runtime_assertssD / % & &EIINN99AAA ::1=LI|t#IaLYq\N+*FG$  ' '+< < '((22I(((,,R4++B/ AACLLRPP)&bhh//#B((Nrd$rxxj4IJ%bhh//#B((Nrd$rxxj4IJB/8C!D$?$??G c2**55b"=DDRH#BGGy:)(ryc[R(a [S5e[RS;a[S[R35eg)NzC++ codegen is disabled)linuxdarwinwin32zUnsupported platform )r*disable_cpp_codegenr5sysplatformris rx!validate_can_generate_cpp_wrapper/GraphLowering.validate_can_generate_cpp_wrapper$s@  % %()BC C <<; ;(+@)OP P UR2'M US[!U 5Ul Xl!gs snnf!,(df  GN=f) Nrrgridrwrrrr)rvrFT)"rdeepcopyrOrHrrrrrrqrrrwr{rr$rqrrrrLrinserting_beforercloner6r1rv recompile InterpreterrrCrArB)r`rr cloned_gm triton_nodesr grid_inputs visited_grids triton_inputs kwargs_inputsvisited_kwargsrrrwrcrdrrnew_node new_outputsrunnerreturned_outputs grid_outputs dynamic_grid new_gridsnew_grids rxextract_autotune_inputs%GraphLowering.extract_autotune_inputsPs MM$,,/ ~6 OO))D?*KK599#9#9#X#XX##D) *,. 24 (* -/ )+ D F+C+ !#uxx}}55-0-= *#**3/  ,[[*F--@@TT L) /0!'  . 1ehhmm(D(Dqvve}!K . 56G*,J11$7"LLNDAG|#,??#@#@TUSW#@#X(+M(: 1 %,,X6 *(6(9 1  (+M(:N%!((+$2$5JqM+8(2M)) $M!P$1 OO))Dww("";/1 * %%i0!::n5 { a +C ,>,@AL$&D !$$ .0  KK/D!H#)#uxx}}==$OOC0$'+   ]5G(HI $$$U8_50 <7@D))$))4%""22FC 4F!G"/o87sAP8BP>> Q c^[U4SjS55(GaSU4Sjjn[RR(a[RR(aWSnTR H&n[ U[R5(dM$Sn O U(aU"5nTRU5 TR5$STl TR5RnU"5n[RR R#5 U"U5 SSS5 ASTl TR$R'5 TR(R'5 TR*R'5 [,R.R0R2R'5 [,R.R0R4R'5 [6R8"5 [R:"SS05 TR5sSSS5 $TR5$!,(df  GN =f!,(df  g=f) zA For GPU, Triton kernels are autotuned and stored as cubin files c3@># UHoTR;v M g7frt)r)rr r`s rxr9GraphLowering.codegen_with_cpp_wrapper..sIv***r])cudaxpuc>SSjn[RRR5nUb[ [ R [5(dUR(aURR5 URVs/sH nUcMUPM nn[R"U[ R 5Vs/sH nU"U5PM nnOU[ [ R [5(a T RO[ R Vs/sH nU"U5PM nnT R(aSSKJn [#T R$5VVs/sH<upxUT R;dM[ XW[R&5(dM:UPM> n nnU H3nXWn [ U [R&5(deU"U 5XW'A M5 U$s snfs snfs snfs snnf)NcNUcg[U[R[R45(aURR $[U[ 5(a [U5$[U[R5(dS[[U55-5eU$)Nz&Unknown type when creating real inputs) r{rrtr2rhintrrr r>r)r4s rx materializeXGraphLowering.codegen_with_cpp_wrapper..extract_real_inputs..materializesy##A enn'EFF vv{{*#Az22%ay()!U\\::Ds4PQ7|S: !ryr))clone_preserve_strides)r4z1Union[torch.SymInt, torch.SymFloat, torch.Tensor]rzUnion[int, float, torch.Tensor])r_guardsTracingContexttry_getr{rd real_inputsrcoutput_stridesclear params_flatrZchainrr. compile_fxrrr r ) rtracing_contextparamrr4rrrr6r/ mutated_inpr`s rxextract_real_inputsCGraphLowering.codegen_with_cpp_wrapper..extract_real_inputss!H!4!"#(--">">"F"F"H".zMM;88'55'66<<>&5%@%@#%@E %@ #"+amm!L#!LA$A!L #K *!--EE!//!"/#/$A/ #&&B*343D3D)E*)EIC4#6#66'{'7F)E'* 2'2&6 )+u||DDDD+A++N (' 2#"W# ##*s*G&G& G+"G0%G5?G5"G5FTNztriton.autotune_at_compile_time)r%list[Union[int, float, torch.Tensor]])rr*tritonautotune_at_compile_timeautotune_with_sample_inputsrr{r+UserDefinedTritonKernelrcodegenr7compile_to_modulecallrr`rarbr"rr!r&rdrHr precomputed_replacementsinv_precomputed_replacementsr,resetpatch)r`ruser_defined_kernelsrrcompileds` rxcodegen_with_cpp_wrapper&GraphLowering.codegen_with_cpp_wrappers II I ID #L}}55==<<+0("oo%b"*D*DEE370!.,&9&; 44[A||~%$) 1138813 [[11HHJ[)K$( $$**,''--/''--/  99??A  ==CCE \\#De"LM<<>NM<<> !!KJNMs H<I< I  IcSSKJn [R"SS5 U"UR5UlSSS5 g!,(df  g=f)z (Re)initializes the scheduler member. When initializing the scheduler, no CUBIN files should be generated (to avoid biasing any benchmarks and pessimizing fusion decisions). r)) Schedulerztriton.store_cubinFN)r@rr*rr)r`rs rx_update_schedulerGraphLowering._update_scheduler" s2 ) \\. 6&t7DN7 6 6s ? A cP[SSS9 UR5 UR5 [RR UR URR5 URRU5 URR5 [R S[RR5 URRUR 5nURR#5 UsSSS5 $!,(df  g=f)NzGraphLowering.codegenTlog_pt2_compile_eventzFFinished codegen for all nodes. The list of kernel names available: %s)rrrrdrdraw_orig_fx_graphrOr@rr(push_codegened_graphrrrHrYgeneraterpop_codegened_graph)r`rs rxrGraphLowering.codegen- s 1 N  " " $  " " $ GG & &t||T^^5I5I J    2 24 8 NN " " $ IIX00  &&//0A0ABF    1 1 3!O N Ns DD D%cl[SSS9 URUlURUlURUlURUlUR UlUR UlUR5 URR5 SSS5 g!,(df  g=f)a This is a more compact version of the `codegen()` above where we codegen this graph as a subgraph of some parent graph. The parent graph is passed as an argument: the intention is to inline codegening of the subgraph in the parent graph's wrapper code (including the generated kernels). The wrapper code is not finalized (via `.generate()` call), as this will be done in the parent graph's `codegen()`. zGraphLowering.codegen_subgraphTrN) rr(r'r7rrrrr@r)r` parent_graphs rxcodegen_subgraphGraphLowering.codegen_subgraph@ s:RV W , 9 9D *55DO+77D  , 9 9D +77D +77D   " " $ NN " " $X W Ws BB%% B3cSn/n/nURRHLnUR5nX- nURXES-45 URXDR 545 MN XU4$)Nrr)r@rget_read_write_buffers_sizesrget_estimated_runtime)r` total_bytes node_counts node_runtimesr num_bytess rxrGraphLowering.count_bytesU sy    NN((D99;I  $K   1n5 6  $(B(B(D!E F ) 66ryzOptional[Callable[[str], None]]save_output_codecp[SSSSS9 UR5sSSS5 $!,(df  g=f)NzGraphLowering.compile_to_modulecode_genT,inductor_code_gen_cumulative_compile_time_us) phase_namerdynamo_compile_column_us)r_compile_to_moduleris rxrGraphLowering.compile_to_moduleh s4  -!"&%S  **,    s' 5cjUR(aUR5OUR5up[U[5(aUR U5nO/[U[ 5(aUnO[S[U535eURce[UR5 [RSUR5 [R"SUR5 [R (a%[#SUR3[$R&S9 [U[ 5(au[(RR+UR5 [(RR-[.R0R3UR5SS-5 U$)Nz Unrecognized wrapper code type: Output code written to: %szCompiled module path: )filer.debug)r7rrr{rb_compile_to_module_linesr/NotImplementedErrorr__file__rrrrrprr*benchmark_kernelprintrstderrrd output_coderospathsplitext)r`r(rxmods rxr GraphLowering._compile_to_moduleq s604/?/?D ) ) +T\\^   l$4 5 5// =C  &; < <C%24 3E2FG  ||''' % . =93<<H  " " *3<<.9 K l$9 : : GG   - GGLL))#,,7:XE F ryc^^ SSKJn [RR(awUR R R5nURSS5nSU-UR RR5-S-nUTR-Tl [Rb[RTR5 [R"STR5 [R "5n["R$"UTRS9 TR&VVs/sHupgXgR(4PM nnnUR+TR5un m [R"S T 5 [,RR/T 5 [,RR1[2R4R7T 5S S -5 [9S U 4S jU4SjS9 [=SSS9 UR?U T U0UR@EURBEURDES9n SSS5 Xl#T Ul$Xl%[RL(a%[RN(aW RQSSS9 W $s snnf![:a [9S U4SjS9 ef=f!,(df  N=f)Nr)) PyCodeCachez"""z\"\"\"z&r""" Compile-time auto-tuning block: z""" zOutput code: %s)coderrrinductor_output_codecH>T[RRT5S.$)N)filename file_path)rr abspath)r srxr 8GraphLowering._compile_to_module_lines.. s $!#!6ryc>TR$rtr3rsrxr r <#5#5ry) payload_fnc>TR$rtrrsrxr r rryzPyCodeCache.load_by_key_pathTr)linemapattrs)timesrepeat)) codecacherr*rrr(kernel_autotune_defsgetvaluereplacekernel_autotune_callsr3rrrprrSinductor_meta_from_configrT begin_compileline_map stack_tracewriterdrrrr r rrrload_by_key_pathrrrrIrJrKbenchmark_harnessprofile_bandwidth_outputbenchmark_compiled_module) r`r(rr  tuning_code inductor_metaline_norrrr r s ` @rxr&GraphLowering._compile_to_module_lines sx + == 1 1#'#4#4#I#I#R#R#T #7#?#?{#S 7&'##99BBDE  "-|/A/A!AL   ) ) 5  * *<+=+= >1<3E3EF&@@B **=|?Q?QR &2%:%:%:MG**+%: $)),*<*<=IC  ! !"> E GG   % GGLL))$/2X= > &6  8PT U..nn..44 / CV$  # #(G(G  ) )! ) < Y  &5     V Us+%J4J BJ7J4JJ14 Kc/n[R"S5n[R"S5nUHn[U[R5(a+UR UR S[U535 MM[U[R5(a+UR UR S[U535 MUR UR55 M U$)Nr_none_shape) rZr[r{r+NoneAsConstantBufferrr6r=rCr)r`rnames shape_counter none_counterrs rx_get_output_namesGraphLowering._get_output_names s!*  q) !D$ 7 788  {%\0B/CDED"":":;;  {&m1D0EFG T]]_- " ryc8URUR5$rt)r8rris rxget_output_namesGraphLowering.get_output_names s%%d&8&899ryc:XR;=(ar URUR5S:H=(aK [URUR55S:H=(a [ URU5S:H=(d XR ;$)Nr)rr)r rrqrr>rrs rx is_unspec_argGraphLowering.is_unspec_arg s %% % B!!$'113q8 BD%%d+44671< B  1 1$ 78EA 3 22 2  3ry)\rrGrrrYrQr;rCrArBr rrr^rIrKrJrrrrr rr7r5rDr,r_rrMr'rrrLrrTrr+rrr8r0r<r r r rr&rrXrrrrr-rWr:r$r/r.rr6r0r3r2rrr%rVrFrrrrOrr]r?rr9r"r#r!rr@rr rrUrr\r(r)NNNFFNNFFFNNNNNNF)&ratorch.fx.GraphModulerzOptional[Sequence[object]]rbzOptional[ShapeEnv]r<z Optional[int]r7rr;rrzOptional[bool]r+z4Optional[Callable[[list[ir.ExternKernelNode]], Any]]rrrrrrrzOptional[dict[str, int]]r Optional[str]rrArzOptional[GraphLowering]r6rArzOptional[Sequence[int]]r8rrr9)rr9)rv torch.Tensorrz=tuple[Sequence[Union[int, Expr]], Sequence[Union[int, Expr]]])rvrBrz)tuple[list[sympy.Expr], list[sympy.Expr]])rzOUnion[ir.TensorBox, ir.StorageBox, ir.Buffer, WorkspaceArg, ir.TorchBindObject]rzSequence[Expr])rz2Union[ir.Buffer, WorkspaceArg, ir.TorchBindObject]rr )r z.Union[torch._inductor.ir.IRNode, device, None]rr-rr)T)rrnrrrrj)r torch.device)r rCrIterator[None]r+)rarkrrrr)r6r>rr>)rar@rzlist[torch.Tensor]rr>rr)rzOrderedSet[Node])r6r>rr9)r rCrr9)rz,torch._subclasses.fake_tensor.FakeTensorMode)rr>rzrz2Union[ir.TensorBox, ir.Buffer, ir.TorchBindObject])rr>rz torch.dtype)rr>rzUnion[int, Expr])rvrrr)rz ir.Operationrr>)r'z ir.Bufferr#rrr>)r, list[str]rr>)r:r8rr9)r6r>rrB)r6rArz Union[Tensor]rr>rt)rr r6rArrE)r6r>rdzOptional[torch.device]rr>)rr>rv tuple[object]rwdict[str, object]rzUnion[Expr, TensorBox, None])rrervrrwdict[str, Any]rr)rrBrr)rr>rvz tuple[()]rwrHrzOUnion[Constant, TensorBox, ShapeAsConstantBuffer, ir.Subgraph, TorchBindObject])rrrvrrwrrr)rr>rvrGrwrHrr9)rr)rrD) rrr tuple[Any]rrIrrJrrIrr9)rrrobject)rrrkzOrderedSet[sympy.Symbol]rr9FNNN rrrrArzOptional[PythonWrapperCodegen]rz!Optional[GraphPartitionSignature]rr9)rrrr9)rz)tuple[ValueWithLineMap, ValueWithLineMap])rrrr9)rzVtuple[int, list[tuple[BaseSchedulerNode, int]], list[tuple[BaseSchedulerNode, float]]])rCompiledModule)r(rbrrN)rrrrF)rrF)r6r>rr)H__name__ __module__ __qualname____firstlineno____annotations__rrhr{r~rrrrr contextlibrrr staticmethodrrrrErrpropertyrrrr rrrr r(r-r;r@rDrRrWrOrirrrrrrrrrrr4r[rrrrrrrrrrrrr8r;r>__static_attributes__ __classcell__res@rxrrBs7"" 6:(,"&!%) "!$7;,0+/04"37 +MC MC3MC& MC  MC  MCMC#MC! MCMCMCMC5MC *!MC")#MC$.%MC&'MC(1)MC*+MC, -MCMC^1# # F# J 2# #  #&EFE EM>M M  M<(4 ((YYv   +     6;zA D E$FQQ ;Q8, 8&FK( @     !)6 @37    "/    &&Rfff" f & fPvp55272727" 27   27h[ [ [ " [  [ z $$$$I:I:I:# I:  I: # I: I:VjX S;S;3KS; S;jQ"'+>BBF #W#W%#W< #W @ #W  #WJ]0C]0 ]0~q" 2q"f 8&%*7 7"9=5<->F,F FP :33ryrc^^\rSrSrSrSU4SjjrSSU4SjjjrSrU=r$) ri z Mostly a helper class for the subgraph lowering. The main goal is to call init_wrapper_code with the subgraph related arguments. c2>Xl[TU]"U0UD6 grt)rrr)r`rrvrwres rxrSubgraphLowering.__init__ s  $)&)ryc`>[TU]SURURRS9 g)NT)rrr)rrr6rr()r`rrrrres rxr"SubgraphLowering.init_wrapper_code s. !)) $ 8 8 " ry)r)rrrvrrwrrr9rLrM) rOrPrQrR__doc__rrrWrXrYs@rxrr sV * "'+>BBF    %  <  @     ryr)rvrrwrrr9)rrErzOptional[torch.dtype])rrrr)rrkrr>rz1Union[Tensor, torch._C.ScriptObject, GraphModule])rrlrdict[Node, tuple[int, ...]])rr`rzdict[Node, object])rrlrr`rr9) __future__rrTrRrZloggingrrrrr4 collectionsrrtypingrrrr r r|r rtorch._loggingtorch.fxr r torch._decomprtorch._dynamo.utilsrr"torch._library.fake_class_registryrtorch._library.utilsrrrtorch._prims_commonrrtorch._subclasses.fake_tensorrtorch._utils_internalr%torch.fx.experimental._backward_statertorch.fx.experimental.sym_noderr%torch.fx.experimental.symbolic_shapesrrrrr r!r"r# torch.fx.noder$torch.fx.passes.reinplacer%torch.utils._mode_utilsr&torch.utils._ordered_setr'torch.utils._sympy.numbersr(rr*r+r,codegen.commonr-r.r/r0r1r2r3r4excr5r6r7r8fx_utilsr9r:r;r<r=r>r?r@rArBrCrDrErFrrGrHrIrJrKrLrMrNrOrPrQrRruntimerSruntime.autotune_cacherTr rUr`rVrWrXrYrZr[r\r]r^r_r`rarb virtualizedrcrdcollections.abcrerfrgrhtypesritorch._higher_order_ops.effectsrjrktorch.fx.graphrlcodegen.wrapperrm dependenciesrnr@rorNtorch._inductor.codecacherp getLoggerrOr_logginggetArtifactLoggerrrrr[r>r*torch._inductor.fb.utilsrrrrrrrrrrrrruryrxrs" #%@@   ,4?:75:?L   1//-!!    %    $8&(FF ;$$5!,:'<<=N5!00<H  yy~~$??, 8 (   ! 6 $50V/ V/+FV/ V/rn"3EHH((n"3bE } ry