΅iRlSSKJr SSKrSSKrSSKrSSKJr SSKJr SSK J r J r J r J r SSKJr SSKrSSKrSSKrSSKJrJrJrJrJr SSKJr \ (aSS KJr S qS qS qS)S*S jjr S+S jr!S+S jr""SS5r#S,Sjr$Sr%Sr&Sr'S-Sjr(S.Sjr)S/S0Sjjr*\RVS/S1Sjj5r,\RVS5r-"SS\#5r."SS\ 5r/S2Sjr0S3Sjr1S4Sjr2S5S jr3S6S!jr4\"S"S#55r5\"S$S%55r6\RnS7S&j5r8S8S'jr9S(r:g)9) annotationsN)deque) dataclass)castoverloadProtocol TYPE_CHECKING)TypeIs)_get_dispatch_stack_at_len_torch_dispatch_stack_pop_torch_dispatch_stack_push_on_torch_dispatch_stack DispatchKey)/set_is_in_mode_without_ignore_compile_internals)SequenceFc(U(a[$[$N)_is_in_torch_dispatch_mode$_is_in_non_infra_torch_dispatch_mode)include_infra_modess V/home/james-whalen/.local/lib/python3.13/site-packages/torch/utils/_python_dispatch.pyis_in_torch_dispatch_moder)s  #2c[$r)0_is_in_any_mode_without_ignore_compile_internalsrr/is_in_any_mode_without_ignore_compile_internalsr1s ;;rc[RR5n[U5H<n[ U5nUR 5(aM%UR 5(aM< g g)NTF)torch_Cr ranger is_infra_modeignore_compile_internals) stack_lenidxmodes r any_torch_dispatch_mode_on_stackr'5s[224IY%c*        ( ( * *   rc^\rSrSrSrSrU4SjrSSjrSrSSjr Sr S r \ S 5r \ SS j5r\ SS j5r\ SS j5rSrU=r$)TorchDispatchModeFa A ``TorchDispatchMode`` allows you to override the meaning of all ``__torch_dispatch__`` overridable functions within a dynamic scope, without having to actually create a tensor subclass or manually monkey-patch functions in the PyTorch API. Some common situations where you should use a mode: * You want to override the meaning of factory functions, or other functions that do not otherwise take a tensor as an argument (these cannot be overridden with tensor subclasses). * You want to override the behavior of all functions without needing to wrap your inputs in tensor subclasses; e.g., if you are just interested in logging intermediate computations. * You want to control the order of execution of various tensor subclasses explicitly, rather than implicitly via the return of ``NotImplemented``. Independent subclasses of :class:`TorchDispatchMode` are compositional: modes can be pushed onto a stack using ``with MyMode():``. When you call functions in the PyTorch API inside your ``__torch_dispatch__`` implementation, by default, they will forward on to the next mode on the mode stack. If you want recursively call back into your current ``__torch_dispatch__`` implementation, either explicitly invoke ``self.__torch_dispatch__(...)``, or use the context manager ``self`` to make PyTorch API self-referential (beware of infinite loops, in this case!) Fc >[TU]"S0UD6 UR5(aQSUR;a@URSn[ U[ 5(d[ R"USS9Ulgggg)N__torch_dispatch__T) recursiver) super__init_subclass___should_skip_dynamo__dict__ isinstance classmethodr_disable_dynamor,)clskwargsraw __class__s rr/#TorchDispatchMode.__init_subclass__kso !+F+  " " $ $#s||3ll#78!#{33-2-B-B3RV-WC*44 %rcUbB[U[RR5(d [ S5eXR S'[ 5Ul[ 5Ul[ 5Ul g)Nz,_dispatch_key must be a torch._C.DispatchKey _dispatch_key) r2rr rAssertionErrorr1rold_dispatch_mode_flags!old_non_infra_dispatch_mode_flags8old_without_ignore_compile_internals_dispatch_mode_flags)selfr;s r__init__TorchDispatchMode.__init__ssX  $mUXX-A-ABB$%STT-:MM/ *49G$>Cg. G Erc[US5(d[5Ul[US5(d[5Ul[US5(d[5Ulgg)Nr=r>r?)hasattrrr=r>r?r@s r"_lazy_init_old_dispatch_mode_flags4TorchDispatchMode._lazy_init_old_dispatch_mode_flagssYt6778=D (t@AABG'D 2 L    I rc[er)NotImplementedErrorr@functypesargsr6s rr,$TorchDispatchMode.__torch_dispatch__s!!rcUR5 URR[5 SqURR[ 5 [ =(d UR 5(+qURR[5 [=(d UR5(+q[[5 [U5 U$)NT) rFr=appendrr>rr"r?rr#r _push_moderEs r __enter__TorchDispatchMode.__enter__s //1 $$++,FG%)" ..55 0  1 L8J8J8L4L - EELL <  = 30022 9 8 <  4 rcJURRSS5nUcURRSS5nURR5qUR R5qURR5q[[5 [U5 g)Nr; _mode_key) r1getr=poprr>rr?rr _pop_mode)r@exc_typeexc_valexc_tbmb_dk_or_mode_keys r__exit__TorchDispatchMode.__exit__s MM--otD  $!% 1 1+t D %)%A%A%E%E%G"  2 2 6 6 8 -  I I M M O 9 8 <  #$rc@[R"SSS9 U"U0UD6nU$)NzP`Mode.push()` is no longer necessary and can be replaced with just `with Mode()`) stacklevel)warningswarn)r5rMr6instances rpushTorchDispatchMode.pushs* ^ ''rcg)NFrr5s rr"TorchDispatchMode.is_infra_modesrcg)aSkip Dynamo when the flag is set to True This is temporary measure to rollout a feature that skips PT2 compilation inside __torch_dispatch__ frames. If this flag is off, we would expect following: class YoloMode(TorchDispatchMode): @classmethod def _should_skip_dynamo(cls): return False def __torch_dispatch__(self, func, types, args=(), kwargs=None): return torch.ops.aten.mul.Tensor(args[0], args[1]) x = torch.ones(5) with YoloMode(): out = torch.compile(torch.add, backend=backend, fullgraph=True)(x, x) # instead of recursively disabling, we are compiling into __torch_dispatch__ assert len(backend.graphs) == 1 Trrhs rr0%TorchDispatchMode._should_skip_dynamos0rc0UR5(agg)aIgnore operators that are compiled via torch.compile. If ``True``, then this TorchDispatchMode ignores operators that are optimized by :func:`torch.compile`. Mechanically, this involves turning off the TorchDispatchMode throughout the whole compilation process, and turning it back on for the runtime of the compiled artifact(s). For example, @torch.compile def f(x): return x.sin().cos() with LoggingMode(): f(x) The above example will not log anything if ``LoggingMode.ignore_compile_internals()`` is True. torch.compile will fuse sin() and cos() into a single operation and this TorchDispatchMode will not be passed sin and cos. If ``False`` (default), :func:`torch.compile` will respect the eager semantics of passing this TorchDispatchMode all operators that would have run during eager execution. The way this will usually happen is that :func:`torch.compile` will just fallback to eager-mode PyTorch. TF)r"rhs rr#*TorchDispatchMode.ignore_compile_internalss8     r)r=r>r?rrNreturnbool)__name__ __module__ __qualname____firstlineno____doc__supports_higher_order_operatorsr/rArFr,rRr]r3rer"r0r#__static_attributes__ __classcell__)r8s@rr)r)FsD',#X   ">%*2rr)c@[5nUS:a[US- 5$g)z_ Return the top user mode on the stack (the next one that would be executed) if there are any. rN)r r )r$s r_get_current_dispatch_moder| s% *+I1}%i!m44 rcU[RRR[RRR4;aX[ S[RRRS[RRRSU35eSSKJn U"U5n[RRU5nUbUb [ S5eUcU$U$)Nzkey must be either FUNCTIONAL (z) or PROXY (z1) _TorchDispatchModeKey, got r)_get_dispatch_mode_pre_dispatchzEAt most one of pre_dispatch_mode and post_dispatch_mode may be active) rr _TorchDispatchModeKey FUNCTIONALPROXYr< torch._opsr~_get_dispatch_mode)keyr~pre_dispatch_modepost_dispatch_modes r_detect_infra_moders  &&11 &&,,-ehh.L.L.W.W-XY 8899??@A   ;7<44S9$);)G S   !! rcSSKJnJn U"U5n[RR U5nU(aU(a [ S5eU(a U"U5nU$U(a[RRU5$g)Nr)r~unset_mode_pre_dispatchzECan't have active infra mode on both pre and post dispatch mode stack)rr~rrr rr<_unset_dispatch_mode)rr~rrrr&s r_unset_infra_moder1slS7<44S9/ S  &s+ xx,,S11rc# # U[RRR[RRR4;a [ S5e[ U5nUv Ub [U5 gg!Ub [U5 ff=f7f)Nz>> H   y%d+A  0 0 3BC   %4 rcU[RRR:Xa SSKJn U"5$Ub)[ U[RR5(a [U5$g)Nr)_pop_mode_from_pre_dispatch) rr rrrrr2rr )rrs rrXrXqsREHH , ,,:*,,yJq%(("@"@AA(++Brc#b# [U5nUv [U5 g![U5 f=f7fr)rXrQ)rolds r_pop_mode_temporarilyr{s& A,C 3 3s / / ,/c#j# SSKJnJn SSKJn SSKJn SSKJn U"5n[U5Vs/sH oa"5PM nnSnSn Sn UH;n [X5(aSn[X5(aSn [X5(dM9Sn M= [5n [U 5Vs/sH n[5PM n nU Hkn[X5(aU (a [S5e[X5(aU(a [S 5e[X5(dMYU (dMb[S 5e X}-v [U 5Hn[U5 M [U5Hn[U5 M gs snfs snf![U 5Hn[U5 M [U5Hn[U5 M f=f7f) Nr)&_len_torch_dispatch_stack_pre_dispatchr)FunctionalTensorMode)SchemaCheckMode)ProxyTorchDispatchModeFTzFCan't have FunctionalMode available both in PreDispatch and Python KeyzNCan't have ProxyTorchDispatchMode available both in PreDispatch and Python KeyzGCan't have SchemaCheckMode available both in PreDispatch and Python Key)rrr#torch._subclasses.functional_tensorr#torch._subclasses.schema_check_moder"torch.fx.experimental.proxy_tensorrr!r2r rXr<reversedrQ)rrrrrmode_len_pre_dispatch_old_pre_dispatch_modeshas_proxy_mode_in_pre_dispatch#has_functional_mode_in_pre_dispatch%has_schema_check_mode_in_pre_dispatchrmode_len old_modesrr&s r_disable_current_modesrsICIBD/45J/K/K!#%/K&+"*/',1) # a 0 0-1 * a . .26 / a ) )48 1 $)*H&+Ho6ooI6 s 1 13 X  c 2 27U `  c + +0U0U Y $$00Y'D t (34D t 5U"7.Y'D t (34D t 5sH/F3E*?F3F3!E/5AF3F3 F3)E4/AF34FT$V$$rrn)rrrsrtrur,rxrrrrrs%rrch\rSrSr%SSjr\SSj5rS\S'\SSSjj5r \SSj5r \SSS jj5r \SS j5r SS jr SS jr \SSS .SSjjj5r \SSS .SSjjj5r \SSS .SSjjj5r Srg)TensorWithFlattenicgrrrEs r__tensor_flatten__$TensorWithFlatten.__tensor_flatten__s#rcgrr) inner_tensors flatten_spec outer_size outer_strides r__tensor_unflatten__&TensorWithFlatten.__tensor_unflatten__srz torch._C.SizeshapeNcgrrr@dims rstrideTensorWithFlatten.strides;>rcgrrrs rrrs'*rcgrrrs rsizeTensorWithFlatten.sizes9NQ   >> ** << ((( #  59 !  2  =A+/" 599)   2 #  59   2  rrc[U[R5=(a [U5[RLnU=(a [ US5=(a [ US5$)a Returns whether or not a tensor subclass that implements __torch_dispatch__ is 'traceable' with torch.compile. In order for a tensor subclass to support TorchDispatchMode-style tracing in PT2, It must implement two magic methods: __tensor_flatten__ and __tensor_unflatten__. It is also expected to obey some restrictions around traceability and aliasing: * The subclass's __torch_dispatch__() implementation should desugar into pytorch dispatcher operations that can be traced into a graph. * The subclass should use return_and_correct_aliasing(). This is needed today to make sure that torch.compile does the right thing in a few cases around input mutation and output aliasing. Expected magic method signatures: attrs, ctx = t.__tensor_flatten__() attrs: list of attribute name strings for inner tensors ctx: dict containing any other subclass-specific metadata needed for unflattening t = MySubClass.__tensor_unflatten__(inner_tensors, ctx, outer_size, outer_stride) inner_tensors: dict mapping attribute name -> tensor for each inner tensor ctx: dict with subclass metadata in the form that __tensor_flatten__() produces outer_size: expected (possibly symbolic) size that the returned subclass instance should have. Note that this arg is useful for certain subclasses that require the shape info to be constructed. In most cases, this arg can be safely ignored. outer_stride: expected (possibly symbolic) stride that the returned subclass instance should have. Note that this arg is useful for certain subclasses that require the stride info to be constructed. In most cases, this arg can be safely ignored. rr)r2rTensortyperD)t is_subclasss ris_traceable_wrapper_subclassrsN<Q -M$q'2MK / A+ , / A- .rc[U[R5=(a8 U[RL=(a [US5=(a [US5$)z@Same as above, but takes a type argument instead of an instance.rr) issubclassrrrD)rs r"is_traceable_wrapper_subclass_typer)sJ 1ell# / U\\ ! / A+ , / A- . rc UbUOUR5nUbUOUR5nUR5upE0nUHnU"U[X55Xg'M [ U5R XeX#5nUR U:wa'[S[ U5SUSUR 35eUR5U:wa+[S[ U5SUSUR535eU$)a Given a traceable, wrapper tensor subclass ``t`` that implements ``__torch_dispatch__`` and holds some inner tensors, and a callback of type ``Callable[[str, torch.Tensor], torch.Tensor]``, `transform_subclass` will construct a fresh instance of the wrapper tensor subclass. It will do so by grabbing each inner tensor attribute from the wrapper, passing them into ``callback`` to get a transformed tensor, and putting each transformed tensor into the fresh tensor subclass instance. Note: this function will not handle ensuring that the fresh subclass gets the same (autograd, and aliasing) metadata as the original tensor. This is generally handled in other subsystems like AOTAutograd. zExpected return value from z.__tensor_unflatten__() to have shape equal to z , but got: z/__tensor_unflatten__() to have stride equal to )rrrgetattrrrrr<) rcallbackrrattrsctxtransformed_tensors_dictattrsubs rtransform_subclassr3s *51668J#/#;<L%%'JE!)1$8H)I & q' & & z C yyJ)$q'3(\SYYK A   zz||#)$q'3+nK ~ G  Jrc^[T[RR5(d[ S[ U535e[U[ 5(d[ S[ U535e[U[[ 45(d[ S[ U535eSU4SjjnURHupVU"X%X65 M g)a Given: an OpOverload, a SchemaInfo (cached information from torchgen about schema), and the inputs/outputs to the OpOverload, this function checks to see if func is a view operator (by checking if any of the outputs in the op's schema are immutable aliases of inputs). If so, this function manually aliases the storage of the output tensor with its corresponding input tensor alias. It does this by unsafely overwriting the storage field of the output tensor to be the same storage as the input. z func must be an OpOverload, got zargs must be a tuple, got z"outs must be a list or tuple, got c P>[U5n[U5nX#La[U5(d[U5(aj[U[5(aUOU/nUHJn[U5[U5LdM[ S[ T5S[U5S[U5S35e [U[5(a UHn[ R"XP5 M g[U[ R5(d[ S[U535e[ R"X5 g)NzCalled z with input of type z and output of type z. But expected types to match.zexpected torch.Tensor, got ) rrr2listr<strr_functionalize_unsafe_setr)argretarg_typeret_typeret_listrrKs ralias_non_inplace_storage<_correct_storage_aliasing..alias_non_inplace_storageps99  # .x 8 81(;;(d33s#H9DG+(!#d),@c L..23i[8VX" c4 //7c5<<00$'B49+%NOO  + +C 5rNrpr) r2r_ops OpOverloadr<rtuplerread_only_alias_match_indexes)rK schema_inforMoutsrarg_idx return_idxs` r_correct_storage_aliasingr ]s dEJJ11 2 2?T |LMM dE " "9$t*FGG dT5M * *A$t*NOO)6V +HH!$-1AB IrcURnU(aUR(dg[U5S:wa [S5e[ [ U55$)Nr{z1Expected alias_set to contain exactly one element) alias_setis_writelenr<nextiter)xrs r_get_write_aliasrs@ I AJJ 9~PQQ Y  rc4\rSrSr%S\S'S\S'S\S'Srg ) AliasInfoizset[str]rrqr str | NonenamerNrrrsrtrurrxrrrrrsN rrcH\rSrSr%S\S'S\S'S\S'S\S'S \S 'S rg ) SchemaInfoizlist[AliasInfo]rMr rqis_inplace_view_opzlist[str] | Noneouts_write_aliaseszlist[tuple[int, int]]rrNrrrrrrs$   )($98rrc URS:XGa[UR5nURS5(d [ S5eUSSnSSKnUR SSU5nUR SS U5nUR S S U5n[RRRU5nURRVs/sHun[URc [5O[URR 5URSL=(a URR"UR$S 9PMw nnUR&Vs/sHun[URc [5O[URR 5URSL=(a URR"UR$S 9PMw nnGO,URRVs/sHun[UR(c [5O[UR(R*5UR(SL=(a UR(R"UR$S 9PMw nnURR&Vs/sHun[UR(c [5O[UR(R*5UR(SL=(a UR(R"UR$S 9PMw nn/n[-U5Heup[-U5HQupU R U R -=(a U R"(+n U (dM?UR/X45 MS Mg UV s/sHn [1U 5PM nn [3S U55nUS:XaSnOGU[5U5:wa![7S[UR5-5e[9[:[U5n[=UU[>R@RBURD;UUS9nU$s snfs snfs snfs snfs sn f)Natenzaten::z6Expected torchgen schema string to start with 'aten::'rz =\[[0, ]+\]z=0z =\[[1, ]+\]z=1z=\[(-?[0-9]+), (-?[0-9]+)\]z=[\1,\2])rrrc3(# UHoSLv M g7frr).0rs r !get_alias_info..sC+BaTM+BszUnsupported schema: )rMr rrr)# namespacer_schema startswithr<rertorchgenmodelFunctionSchemaparse argumentsflat_allr annotationsetrrrreturns alias_info before_set enumeraterPrsumr RuntimeErrorrrrrTag inplace_viewtags)rKtorchgen_schema_strr(torchgen_schemaa arg_schemas out_schemasrr  schema_argr  schema_outis_read_only_alias_matchrouts_write_aliases_list non_nonesrrs rget_alias_inforDs ~~!$,,/"--h77 H  2!"5!ff^T;NO ff^T;NO ff *K9L #..77==>QR%..77  8 \\1CEs1<<;Q;Q7RT1Kall6K6KVV  8   $%,,  - \\1CEs1<<;Q;Q7RT1Kall6K6KVV  -   (\\++  , \\1CEs1<<;R;R7ST1Kall6K6KVV  ,   $\\))  * \\1CEs1<<;R;R7ST1Kall6K6KVV  *   %'!(5&/ &< "J$$z';';;(* ))) %('-44g5JK '= 6&11%0[1C+BCCIA~/3 c12 21C 4EEFF!$s)-DE  !9911TYY>-&CK Q        &1s" AcSnUHn[U[R5(dM$[RR [RR UR R55nUc"[SUR R35e[RRUR5U5n U(+$ U(+$)a Suppose that an operator has CompositeImplicitAutograd decomp registered. Would autograd have used this decomposition? It will only use it if there isn't an explicit backend registration for the device as well. This function will tell if this would have occurred. Why do we need to apply these decompositions later? When inference mode is on, the autograd key is bypassed entirely, so a lower level mode cannot rely on the decomposition have been applied. It's easy to accidentally never apply the decomposition, resulting in an operator showing up in a graph that is unexpected. Why do we need to AVOID applying the decomposition when autograd wouldn't have decomposed? If autograd doesn't decompose, this means in eager mode we would have run the fused kernel. It must be possible to trace this fused kernel directly into the graph for fidelity with eager (NB: a user has the option of then further decomposing at proxy tensor mode via decomposition table, but we must preserve it to proxy mode to have the choice.) Why does functionalization need to also perform the test here? This is because some CompositeImplicitAutograd decompositions are not functional. If we are eventually going to decompose, we need to do this while we can still turn functionalization back on, so those decompositions get functionalized. So an early decomposition in functionalization may still be necessary. Note that if proxy tensor decomposition process could turn functionalization back on, this wouldn't be necessary, and maybe that is a useful thing to do anyway because the decomposition table is user specified and a user could violate the functional decomp requirement with a bad decomp. If this happened, then you could always pass through functionalization. Fz(failed to parse dispatch key for device ) r2rrr _parse_dispatch_key_dispatch_key_for_devicerrr<%_dispatch_has_kernel_for_dispatch_keyr)rK flat_argshas_backend_registrationr< backend_keys rautograd_would_have_decomposedrL+sD %  a & &((6611!((--@K"$>qxx}}oN (-xx'U'U [( $ ' '')(( ''rc ^[T5nU4Sjn[TXA[U[5(dU4OU5 UR(Ga[ U5VVs/sH"upg[ URU5cM UPM$ nnn[U5S:wa [S5eSSK J n [USU 5(d[RRR5 [R R#5n [R R%S5 T"U0UD6 [R R%U 5 SSS5 UR&n U cU$[U 5S:Xa U"U SXAU5$[)U5"U V s/sH n U"XX5PM sn 5n U $s snnf![R R%U 5 f=f!,(df  N=fs sn f)a. This function should be used by wrapper tensor ``__torch_dispatch__`` subclasses that would like to work with torch.compile. It ensures that the subclass properly implements the aliasing behavior of every op, which is needed for correctness in AOTAutograd. This function will handle: * When we see a view op, we will alias the storages of any input and output tensor subclasses * When we see an inplace or out= op, we will directly return the corresponding input tensor, instead of returning a (potentially) fresh output tensor. c>[RRRT X#S9upE[ UR 5VVs/sHupgXR ;dMUPM nnn[U5S:wa [S5eUSn UR U n U RbU RU;aXZR$XI$s snnf)N)rMr6r{z>Expected exactly one argument index for the given output aliasr) rfxoperator_schemasnormalize_functionr4rMrrr<r) output_aliasrrMr6new_args new_kwargsrr< arg_indicesr%arg_inforKs rget_arg_from_alias7return_and_correct_aliasing..get_arg_from_aliasys$xx88KK t L  $K$4$45 5$!9TA5   { q  P !n##C( == $*)Dmm, ,} s CCNr{z5expected exactly one mutated arg for inplace_view opsr)FunctionalTensorT)rDr r2rrr4rrMrr<rrYrutils _mode_utils no_dispatchr _meta_in_tls_dispatch_include!_set_meta_in_tls_dispatch_includerr)rKrMr6outrrWrr mutated_argsrY meta_in_tlsschema_info_outs_write_aliases write_aliasouts_to_returns` rreturn_and_correct_aliasingrees$!&K* kz#u/E/E#3 %%% "$ ' 0 0 34 '   |  ! G  I,q/+;<<((446$hhDDF ::4@L$)&)HH>>{K7&1%C%C"%-  )*a/! *1 -{&  #Y > =  $ G= N ] 2HH>>{K760 s6F';F' >GF-'G G"-!GG G)T)rrqrprqro)rpzTorchDispatchMode | None)rpzlist[TorchDispatchMode])r&r)rprr)rz3DispatchKey | torch._C._TorchDispatchModeKey | None)rzDispatchKey | None)robjectrpzTypeIs[TensorWithFlatten])rrrpzTypeIs[type[TensorWithFlatten]])NNr)rpr)rpr)rKztorch._ops.OpOverloadrIzSequence[torch.Tensor | object]rprq); __future__r contextlib functoolsrb collectionsr dataclassesrtypingrrrr typing_extensionsr rr)torchgen.modeltorch._Cr r r rrtorch._C._dynamo.guardsrcollections.abcrrrrrrr'r)r|rrrrrQrXcontextmanagerrrrrrrrr rrrcacherDrLrerrrrtsr"!::$ T(#',$380<"CCL42" # A!*,  55p%-%;;|#L'T?CD !     9 9  9$ __D7( 7(,K7( 7(t`r