ȅiRSSKJr SSKrSSKrSSKrSSKrSSKJrJrJ r J r SSK r SSK J r SSKJr SSKJr SSKJr SS KJrJr SS KJrJr SS KJr \(aSS KJr SS KJr SSK J!r!J"r" SSKJ#r# \RH"\%5r&\RN"SS55r(\RN"SS55r)\RN"SS55r*\RN"SS55r+S+Sjr,S,Sjr-S-Sjr.S.Sjr/\RN"SS55r0S/Sjr1S0Sjr2\RN"S S!55r3S1S"jr4S2S#jr5S3S$jr6S3S%jr7S4S&jr8S5S'jr9S6S(jr:\5\6\7/4S7S)jjr;S8S*jrS[TR5S[TR53$)NzHsucc_nodes must be a subset of succ_nodes_for_ordering. len(succ_nodes)=z, len(succ_nodes_for_ordering)=)lenr2r3selfsr);MemoryPlanningInfoForBuffer.__post_init__..7s2"4??344STWX\XtXtTuSvxr()torch_checkr6r2r3r7s`r) __post_init__)MemoryPlanningInfoForBuffer.__post_init__4s1  C(D(D$E E x r(r N)returnNone)r"r#r$r%r-r&r. dataclassesfieldr r2r3r=r'r r(r)r+r+'sUJIs0;0A0A"1J->I=N=N">: r(r+c\rSrSr%SrS\S'SrS\S'\R"\ S9r S\S'\R"\ S9r S \S '\R"\ S9r S \S 'S r g )MemoryPlanningInfoForNode<rrindexsizer/z7OrderedSet[Union[SchedulerBuffer, FreeableInputBuffer]] pred_buffersr1 pred_nodesr2r N)r"r#r$r%rFr&rGrArBr rHrIr2r'r r(r)rDrD<siE3ND#M*5I1<0A0A"1J-1<0A0A"1J-r(rDc^\rSrSr%S\S'\R "\S9rS\S'S Sjr S Sjr S r g ) FreeableInputBufferKrnamer/r+ mpi_buffercUR$NrMr7s r)get_nameFreeableInputBuffer.get_nameRs yyr(c,[UR5$rP)hashrMr7s r)__hash__FreeableInputBuffer.__hash__UsDIIr(r N)r?r)r?r) r"r#r$r%r&rArBr+rNrRrVr'r r(r)rKrKKs, I.9.?.?3/J+r(rKc lSSjn[R"[5n[R"[5n[5nUHnURR HnUR U;dM[U5(aM'XGR RU5 U"U5XWR '[U[5(aUR(aMX7R RU5 M M [5nUH n [U [XYX9XIS95X'M" U$)z Create and keep track of all input buffers that can be freed during the program Returns: A dictionary containing all freeable input buffers, keyed by their names. c@[RRU5$rP)rgraphget_dep_size_hint)deps r)_dep_size_hint.get_freeable_input_buf.._dep_size_hintdsww((--r()r.r2r3)r\rr?r) collections defaultdictr dict read_writesreadsrMradd isinstanceris_fakerKr+) nodes graph_inputsr]dep_name_to_succ_nodes#dep_name_to_succ_nodes_for_orderingdep_name_to_sizenoder\name_to_freeable_input_bufdep_names r)get_freeable_input_bufroYs.  +  +((,v##))Cxx<'-c228AEEdK1?1D$XX.&sG44.xx8<UR5[RRR;aST UR5'g[ UR R[5(aST UR5'g[ UR R[5(aSnURHjn[ UR T5(aM UR R5H,n[ UR T5(dM UT"US5- nM. Ml U(aSOUS4T UR5'U$[RRRUR R5SS9[UR R!55-nU(aSOUU4T UR5'U$)N)rrrT)fallback)rRrrZ schedulermutation_real_namererllayoutrr users get_outputssizevars size_hint get_numelr get_dtype) sched_bufuser_of_MultiOutputLayoutr-userbufbuf_sizerqrr_compute_and_update_buf_sizesched_buf_to_sizes r)rGcompute_size_for_scheduler_buffer.._compute_and_update_buf_sizes    177#4#4#G#G G6< i002 3  --z : :6< i002 3  --/@ A AJ!dii4499002C!#((K88"&B3&MM 3(/J7 i002 3 ww''11((*Q2y~~779:;H/H7 i002 3Or()F)r~rrboolr?r)irrqrurrravaluesrR) name_to_bufr~rqrrrrs @@@@r)!compute_size_for_scheduler_bufferrswT %48FGL"?C @!'')     '8 8 ( 3 * r(c[U5n[R"[5n[R"[5nUHxnURHenXFR R U5 [U[5(aUR(aMHX6R R U5 Mg Mz [[RRRR55H"upxX8==UU-ss'XH==XG-ss'M$ UH$n [!X)SX)SX9XIS9XlM& g)z For each SchedulerBuffer, assign its size info and successor nodes. A buffer's successor nodes determines when a buffer can be freed. rr )r-r.r2r3N)rr_r`r unmet_dependenciesrMrdrerrfreversedrrZrurvitemsr+rN) rgrrrirjrlr\mutating_buf_name real_buf_namebuf_names r)1assign_memory_planning_info_for_scheduler_buffersrs6:+F  +  +(**C 0 9 = =d CsG,,&xx044T: +-5 ,,224-( -1G 2  - ,: / B : - +F(215'1!4-7$G$Q , ( r(c [R"[5n0n[R"[5nUHn[SUR555nXU'UHn XIR U5 M UR5H3n U R R Hn XiR U 5 M M5 M UR5H3n U R R Hn XiR U 5 M M5 [U5Heup[SUR555n XWnXGnURU5 URU5 [U U XgXGUS9Ul Mg g)zD Assign to each scheduler node its predecessor and successor nodes. c3b# UH%nURRHnUv M M' g7frP)rNr3).0buffer succ_nodes r) Bassign_memory_planning_info_for_scheduler_nodes..'s0 ,#..FF  F ,s-/c3L# UHoRRv M g7frP)rNr-)rrs r)rr?sWDV&**55DV"$)rFrGrHrIr2N) r_r`r ryrdrNr2r enumeratesumdiscardrDmpi_node)rgname_to_fused_noderrmnode_to_pred_nodesnode_to_succ_nodesnode_to_pred_buffersrlr2rrfreeable_bufferrFr-rIs r)/assign_memory_planning_info_for_scheduler_nodesrs|  +RT  +  **,  $.4 $I  ) - -d 3$ &&(F#..99 $/33F;:)&6<<>(33>>I + / / @?? !' WDDTDTDVWW '- '-  4 4 1-3)/!   (r(cH\rSrSr%S\S'S\S'S\S'S\S'S\S'S rg ) BufferInfoiQz+Union[SchedulerBuffer, FreeableInputBuffer]rrr-r. start_stepend_stepr Nr!r r(r)rrQs 77ONOMr(rc ^ [U5VVs0sHup4XC_M snnm /n0nSU 4SjjnUR5HeupSn X;aU"U 5upU ceXU 'UR[U U RR U RR SU 55 Mg [U5Hup4UR 5Hn U R5nSn X;a U"U 5upU S:XaUn XFU 'O U ceXU 'UR[U U RRU RR UU 55 M M UT U4$s snnf)z^ Compute buffer allocation and deallocation sizes and map their lifetime to the node schedule c>SnSnURRnU(aUHnTUnXQ:dMUnUnM UceX4$)N)rNr2)rmax_stepmax_step_snoder2rstep node_to_steps r)_get_end_step_and_snode8compute_memory_timeline.._get_end_step_and_snodets[6:^^.. ' #I.?#H%.N ( "- --''r(rr)rz+Union[FreeableInputBuffer, SchedulerBuffer]r?z'tuple[int, Optional[BaseSchedulerNode]]) rrappendrrNr.ryrRr-)rgrm graph_outputsrrl buf_info_listbuf_to_snode_last_userr input_bufrend_step_snoder~rs @r)compute_memory_timelinerZs"&/u%52%5zt %52L ')M  ( 8 ( 0 ( :??A  ('>y'I $H!- --/=) , $$..$$..    B$ & ))+I!))+HH,+B9+M(r>#H7;)4)5557E)4  ((33((22  ,'4 ,(= ==M2sEc[XU5un n[[U5S-5Vs/sHnSPM nnUHFnXVR==UR- ss'XVR S-==UR -ss'MH SnSn/n [[U5S-5H&n XU - nU RU5 [Xx5nM( Xy4$s snf)z Given a list of nodes in their execution order, estimate the peak memory, by keeping track of the liveliness of SchedulerBuffers and FreeableInputBuffers. Returns: int: peak memory List[int]: memory usage at each node (or each step). r r) rranger6rr-rr.rmax) rgrmrr_memorybuf_info max_memory cur_memorymemories_at_nodests r)estimate_peak_memoryrs2 =M1a s5zA~. /.Aa.F /"""#x':'::#  1$%););;%" JJ 3u:> "Qi   ,0 #  **!0s Cc*\rSrSr%S\S'S\S'Srg) SNodeMemoryirr-r.r Nr!r r(r)rrs ONr(rc|[XU5up4n[[U55Vs/sHn[SS5PM nnUHknXgR=R UR - slUR S:wdM@XgR =RUR- slMm 0n[U5H upXiX'M Sn Sn /n [[U55HJnXnR nXnRnX- n U n[X5n U U-n U nU RUU45 ML U U UU4$s snf)a Alternative version of estimate_peak_memory, that respects the fact, that every SchedulerNode has multiple phases: 1. alloc ( outputs ) 2. run_kernel 3. dealloc last_use buffers estimate_peak_memory collapses memory into one value: size_alloc - size_free While peak memory happens after alloc. Duplicating the code to not migrate all callsites at once, In future usages of estimate_peak_memory will migrate to this version. rr) rrr6rrr-rr.rrr)rgrmrrrrstep_idx_allocfreersnodes_allocfreeirlrrsnodes_curr_memoryrallocfree post_alloc post_frees r)estimate_peak_memory_allocfreersZ./F =/+M+ 6;3u:5FG5F+a+5FG"../::h>Q>QQ:    " 00 1 ; ;x?Q?Q Q ;" U#!3!6$JJ 3u: "%00!$..  0 d  !!:y"9:   3HsD9c^^^"SS[5n"SS[5n[5m[5n[5nTHGn[URR 5SS.TU'TUSS:XdM6UR U5 MI [UR55[UR55-H?n S[U RR5U R5U;aS OS-0Xi'MA [S UR555n Sn UHGn X;aXU RR- n M%X;dM,XU RR- n MI [X5n X- mTHnURRH4n XiSS :XdMTUS ==U RR- ss'M6 UR!5H4n XiSS:XdMTUS ==U RR- ss'M6 M /n["R$nSnU[T5:GaU(GaUS:a%['S U55U:a['UU4S jS9nO['UUU4SjS9nUR)U5 UR+U5 US - nU URR,- n [X5n U TUS -n X- mURRH@nTUSS:deTUS==S -ss'TUSS:XdM/UR U5 MB URRHmn XiSS:deXiS==S -ss'XiSS :XdM,U RRH'nTUS ==U RR- ss'M) Mo U[T5:a U(aGMU[T5:a [/S5eU$)a A bfs-based greedy topological order. LPMF stands for "Least Peak Memory First". The idea is from this paper: Buffer memory optimization for video codec application modeled in Simulink https://www.cs.york.ac.uk/rts/docs/DAC-1964-2006/PAPERS/2006/DAC06/PDFFILES/P0689.PDF The algorithm maintains the max memory so far. At every iteration, for each scheduleable node, it computes: - how much memory needs to be allocated for the output buffers of this node; - how much memory can be freed as a result of executing this node. This gives us two values for each node: (1) mem1: memory during the execution of the node; (2) mem2: memory after executing the node, after some input buffers are freed. The greedy approach select as follows: (i) if there are nodes whose mem1 values are below the max memory so far, then pick the node with the lowest mem2 value; (ii) otherwise, pick the one with the lowest mem1 value. c*\rSrSr%S\S'S\S'Srg)'topological_sort_lpmf..NodeInfoi6rindegreememory_to_freer Nr!r r(r)NodeInfor6s  r(rc \rSrSr%S\S'Srg))topological_sort_lpmf..BufferInfoi:r outdegreer Nr!r r(r)rr:sr(rr)rrrrr c3N# UHnURRv M g7frPrNr.)rrs r)r(topological_sort_lpmf..Ss#[SURR5[T5S9$)Nc3N# UHnURRv M g7frPrrF)rrs r)r:topological_sort_lpmf....|s#)AI"**00)Ar)default)minrr2r6)rlrgs r)r9'topological_sort_lpmf..{s*)-)A)A J "r(keyc>URRT:aURROSURRTUS- URR4$)Nrr)rrGrF)rl memory_gap node_infos r)r9rsQ*.--*<*V>VV:"; 7W c%j %6%6d3u:QRR Or(c F^ "SS[5n[5m [R"SS55nS U 4Sjjn/nUHYn[ UR R 5SS.T U'T USS :XdM6[R"XB"U"U5U55 M[ /nS nU[ U5:aU(a[R"U5Rn[ U5T US 'URU5 US - nUR RHSn T U SS :deT U S==S -ss'T U SS :XdM/[R"UU"U"U 5U 55 MU U[ U5:a U(aMU[ U5:a [S 5eU$)a A BFS topological sort that selects nodes whose dependencies are executed the earliest. This follows a FIFO idea. Specifically, at every iteration, for each node that is schedulable, we gather the order in which its predecessor nodes are executed, and this sorted list of execution orders of predecessor nodes defines the priority. We select the node whose predecessors nodes are executed the earliest. The FIFO idea aims to reduce the liveness duration of buffers created. c*\rSrSr%S\S'S\S'Srg)&topological_sort_bfs..NodeInfoirrrr Nr!r r(r)rrs   r(rc4\rSrSr%S\S'S\S'S SjrSrg) .topological_sort_bfs..NodeWithPriorityi list[int]priorityrrlcURUR:XaAURRRURRR:$URUR:$rP)rrlrrF)r8others r)__lt__5topological_sort_bfs..NodeWithPriority.__lt__sP}}.yy))//%**2E2E2K2KKK==5>>1 1r(r N)rNodeWithPriorityr?r)r"r#r$r%r&rr'r r(r)rrs 2r(rc>TUSS:Xde[[U4SjURR555nU$)Nrrc34># UH nTUSv M g7f)rNr )r pred_noders r)r?topological_sort_bfs.._node_priority..s?W) )$W-?Ws)sortedr rrI)rl exec_ordersrs r)_node_priority,topological_sort_bfs.._node_prioritysKz*a/// ?C}}?W?W  r(r)rrrrrr z3Failed to schedule, while loop ran too long for bfs)rlrr?r)rrarA dataclassr6rrIheapqheappushheappoprlrr2r) rgrrr rrlrrrrrs @r)topological_sort_bfsrs9486I22213'*4==+C+C'DrR $ T?: &! + NN!#3N44H$#O )+HI c%j %6 &78== ,/M - ) &Q '//::IY' 3a7 77 i  , 1 ,#J/14%$^I%> J ; c%j %6%6"3u:PQQ Or(c~^^^^^[5m[5m/m[5mSUUUUU4SjjmUHnUR5HnUTU'M M! UHCnURR[ SURR 55-TU'ME [UU4SjS9H nT"U5 M T$)a This is a DFS topological sort. The setup is similar to `topological_sort_schedule` in scheduler.py. The difference is the order nodes are visited in the outer loop. In `topological_sort_schedule`, nodes are visited in their original order. In this function, nodes are visited based on their priority -- for each node, we compute the total memory of all buffers it reads from or writes to, and we visit the nodes in ascending order of this priority. c>UT;a{TRU5 URVs/sH$nURT;dMTURPM& nn[UU4SjS9H nT"U5 M TR U5 ggs snf)Nc:>TUURR4$rPrnsize_with_readss r)r95topological_sort_dfs..visit.. s/!*N>N)Or(r)rdrrMrr) rr\ dep_nodesrl name_to_noderesultseenrvisits r)r#topological_sort_dfs..visits D= HHQK///C88|+' SXX&/  Od  MM!  s BBc3L# UHoRRv M g7frPr)rpred_bufs r)r'topological_sort_dfs..s9 :Th   ) ):Trc:>TUURR4$rPrrs r)r9&topological_sort_dfs..s_Q-?AQAQ,Rr(r)rrr?r@)r raget_buffer_namesrrGrrHr)rgrlrMrrrrrs @@@@@r)topological_sort_dfsr#s+5,D15L&(F48FO  ))+D!%L , $ 2 2S9 :>--:T:T9 6 ! u"RS d T Mr(c^^^^^Sunmm[RX5m/mSUUUUU4SjjmUHnTUU:XdMT"U5 M g)z Validate that the graph is acyclic by checking predecessor relationships. Raises: RuntimeError: If a cycle is detected in the graph )rr c>TUT:XagTUT:XaOTRU5 SRTVs/sHoR5PM sn5n[SUS35eTTU'TRU5 URR HnX :wdeT"U5 M TR 5 TTU'gs snf)Nz -> z_Cycle detected in memory planning graphPath containing cycle (i -> j: j is a dependency of i): zG This indicates invalid dependency relationships in the scheduler graph)rjoinrRrrrIpop)rl path_inforBLACKGRAYcolor dfs_visitpaths r)r-)validate_graph_acyclic..dfs_visit-s ;%   ;$  KK  $F]]_$FGIKKT+VYZ  d  D11I$ $$ i 2  d !%GsB<N)rlrr?r@)rafromkeys)rgWHITErlr*r+r,r-r.s @@@@@r)validate_graph_acyclicr2sM!E4 MM% 'E$&D2 ;%  dOr(c rUHnUR5HnUR5nXQ;a[USUR5S35eXU:wa6[SUSUSUR5SXR5S3 5eXR;dM|[SUS UR5S 35e M g ) a> Validate that for each node's output buffer, the name_to_buf mapping is correct. For each output buffer buf, we should have name_to_buf[buf.get_name()] == buf. Also validate that no buffer names overlap with freeable input buffer names. Raises: RuntimeError: If buffer name mapping is incorrect or names overlap z from zN is not found in name_to_buf mapping. This indicates a missing buffer mapping.z&Buffer name mapping is incorrect for 'z'.Expected name_to_buf['z '] to be zbut got z/This indicates some buffers share the same namez Buffer name conflict detected: 'z ' from node z/ is also used as a freeable input buffer name. N)ryrRr debug_str)rgrrmrlrrs r)validate_unique_buffer_namesr5Ks##%C||~H*"jt}}&78@A $+">@AEG5"6xj T]]_L]^EF+&r(ch[X5n[X5 [XX5 [XU5upgXe4$)a Prepare planning info. As nodes are scheduled one at a time, these help keep track of when a buffer can be freed, and when a node can be scheduled Returns: int: peak memory estimation dict[str, FreeableInputBuffer]: name to freeable input buffer )rorrr)rgrrrhrrmestimated_peak_memoryrs r)prepare_planning_infor8tsD"8!L5eI3 ; 4 =  ! <.sammr(r) torch_loginfor6r8r reorder_for_peak_memory_debugexport_graph_for_simulatorr2r5r exceptionrrrrrr" Exceptionr rrrr)rgrrrhrmethodsr7rmpeak_memory_diff_methodsrrrrelem best_results r)reorder_for_peak_memoryrLs"NN;SZH8M  95++"  &     u%$U9ST8:##zBNN-/DE ..{u u:U+ ++1=NK % + + V__E  NN/+ N*  >VW>VdKK!1!11>VW .4KLK   [ ?@{{ :     :FOO L;; Xs+E<&B F2? G4n U R 5[U R55S .nU R U5 M@ U US .nS S K nS S KnS S KnS SKJn UR$R'U"55S S-mUR)USS9mUR*R-SU4SjU4SjS9 g s snn fs snfs sn f)z This is for debugging purposes. It will dump a json file that records graph information. The graph can then be used in a simulator: https://fburl.com/code/3l3d3qi4 cf\rSrSr%S\S'S\S'S\S'S\S'S\S 'S\S 'S \S 'S \S 'Srg)-export_graph_for_simulator..ORMBufferirrMrr-r.rGris_input is_output list[str]deps unmet_depsr Nr!r r(r) ORMBufferrOs+  r(rUc*\rSrSr%S\S'S\S'Srg)+export_graph_for_simulator..ORMNodeirrMrR buffer_namesr Nr!r r(r)ORMNoderWs  r(rYc*\rSrSr%S\S'S\S'Srg),export_graph_for_simulator..ORMGraphiz list[ORMNode]rgzlist[ORMBuffer]buffersr Nr!r r(r)ORMGraphr[s  r(r]T)rMr-r.rGrPrQrSrTNF)rMrX)rgr\r)get_graph_being_compiled_fusedr%)indentartifactc>TSS.$)Nstring)rMencodingr rQsr)r9,export_graph_for_simulator..Os  r(c>T$rPr )g_strsr)r9reSs5r() metadata_fn payload_fn)rrrNr.rryrR defining_oprrHr-rr"jsonosr;functorch.compiler^r.splitextdumps_loggingtrace_structured)rgrmrrhrrUrYr] orm_buffers orm_nodesrrorm_buf_input_bufferrlrrr~rrSorm_buf_scheduler_bufferorm_nodegrkrlr;r^rgrMs @@r)rErEsI ) !9!$&K!I :??A#..88"--77((22!2 +  /0 B)./(-9I9I9K# 9K/ +002)  ( /%%..0h||$ $    $  #..99"--77((22!2)-)-X1M /   34+ 30MMO !6!6!89  " A  : 77  46 7 :X ED JJqJ #E NN## ! $g/  s,1I?8J J *J )rgrrhOrderedSet[str]r?dict[str, FreeableInputBuffer])rdict[str, SchedulerBuffer]r?zdict[str, tuple[int, int]])rgrrrzr?r@) rgrrdict[str, BaseSchedulerNode]rrzrmryr?r@)rgrrmryrrxr?z{tuple[list[BufferInfo], dict[BaseSchedulerNode, int], dict[Union[FreeableInputBuffer, SchedulerBuffer], BaseSchedulerNode]])rgrrmryrrxr?ztuple[int, list[int]])rgrrmryrrxr?ztuple[int, list[tuple[int, int]], dict[BaseSchedulerNode, SNodeMemory], dict[Union[FreeableInputBuffer, SchedulerBuffer], BaseSchedulerNode]]) rgrrmryrrzrrxr?r)rgrr?r)rgrr?r@)rgrrrzrmryr?r@) rgrrrzrr{rhrxrrxr?z*tuple[int, dict[str, FreeableInputBuffer]])rgrrrzrr{rhrxrrxrHz,list[Callable[..., list[BaseSchedulerNode]]]r?r) rgrrmryrr{rhrxrrxr?r@)= __future__rr_rAr loggingtypingrrrrr;torch._environmentrtorch._utils_internalr torch.utils._ordered_setr r rr rutilsrr virtualizedrcollections.abcr dependenciesrrurrr getLoggerr"rBr rr+rDrKrorrrrrrrrrrr#r2r5r8rLrEr r(r)rsx" << (0/-9(!=!   h '       (        .& ".&!.&$.&bU+UUp. ". +.  . b: ": 4: ,: !? :  : | V> "V> >V>#V> V>r#+ "#+ >#+##+ #+L  : ": >:#: :zL "L >L,L# L  L^EP'T+\& "&+&!?& &R= "=+=5=" = # = 0 =H = V "V+V5V" V # V : VVrj "j >j5j" j # j  jr(