bCi.SSKJrJr SSKJr SSKJr SSKJr SSK J r "SS\5r "S S \ 5r "S S \ 5r g ))ABCabstractmethod)deque)ceil)Optional)loggerc \rSrSr%Sr\\S'\\\ \4\S'\ S\S\S\ \S\ \4S j5r S\S\ \SS 4S jr\ S\S \S \S\ \4Sj5r\ S\S \S \S\ \4Sj5r\ S\S \S \S\\\44Sj5rSrg )CacheAllocatorzAbstract base class for cache managers. Cache managers keep track of per-request cache allocations, determine when a new physical block needs to be allocated and compute physical indices for reading or writing to the cache._index _block_tablen_blocks request_id free_blocksreturncg)zpAllocates n_blocks for a given request_id. Returns the num of blocks allocated if successful and None otherwise.Nselfrrrs s/home/james-whalen/.local/lib/python3.13/site-packages/transformers/generation/continuous_batching/cache_manager.pyallocate_blocksCacheAllocator.allocate_blockss NcXR;a-URRU5nURU5 g[R"SUR SU35 g)z.Frees all blocks associated with a request_id.zCacheAllocator z7 attempted to free blocks for non-existent request_id: N)rpopextendr warningr )rrrblocks_to_frees rrCacheAllocator.free_blocks$sU ** *!..22:>N   ~ . NN!$++.efpeqr r past_length query_lengthcg)zUReturns the physical indices of where to read request_id's cache in the cache tensor.Nrrrr!r"s rget_read_indicesCacheAllocator.get_read_indices. rcg)zVReturns the physical indices of where to write request_id's cache in the cache tensor.Nrr$s rget_write_indices CacheAllocator.get_write_indices3r'rcg)gReturns the attention type of the cache allocator and the key sequence length for the given request_id.Nrr$s r get_seqlens_kCacheAllocator.get_seqlens_k8r'rr)__name__ __module__ __qualname____firstlineno____doc__int__annotations__dictstrlistrrrrrr%r)tupler-__static_attributes__rrrr r s'y KsDI~&&   5QT: ZbcfZg  cc t 3 S PS X\]`Xa   C c QT Y]^aYb    # S UZ[^`c[cUd  rr c \rSrSrSrS\S\SS4SjrS\S \S \\S\ \4S jr S \S \S \S\ \4Sjr S \S \S \S\ \4Sjr S \S \S \S\\\44SjrSrg)FullAttentionCacheAllocator>z3Cache manager for a group of full attention layers.index block_sizerNc*XlX l0Ulg)zInitializes the cache manager for a group of full attention layers. Args: - index: the index of the associated layer group - block_size: the size of the blocks in the cache N)r r?r)rr>r?s r__init__$FullAttentionCacheAllocator.__init__As  $rrrrc^[T5U:agX R;a/URU'URURU4Sj[U555 U$)zAllocate blocks for a given request_id. Returns the number of blocks allocated if successful and None otherwise. For group of full attention layers, we always allocate the number of requested blocks.Nc3D># UHnTR5v M g7fNpopleft.0_rs r >FullAttentionCacheAllocator.allocate_blocks..Rs,\Oq[-@-@-B-BO )lenrrrangers `rr+FullAttentionCacheAllocator.allocate_blocksKsZ { h & .. .,.D  j ) *%,,,\ERZO,\\rr!r"c URRU5nUc[SU35e/n[X#-5HDnX`R-nX`R-nXGUR-U-n UR U 5 MF U$)zReturns the physical indices of where to read request_id's cache. For a group of full attention layers, we first write the new cache to the cache tensor and then read the entire cache from the beginning to the end.!No block table found for request rget ValueErrorrOr?append rrr!r" block_tablephysical_indicesi block_idx block_offsetphysical_indexs rr%,FullAttentionCacheAllocator.get_read_indicesUs''++J7  @ MN N{12A__,I.L(3dooE TN  # #N 3 3  rcURRU5nUc[SU35e/n[X"U-5HDnX`R-nX`R-nXGUR-U-n UR U 5 MF U$)zReturns the physical indices for writing to the cache. For a group of full attention layers, we write the new cache as a continuation of the existing cache for the same request.rRrSrWs rr)-FullAttentionCacheAllocator.get_write_indiceses''++J7  @ MN N{,$>?A__,I.L(3dooE TN  # #N 3 @  rcX#-nSU4$)r,full_attentionrrrr!r" seqlens_ks rr-)FullAttentionCacheAllocator.get_seqlens_kts. **r)rr r?r/r0r1r2r3r4rAr7rrrr8r%r)r9r-r:rrrr<r<>s=cst5QT:ZbcfZg 3 S PS X\]`Xa  C  c  QT  Y]^aYb  ++#+S+UZ[^`c[cUd+rr<c \rSrSrSrS\S\S\SS4SjrS \S \S \\S\ \4S jr S \S \S\S\ \4Sjr S \S \S\S\ \4Sjr S \S \S\S\\\44SjrSrg)SlidingAttentionCacheAllocatorzz2Cache manager for sliding window attention layers.r>r?sliding_windowrNcXlX lX0l[URUR- 5Ul0Ulg)zInitializes the cache manager for a group of sliding window attention layers. Args: - index: the index of the associated layer group - block_size: the size of the blocks in the cache - sliding_window: the size of the sliding window N)r r?rjr_max_blocks_per_requestr)rr>r?rjs rrA'SlidingAttentionCacheAllocator.__init__}s8 $,'+D,?,?$//,Q'R$rrrrcN^X R;a/URU'[URU5nX@R:Xag[XA-UR5nXT- n[T5U:agURUR U4Sj[ U555 U$)zAllocate blocks for a given request_id. Returns the number of blocks allocated if successful and None otherwise. For group of sliding window attention layers, we only allocate up to the point where we can fit an entire sliding window in the cache tensor.rNc3D># UHnTR5v M g7frErFrHs rrKASlidingAttentionCacheAllocator.allocate_blocks..s,cLbq[-@-@-B-BLbrM)rrNrlminrrO)rrrralready_allocatedafter_allocationactual_n_blockss ` rr.SlidingAttentionCacheAllocator.allocate_blockss .. .,.D  j ) 1 1* =>  < < <0;T=Y=YZ*> { o - *%,,,cERaLb,ccrr!r"cURRU5nUc[SU35eX R:aSO X R-n[ X RS- 5n/n[ XUU-5HRnXR-nXR -n XR -n XIUR -U -n URU 5 MT US/U--$)aReturns the physical indices of where to read request_id's cache in the cache tensor. For a group of sliding window attention layers, we read from the cache tensor before writing on it, because the new cache can overwrite the old one. To form the cache + new key / values states, we read the at most sliding_window - 1 cache page and then manually add the new key / values states after. Hence the -1 indices which indicate where to store the new key or values indices.rRrrrrTrUrjrqrOr?rV) rrr!r"rX start_index cache_lengthrYrZr[r\r]s rr%/SlidingAttentionCacheAllocator.get_read_indicess''++J7  @ MN N&)<)<?A $$ $A__,I.L(3dooE TN  # #N 3 @  2$"555rcURRU5nUc[SU35eX R-n[ X0R5nX6- n/n[ XUU-5HRn XR-n XR -n XR -n XJUR -U -n URU 5 MT US:a S/U-U-nU$)a2Returns the physical indices of where to write request_id's cache in the cache tensor. For a group of sliding window attention layers, we write the new cache in rolling-buffer kind of way: if we reach the end of the allocated physical cache, we start writing from the beginning of the physical cache again.rRrrwrx) rrr!r"rXryrzpadding_lengthrYrZr[r\r]s rr)0SlidingAttentionCacheAllocator.get_write_indicess ''++J7  @ MN N!$7$77 <)<)<= %4{,$>?A $$ $A__,I.L(3dooE TN  # #N 3 @ A  "tn47GG rc@U[X RS- 5-nSU4$)r,rsliding_attention)rqrjrcs rr-,SlidingAttentionCacheAllocator.get_seqlens_ks' 3{4G4G!4K#LL "I--r)rr rlr?rjrfrrrrhrhzs< c s C D 5QT:ZbcfZg&636S6PS6X\]`Xa6. C c QT Y]^aYb 0..#.S.UZ[^`c[cUd.rrhN)abcrr collectionsrmathrtypingrrequestsr r r<rhrrrrs>$$ S$ N9+.9+xU.^U.r