bCiZ!SSKrSSKJrJr SSKJr SSKJr SSKrSSK J r SSK J r \ R"S5rS \\R \\\44S jr"S S \5r\"S S55r\"SS55rg)N) dataclassfield)Enum)Optional)logging)tracedContinuousBatchingLoggerreturnc`[RR5(a[R"S5n[RR 5 [RR 5 [RR U5Rn[RRU5n[RRU5nO[RRR5(a[RRR5(aX[R"S5n[RR5nU[RR5- nSnO[R"S5nSnSnSnXX#4$)Ncudampsrcpu)torchr is_availabledevice empty_cache synchronizeget_device_properties total_memorymemory_reservedmemory_allocatedbackendsris_builtdriver_allocated_memoryrecommended_max_memory)rrreserved_memoryallocated_memorys n/home/james-whalen/.local/lib/python3.13/site-packages/transformers/generation/continuous_batching/requests.pyget_device_and_memory_breakdownr s zz  f%    zz77?LL **44V< ::66v>    ( ( * *u~~/A/A/J/J/L/Le$yy88: '%))*J*J*LLe$   BBc4\rSrSrSrSrSrSrSrSr Sr S r S r g ) RequestStatus5z5Status of a generation request through its lifecycle.pending prefillingprefilling_splitsplit_pending_remainderdecodingfinishedfailedN) __name__ __module__ __qualname____firstlineno____doc__PENDING PREFILLINGPREFILLING_SPLITSPLIT_PENDING_REMAINDERDECODINGFINISHEDFAILED__static_attributes__r,r!rr#r#5s*?GJ)7HH Fr!r#c\rSrSr%Sr\\S'\"\S9r \\ \S'\"\S9r \\ \S'\"\S9r \\ \S'Sr\\\S '\R"r\\S '\"\R&S9r\ \S 'S rg) GenerationOutputAaTracks the output of a generation request. Attributes: request_id (str): The ID of the generation request. prompt_ids (list[int]): The IDs of the prompt tokens. generated_tokens (list[int]): The generated tokens. logprobs (list[float]): The log probabilities of the generated tokens. error (Optional[str]): Any error message associated with the request. When None, the request was successful. status (RequestStatus): The status of the request. created_time (float): The time the request was created. request_iddefault_factory prompt_idsgenerated_tokenslogprobsNerrorstatus created_timer,)r-r.r/r0r1str__annotations__rlistr@intrArBfloatrCrr#r2rDtimerEr9r,r!rr;r;Asy O!$7JS 7"'"=d3i=!$7Hd5k7E8C=)11FM1 :L%:r!r;c\rSrSr%Sr\\S'Sr\\ \ \S'Sr \\ \ \S'\ "\ S9r \ \ \S'\ "\ S9r\ \ \S 'S r\ \S 'S r\ \S '\R$r\\S 'Sr\ \S'Sr\ \S'\ "\R,S9r\\S'Sr\\\S'Sr\\\4\S'\S\4Sj5r\R<S\4Sj5rSrS\ 4Sjr S\ 4Sjr!\"S\ S\#4Sj5r$Sr%S r&S!r'g)" RequestStateXa=Tracks the state of a generation request through its lifecycle. Attributes: request_id (str): The ID of the generation request. full_prompt_ids (list[int] | None): The tokens IDs of the full prompt. prompt_ids (list[int] | None): The tokens IDs currently being processed. remaining_prompt_ids (list[int]): The tokens IDs remaining to be processed (for split requests). static_outputs (list[int]): The generated tokens. allocated_blocks (int): The number of blocks allocated to the request. position_offset (int): The current position in the sequence for position_ids. status (RequestStatus): The status of the request: can be one of PENDING, PREFILLING, PREFILLING_SPLIT, SPLIT_PENDING_REMAINDER, DECODING, FINISHED, FAILED max_new_tokens (int): The maximum number of new tokens to generate. eos_token_id (int): The ID of the end-of-sequence token. created_time (float): The time the request was created. error (Optional[str]): Any error message associated with the request. When None, has had no error yet. r=Nfull_prompt_idsr@r>remaining_prompt_idsstatic_outputsrallocated_blocksposition_offset_statusmax_new_tokens eos_token_idrErC)rWrWlifespanr cUR$)N)rTselfs rrDRequestState.status{s ||r!valuec UR[R:Xa[R"5S4UlOMU[R :Xa9URS[R"54UlUR 5 Xlg)NrWr)rTr#r2rKrYr7log_end_of_request)r\r^s rrDr]s` <<=00 0!YY["-DM m,, ,!]]1-tyy{;DM  # # % r!c "[UR5nUR5nURSUR- nURSUR- n[ R SURSU<SU<SU<SU<3 5 g)NrzRequest z finished: prefill_len = z decode_len = z start_time = z end_time = )lenrO generated_lenrYrEloggerinfor=)r\ prefill_len decode_len start_timeend_times rr`RequestState.log_end_of_requests$../ '') ]]1%(9(99 ==#d&7&77 t''A;2B/J?RaT^Sbbodlcp q r!cUR$)zCGet the current length of the sequence (prompt + generated tokens).)rSr[s r current_lenRequestState.current_lens###r!c,[UR5$)z*Get the number of tokens generated so far.)rcrQr[s rrdRequestState.generated_lens4&&''r!token_idc`UR[R:wagXR:H=(a URS:gnUR 5UR :nU(aU(aUR RU/5 U(dU(a[RUlgg)zUpdate the request with a newly generated token and check for completion. Args: token_id: The token ID to add to the output sequence Returns: bool: True if the request is now complete, False otherwise FrWT) rDr#r6rXrdrVrQextendr7)r\rqis_eos is_max_lens rupdate_with_tokenRequestState.update_with_tokens ;;-00 0...J43D3D3J'')T-@-@@ 6    & &z 2 Z'00DKr!c nSUR3SUR3SUR53S[UR53S[UR 53SUR 3S[UR53SUR3S UR3/ nS S RU5-S -$) Nz request_id=zstatus=z out_tokens=z query_length=zremaining_tokens=z kv_length=zfull_prompt_length=zallocated_blocks=zgenerated_tokens=zRequestState( z, z )) r=rTrdrcr@rPrSrOrRrQjoin)r\msgs r__repr__RequestState.__repr__s$//* +dll^ $$,,./ 0C01 2D$=$= >? @--. /!#d&:&:";!< = 5 56 7 3 34 5  #W\\#%66>>r!c [URURURUR/UR S9$)z7Convert the request state to a GenerationOutput object.)r=r@rDrArBrC)r;r=rOrDrQrCr[s rto_generation_output!RequestState.to_generation_outputs9++;;!00**   r!)rTrYrD)(r-r.r/r0r1rFrGrOrrHrIr@rrPrQrRrSr#r2rTrVrXrKrErJrCrYtuplepropertyrDsetterr`rmrdr boolrvr{r~r9r,r!rrMrMXsI&O+/OXd3i(/&*Jc#*&+D&A$s)A %d ;NDI;cOS*22G]2NCL# :L%:E8C=$,HeE5L!,   ]]M $S$(s(  #$ 4 ?  r!rM)rK dataclassesrrenumrtypingrr utils.loggingr utils.metricsr getLoggerrerrrIr r#r;rMr,r!rrs ( $#   5 6Cu||S#s/J)KC, D  ;; ;, s s  s r!