^h("$SSKJrJrJr SSKJrJr SSKrSSKJ r SSK J r SSK J r /SSSSSSSSSSSSSSSS 4S \S \S \\S \\S\\S\\S\\S\\S\\S\\S\\S\\S\\S\\S\\S\\4 SjjrSrSrg))FIRST_COMPLETEDThreadPoolExecutorwait)ListOptionalN) print_verbose)get_optional_params)handleriXdmodelmessages functions function_call temperaturetop_pnstream max_tokenspresence_penaltyfrequency_penalty logit_biasuserrequest_timeouttimeout max_workersc p[5nUn/nUnSnURSS5S[R;a*URSS5SnURSS5SnUS:XaF[ UUUUUU=(d SUU U U U U UUS9n[ R "UU[RUS9nU$S n[US 9nU"US 5HnUHvnUR5nURS 5 UUS '0nSU;aURS5nUR"[R40UDUD6nURU5 Mx M SSS5 /nUH#nURUR55 M% U$!,(df  N;=f![an URU 5 Sn A M[Sn A ff=f)a Batch litellm.completion function for a given model. Args: model (str): The model to use for generating completions. messages (List, optional): List of messages to use as input for generating completions. Defaults to []. functions (List, optional): List of functions to use as input for generating completions. Defaults to []. function_call (str, optional): The function call to use as input for generating completions. Defaults to "". temperature (float, optional): The temperature parameter for generating completions. Defaults to None. top_p (float, optional): The top-p parameter for generating completions. Defaults to None. n (int, optional): The number of completions to generate. Defaults to None. stream (bool, optional): Whether to stream completions or not. Defaults to None. stop (optional): The stop parameter for generating completions. Defaults to None. max_tokens (float, optional): The maximum number of tokens to generate. Defaults to None. presence_penalty (float, optional): The presence penalty for generating completions. Defaults to None. frequency_penalty (float, optional): The frequency penalty for generating completions. Defaults to None. logit_bias (dict, optional): The logit bias for generating completions. Defaults to {}. user (str, optional): The user string for generating completions. Defaults to "". deployment_id (optional): The deployment ID for generating completions. Defaults to None. request_timeout (int, optional): The request timeout for generating completions. Defaults to None. max_workers (int,optional): The maximum number of threads to use for parallel processing. Returns: list: A list of completion results. N/rvllmF)rrrrrrstoprrrrrr custom_llm_provider)r rcustom_prompt_dictoptional_paramsc3V# [S[U5U5H nXX!-v M g7f)z)Yield successive n-sized chunks from lst.rN)rangelen)lstris W/home/james-whalen/.local/lib/python3.13/site-packages/litellm/batch_completion/main.pychunks batch_completion..chunks_s(1c#h*aen$+s')rr rrkwargs)localssplitlitellm provider_listr vllm_handlerbatch_completionsr#rcopypopsubmit completionappendresult Exception)!r rrrrrrrr!rrrrr deployment_idrrrr.argsbatch_messages completionsr"r$resultsr+executor sub_batch message_listkwargs_modifiedoriginal_kwargsfutureexcs! r*batch_completionrH s` 8DNK E {{31!6!66#kk#q1!4 C#A&f$-'#?U!-/! 3 "00#&99+  L N; %  K 8H#NC8 $-L&*iikO#'' 62>OJ/&(O?2*9*=*=h*G%__**.=APF &&v.%.99"!F $v}}/" N19 8* $s## $s%F> F  F5F00F5c8^SU;aURS5 SU;aUSmURS5 0n[[T5S9nTH+nUR"[R /UQ7SU0UD6X$'M- [ UR5U4SjS9H1upEUR5cMUR5s sSSS5 $ SSS5 gSU;Ga2USnURS5 URS5 URS 05n0n[[U5S9nUHSnUR5Hn X;dM XX'M 0UEUEnUR"[R 40UD6X(S'MU U(a{[S 5 [UR5[S 9up[S U 35 U HnUR5n U s sSSS5 $ [S5 [U5 U(aM{SSS5 gg!,(df  g=f![a^ [S 5 [U5 0n UR5HupX^:Xa[SU 35 MXU 'M U n[SU35 Mf=f!,(df  g=f)a Send a request to multiple language models concurrently and return the response as soon as one of the models responds. Args: *args: Variable-length positional arguments passed to the completion function. **kwargs: Additional keyword arguments: - models (str or list of str): The language models to send requests to. - Other keyword arguments to be passed to the completion function. Returns: str or None: The response from one of the language models, or None if no response is received. Note: This function utilizes a ThreadPoolExecutor to parallelize requests to multiple models. It sends requests concurrently and returns the response from the first model that responds. r modelsr-c,>TRUS5$)Nr)index)xrJs r*)batch_completion_models..sv||AaD/A)keyN deployments model_listr.z waiting for next result ) return_whenz done list z3 got an exception, ignoring, removing from futuresz removing keyz new futuresz done looping through futures )r6rr'r7r1r8sorteditemsr:keysrrvaluesrr;)r=r.futuresrAr rFrR nested_kwargs deploymentrQdone_r: new_futuresvaluerJs @r*batch_completion_modelsr`s&& 76! 8 CK 8H!)&&")-"5:">D" "( %A" ==?.!==?*9 8 " 9t _ & ]+  =! <  8R0  C ,< =) !;;=C-*0+  ) 9J8-8/7&&0*007+,*@Aw~~/_M D623"F!!'% )> ="#*DEg&7'>R u9 8t )%!%S&g.&( *1--/JC% - SE.B C (38C 0 +: #.% G9&=> !+> =R sW A"H/H H)J  A=J H J $J  H A$JJ JJ  JcSSKnSU;aURS5 SU;aUSnURS5 O [S5e/nURR [ U5S9n[ U5H]upgUR"[R/UQ7SU0UD6nUR5cM>URUR55 M_ SSS5 U$!,(df  U$=f)a Send a request to multiple language models concurrently and return a list of responses from all models that respond. Args: *args: Variable-length positional arguments passed to the completion function. **kwargs: Additional keyword arguments: - models (str or list of str): The language models to send requests to. - Other keyword arguments to be passed to the completion function. Returns: list: A list of responses from the language models that responded. Note: This function utilizes a ThreadPoolExecutor to parallelize requests to multiple models. It sends requests concurrently and collects responses from all models that respond. rNr rJz'models' param not in kwargsr-) concurrent.futuresr6r;rYrr' enumerater7r1r8r:r9) r=r. concurrentrJ responsesrAidxr rFs r*%batch_completion_models_all_responsesrgs$& 76! 8677I    . .3v; . G8#F+JC__W%7%7V$VeVvVF}}*  1, H  H G s(AC2#C C.)rbrrrtypingrrr1litellm._loggingr litellm.utilsr llms.vllm.completionr r3strfloatintbooldictrHr`rgrPr*rrsHHH!*-:  $#'#'!! $(,)-!%%) !$'q qq~ q C= q % q E?q}q TNq quoq qq 3-q"c]#q$c]%q&#'qlSl&rP