^hׅ PSrSSKrSSKrSSKrSSKrSSKJrJrJrJ r J r J r J r J r JrJr SSKJr SSKrSSKJrJr SSKJr SSKJr SSKJr SS KJrJrJrJ r J!r!J"r" \(a SS K#J$r% SS K&J'r' O\r%\r'"S S \5r("SS5r)SS\ S\ \\S4S\ \*\44Sjjr+g)a This contains LLMCachingHandler This exposes two methods: - async_get_cache - async_set_cache This file is a wrapper around caching.py This class is used to handle caching logic specific for LLM API requests (completion / embedding / text_completion / transcription etc) It utilizes the (RedisCache, s3Cache, RedisSemanticCache, QdrantSemanticCache, InMemoryCache, DiskCache) based on what the user has setup In each method it will call the appropriate method from caching.py N) TYPE_CHECKINGAnyAsyncGeneratorCallableDict GeneratorListOptionalTupleUnion) BaseModel) print_verboseverbose_logger)S3Cache)1_assemble_complete_response_from_streaming_chunks)RerankResponse) CallTypes EmbeddingEmbeddingResponse ModelResponseTextCompletionResponseTranscriptionResponse)LoggingCustomStreamWrappercP\rSrSr%SrSr\\\S'Sr \\ \S'Sr \ \S'Sr g) CachingHandlerResponse<a This is the response object for the caching handler. We need to separate embedding cached responses and (completion / text_completion / transcription) cached responses For embeddings there can be a cache hit for some of the inputs in the list and a cache miss for others N cached_resultfinal_embedding_cached_responseF embedding_all_elements_cache_hit)__name__ __module__ __qualname____firstlineno____doc__rr r__annotations__r rr!bool__static_attributes__r"Y/home/james-whalen/.local/lib/python3.13/site-packages/litellm/caching/caching_handler.pyrr<s7 $(M8C='CG#X.?%@G %dr+rc\rSrSrS\S\\\4S\R4Sjr S)S\S\S\ S\RS \S \\\4S \ \ \S 4S \ 4SjjrS)S\S\S\ S\RS \S \\\4S \ \ \S 4S \ 4SjjrS\ \S\\ \\\4S \\\4S\ S\RS\S \ \ \\44SjrS\ S\S\RS\RS \4 SjrS\ S\S\RS\RS\4 SjrS \S \\\4S \ \S 4S \ \4SjrS)S\S \S \\\4S\ S\S \ \S 4S\ \S \ \\\\\\\44SjjrS\S \S\ S\S \4 SjrS)S\S\S \\\4S \ \ \S 44SjjrS)S\S \\\4S \ \ \S 44Sjjr S\S \\\4S \4S jr!S\S \4S!jr"S"\4S#jr#S"\4S$jr$S*S\ S\S \\\4S\S%\S&\4 S'jjr%S(r&g)+LLMCachingHandlerJoriginal_functionrequest_kwargs start_timecD/Ul/UlX lXlX0lgN)async_streaming_chunkssync_streaming_chunksr1r0r2)selfr0r1r2s r,__init__LLMCachingHandler.__init__Ks( <>#:<",!2$ r+Nmodel logging_obj call_typekwargsargs.returnc # SSKJn U=(d SnSn Sn Sn URSS5c[RcURSS5SLGaURS05RS S5SLGa`[RGbNUR US 9(Ga9[ R"S 5 URUUUS 9IShvN n U Gbq[U [5(Gd[[ R"S 5 Sn [RR5n [R"UURSS5URSS5URSS5S9un nURUUUU SS9 URnUR!U UUUUURSS5US9n URSS5SLaUR#UU UU U S9 [RR$"S0UD6n[U [&5(d[X5(a[)U S5(aXR*S'[-U S9$U[.R0R2:XawU bt[U [5(a_[RbN[[RR[45(d!UR7U U UUUUS9un n [-U U S9$[ R"SU 35 [-U U S9$GN27f)a Internal method to get from the cache. Handles different call types (embeddings, chat/completions, text_completion, transcription) and accordingly returns the cached response Args: model: str: original_function: Callable: logging_obj: LiteLLMLoggingObj: start_time: datetime.datetime: call_type: str: kwargs: Dict[str, Any]: args: Optional[Tuple[Any, ...]] = None: Returns: CachingHandlerResponse: Raises: None rrr"NFcachingTcachezno-cacher0Checking Cache)r<r=r>z Cache Hit!custom_llm_providerapi_baseapi_keyr:rErFrGr;r:r=ris_asyncrr<r=r;r:rEr>streamr;rr2end_time cache_hit_hidden_params cache_keyr)r rr=r;r2r:)r r!zCACHE RESULT: )rr ) litellm.utilsrgetlitellmrB _is_call_type_supported_by_cacherdebug_retrieve_from_cache isinstancelistdatetimenowget_llm_provider'_update_litellm_logging_obj_environmentr#(_convert_cached_result_to_model_response!_async_log_cache_hit_on_callbacks!_get_preset_cache_key_from_kwargsr hasattrrPrr aembeddingvaluer(_process_async_embedding_cached_response)r7r:r0r;r2r<r=r>rr r!rrOrN_rQs r,_async_get_cache"LLMCachingHandler._async_get_cacheXs < 6zrGK'16('+ ZZ 4 ( 0W]]5Nzz)U+t3 JJw # ' ' E :$ F}}(T-R-R"3.S.$$%56&*&?&?'!'@'! !,Z t5T5T"((6 $I'00446H%,%=%=#,2JJ7Ld,S!'J!= & 9d ; &NE1a @@$/#%&3!% A!2 : :I$($Q$Q&3"+%$/#,2JJ7Ld,S!%R%Mzz(E2e;>>(3*7'1%-&/ ?!( O O! !I#=)<<%mII!-1ABBDM44[A1 NN!5!5!;!;;%1"=$77 1& ++WEE8W&3%$/#-# F7828W9Y ~m_=>%',K  ]!sC L K= H2Lc  SSKJn U=(d SnUR5n U R[ UR U55 Sn [ RGbURUS9(Ga[S5 [ RR"S0U D6n U Gb^SU ;aGOVURnURU UUUUURSS5US9n S n [RR5n [ R "U=(d S URSS5URS S5URS S5S 9unn nnUR#UUUU SS9 [$R&"UR(XX4S9R+5 [ RR,"S0UD6n[/U [05(d[/X5(a [3U S5(aUU R4S'[7U S9$[7U S9$)Nrrr"rCrDdetailrErKTrFrGrHFrItargetr>rPrQrR)rSrcopyupdateconvert_args_to_kwargsr0rUrBrVr get_cacher#r_rTr[r\r]r^ threadingThreadsuccess_handlerstartrarYr rbrPr)r7r:r0r;r2r<r=r>r new_kwargsrrOrNrEdynamic_api_keyrFrQs r,_sync_get_cache!LLMCachingHandler._sync_get_caches 6zr[[]  "&&  (, == $)N)N/*O*  * +#MM33AjAM(}, 1 : :I$($Q$Q&3"+%$/#,2JJ7Ld,S!%R%M!%I'00446H  00#kr,2JJ7Ld,S!'J!= & 9d ;  +' @@$/#%&3!& A$$*::+Meg ' O O! !I#=)<<%mII!-1ABBDM 44[A1 NN%MBBr+r rc Sn/n/n [U5H3upU cURUSU 5 M!U RX45 M5 USn XS'[U 5S:ay[S[U 535 [ UR S5S/[U 5-S9nSUR S 'U H'n U upU cM [U S U S S 9URU 'M) [U5S:XaSnSn[RR5n[R"UUR S S5UR S S5UR SS5S9unnnnURUUUUSSS9 URUUUUUS9 X4$X4$)a1 Returns the final embedding cached response and a boolean indicating if all elements in the list have a cache hit For embedding responses, there can be a cache hit for some of the inputs in the list and a cache miss for others This function processes the cached embedding responses and returns the final embedding cached response and a boolean indicating if all elements in the list have a cache hit Args: final_embedding_cached_response: Optional[EmbeddingResponse]: cached_result: List[Optional[Dict[str, Any]]]: kwargs: Dict[str, Any]: logging_obj: LiteLLMLoggingObj: start_time: datetime.datetime: model: str: Returns: Tuple[Optional[EmbeddingResponse], bool]: Returns the final embedding cached response and a boolean indicating if all elements in the list have a cache hit FNinputrzEMBEDDING CACHE HIT! - r:)r:dataTrO embedding)r}indexobjectrErFrGrH)r;r:r=rrJ is_embeddingrM) enumerateappendlenrrrTrPrr|r[r\rUr]r^r`)r7r rr=r;r2r:r!remaining_list non_null_listidxcroriginal_kwargs_inputvalrOrNrErwrFs r,re:LLMCachingHandler._process_async_embedding_cached_response*s:27(  /GCz%%fWoc&:;$$cY/ 0 !'w(w }  ! 3C 4F3GH I.?jj)Vc"788/ +KO + : :; G$>@I"$[/!*A388=% ~ ! #I/3 ,((,,.H (($*JJ/Dd$KJ5 9d3   #  8 8'=! 9   2 2'=%!# 3 3T T.PPr+_caching_handler_responseembedding_responserNcURcU$Sn/nURRHInUc2URb%URURU5 US- nM8URU5 MK XaRlSURRS'XC- R 5S-URlUR$)a Combines the cached embedding response with the API EmbeddingResponse For caching there can be a cache hit for some of the inputs in the list and a cache miss for others This function combines the cached embedding response with the API EmbeddingResponse Args: caching_handler_response: CachingHandlerResponse: embedding_response: EmbeddingResponse: Returns: EmbeddingResponse: rTrOi)r r|rrP total_seconds _response_ms)r7rrr2rNrfinal_data_listitems r,2_combine_cached_embedding_response_with_api_resultDLLMCachingHandler._combine_cached_embedding_response_with_api_results( % D D L% %-MMRRD| 2 7 7 C&&'9'>'>s'CDq&&t, SJYAAF  "AAPP    ! -/DR!!AAN)HHHr+rOc[R"URX#XE55 [R"UR X#XE4S9R 5 g)aP Helper function to log the success of a cached result on callbacks Args: logging_obj (LiteLLMLoggingObj): The logging object. cached_result: The cached result. start_time (datetime): The start time of the operation. end_time (datetime): The end time of the operation. cache_hit (bool): Whether it was a cache hit. rlN)asyncio create_taskasync_success_handlerrrrsrtru)r7r;rr2rNrOs r,r`3LLMCachingHandler._async_log_cache_hit_on_callbackssO$   - -8  ..XA  %'r+c\# [RcgUR5nUR[ UR U55 SnU[ RR:Xa[US[5(a/n[US5HWupx[RR"S00UESU0ED6n URRU S95 MY [R "U6IShvN nUb.[U[5(a[#SU55(aSnU$[RR%5SLa*[RR"S0UD6IShvN nU$[RR&"S0UD6nU$NN*7f)a- Internal method to - get cache key - check what type of cache is used - Redis, RedisSemantic, Qdrant, S3 - async get cache value - return the cached value Args: call_type: str: kwargs: Dict[str, Any]: args: Optional[Tuple[Any, ...]] = None: Returns: Optional[Any]: Raises: None Nr{)rQc3(# UHoSLv M g7fr4r").0results r, 9LLMCachingHandler._retrieve_from_cache..sBM&~MsTr")rUrBrnrorpr0rrcrdrYrZr get_cache_keyrasync_get_cachergatherall_supports_asyncrq) r7r<r=r>rvrtasksripreset_cache_keys r,rX&LLMCachingHandler._retrieve_from_caches( == [[]  "&&  (,  ,,22 2z w 8 8 E#Jw$78#*==#>#>$00Wa0$  W]]::EU:VW 9 #*..%"88M(Z t-L-LBMBBB$(M  }},,.$6&-mm&C&C&Qj&Q Q !( 7 7 E* E 9!Rs%DF,F(A;F,?F*)F,*F,rEcSSKJn U[RR:XdU[R R:XaL[ U[5(a7URSS5SLaURUUUUS9nOU"U[5S9nU[RR:XdU[RR:XaK[ U[5(a6URSS5SLaURUUUUS9nGO1[S0UD6nGO$U[RR:XdU[RR:Xa&[ U[5(aU"U[!5SS 9nOU[R"R:XdU[R$R:Xa[ U[5(a U"US S S 9nOhU[R&R:XdU[R(R:Xa,[ U[5(aS USS .n U"U[+5SU S9n[-US5(a;UR.b.[ UR.[5(aSUR.S'U$)a Internal method to process the cached result Checks the call type and converts the cached result to the appropriate model response object example if call type is text_completion -> returns TextCompletionResponse object Args: cached_result: Any: call_type: str: kwargs: Dict[str, Any]: logging_obj: LiteLLMLoggingObj: model: str: custom_llm_provider: Optional[str] = None: args: Optional[Tuple[Any, ...]] = None: Returns: Optional[Any]: r) convert_to_model_response_objectrLFT)rr<r;r:)response_objectmodel_response_objectr})rr response_typeNrerankz whisper-1)r:rErOaudio_transcription)rrr hidden_paramsrPrOr")rSrr acompletionrd completionrYdictrT_convert_cached_stream_responseratext_completiontext_completionrrcr}rarerankratranscription transcriptionrrbrP) r7rr<r=r;r:r>rErrs r,r_:LLMCachingHandler._convert_cached_result_to_model_responses@J C ..44 4I00666--zz(E*d2 $ D D"/' + !E! !A$1*7/! 3399 9I55;;;--zz(E*d2 $ D D"/' + !E! !7 G G --33 3I//555--< -&7&9)M **00 0IAQAQAWAW4W--< -&*&M 1177 7I33999--$':!M = -&;&=3+ M M#3 4 4,,8=77>>8[ R""[RR$U4US9R'5 g[R"[RR("UR+540UD65 g[R"[RR("U40UD65 gg7f)a Internal method to check the type of the result & cache used and adds the result to the cache accordingly Args: result: Any: original_function: Callable: kwargs: Dict[str, Any]: args: Optional[Tuple[Any, ...]] = None: Returns: None Raises: None Nr0r=r{)rmr>r=)rnrorprUrB_should_store_result_in_cacherYrrrrrZrrrasync_add_cache_pipelinerrrs add_cacheruasync_add_cachemodel_dump_json)r7rr0r=r>rvs r,async_set_cache!LLMCachingHandler.async_set_caches,[[]  "!  ==   - -/ . 67#8#899fg&?&?@@f&;<<fn55v'899":g#6== 1& ++W'' >>vTT  3 3W==$$&}}66$Y)eg'' 55"2248B ##GMM$A$A&$WJ$WXC sIIcUR5nUR[URU55 [R cgUR URUS9(a![R R"U40UD6 g)z5 Sync internal method to add the result to the cache Nr)rnrorpr0rUrBrr)r7rr=r>rvs r,sync_set_cache LLMCachingHandler.sync_set_cachesz[[]  "&&  ==   - -"44Z .  MM # #F 9j 9r+c0[RSL=(a~ [RRSL=(a[ [UR5[RR;=(a$ UR S05R SS5SL$)z Helper function to determine if the result should be stored in the cache. Returns: bool: True if the result should be stored in the cache, False otherwise. NrBzno-storeFT)rUrBsupported_call_typesstrr#rT)r7r0r=s r,r/LLMCachingHandler._should_store_result_in_cachesy]]$ & M 22$> M&//0GMM4V4VV MGR(,,Z?tK  r+c[RbM[RRb2[UR5[RR;agg)a Helper function to determine if the call type is supported by the cache. call types are acompletion, aembedding, atext_completion, atranscription, arerank Defined on `litellm.types.utils.CallTypes` Returns: bool: True if the call type is supported by the cache, False otherwise. TF)rUrBrrr#)r7r0s r,rV2LLMCachingHandler._is_call_type_supported_by_cachesB MM % 22>%../7==3U3UUr+processed_chunkc # [UUR[RR5URUR SS9nUb.UR UURURS9IShvN ggN7f)z Internal method to add the streaming response to the cache - If 'streaming_chunk' has a 'finish_reason' then assemble a litellm.ModelResponse object - Else append the chunk to self.async_streaming_chunks Trr2rNr1streaming_chunksrJN)rr0r=)rr2r[r\r1r5rr0r7rcomplete_streaming_responses r, _add_streaming_response_to_cache2LLMCachingHandler._add_streaming_response_to_caches >"&&**,..!88   $ ' 2&&2"&"8"8**'   3 sA4A?6A=7A?c [UUR[RR5URUR SS9nUbUR UURS9 gg)zA Sync internal method to add the streaming response to the cache FrN)rr=)rr2r[r\r1r6rrs r,%_sync_add_streaming_response_to_cache7LLMCachingHandler._sync_add_streaming_response_to_cache+sn >"&&**,..!77   $ ' 2   2**   3r+rJrc>URSS5UURSS5URS05URS05URSS5URS05S .n[Rb$[RR"S0UD6US 'OSUS 'UR UURS S50UU(dURS S5OURS S5URSS5[ U5SURSS5S9 g)a Helper function to update the LiteLLMLoggingObj environment variables. Args: logging_obj (LiteLLMLoggingObj): The logging object to update. model (str): The model being used. kwargs (Dict[str, Any]): The keyword arguments from the original function call. cached_result (Any): The cached result to log. is_async (bool): Whether the call is asynchronous or not. is_embedding (bool): Whether the call is for embeddings or not. Returns: None logger_fnNrFrkmetadata model_infoproxy_server_requeststream_response)rrrFrrrrrusermessagesr{rGrLF) r:roptional_paramslitellm_paramsr{rGoriginal_responseadditional_argsrLr")rTrUrBraupdate_environment_variablesr)r7r;r:r=rrJrrs r,r^9LLMCachingHandler._update_litellm_logging_obj_environmentAs0 K6# :r2 :r2 **\26$*JJ/Et$L%zz*;R@  == $ ??I&I - .26N- .00FD))$ :r*ZZ,JJy$/!-0 ::h. 1 r+)r5r0r1r2r6r4)F)'r#r$r%r&rrrrr[r8LiteLLMLoggingObjr r rrgrxrr r)rerr`rXr rrrrrr_rrrrrVrrr^r*r"r+r,r.r.Js  #  S#X  %%  *+/@ @ $@ ' @ %% @  @ S#X@ uS#X'@  @ T+/NCNC$NC' NC %% NC  NCS#XNCuS#X'NC NC`XQ)12C)DXQHT#s(^45XQS#X XQ ' XQ %% XQXQ x)*D0 1XQt'I#9'I.'I%% 'I ## 'I  'IR&%%  ##   833&*38n3  # ,m8]:#6 &6 6 S#X 6  6  6 6 6 r+r.r0r>.r?c[R"U5n[URR 55n0nU(a-[ U5HupVU[ U5:dMX5nXdU'M U$r4)inspect signaturerZ parameterskeysrr)r0r>r param_namesargs_to_kwargsr~arg param_names r,rprpzso !!"34Iy++0023KN #D/JEs;''(/ -0z** r+r4),r'rr[rrrtypingrrrrrrr r r r pydanticr rUlitellm._loggingrrlitellm.caching.cachingr(litellm.litellm_core_utils.logging_utilsrlitellm.types.rerankrlitellm.types.utilsrrrrrr*litellm.litellm_core_utils.litellm_loggingrrrSrrr.rrpr"r+r,rs    :+0W1 Y m  m  d'+ 5c? # #s(^r+