^h BSSKrSSKrSSKrSSKrSSKrSSKJr SSKJrJ r J r J r J r J r SSKJr SSKJrJr SSKJr SSKJr SSKJr SS KJr SSKrSS KJr SS K7 SS KJr SS K J!r! SSK"J#r# SSK$J%r% SSK&J'r' SSK(J)r) SSK*J+r+ SSK,J-r- SSK.J/r/ SSK0J1r1 Sr2"SS\3\5r4"SS5r5\6RnSSS/SQ4S\ \6S\ \3S\ \3S \ \3S!\ \ \84 S"jjr9\6RnSSS/SQ4S\ \6S\ \3S\ \3S \ \3S!\ \ \84 S#jjr:S$r;g)%N)Enum)AnyDictListOptionalSetUnion)TranscriptionCreateParams)"CompletionCreateParamsNonStreamingCompletionCreateParamsStreaming)r )r )EmbeddingCreateParams) BaseModel)verbose_logger)*) RerankRequest)all_litellm_params) BaseCache) DiskCache) DualCache) InMemoryCache)QdrantSemanticCache) RedisCache)RedisSemanticCache)S3Cachec[R"U5 [R(a [ U5 gg![ a gf=f)N)rdebuglitellm set_verboseprint Exception)print_statements Q/home/james-whalen/.local/lib/python3.13/site-packages/litellm/caching/caching.py print_verboser$0s= _-    / "     s6: AAc\rSrSrSrSrSrg) CacheMode9 default_on default_offN)__name__ __module__ __qualname____firstlineno__r(r)__static_attributes__r*r#r&r&9s JKr0r&c:z\rSrSr\R \RSSSSSSSS/SQSSSSSSSSSSSSSSSSSSSSS4 S\\S\\S \\ S \\ S \\ S \\ S \\ S\\ S\\ S\\ S\\ \ S\\ S\\ S\\ S\\ S\\\ \ 4S\\ S\\ S\\ S\\ S\\S\\ S\\S\\ S\\ S \\ S!\\ S"\\ 48S#jjrS$\ 4S%jrS&\ S'\S$\\ 4S(jrS'\S$\ 4S)jrS*\S+\\ S$\\ 4S,jrS'\S$\ 4S-jrS$\\ 4S.jrS/\ S$S4S0jrS$\\ 4S1jrS$\\ 4S2jrS$\\ 4S3jrS$\\ 4S4jrS$\\ 4S5jrS$\\ 4S6jr S$\\ 4S7jr!\"S8\ S$\ 4S9j5r#S:\ S$\ 4S;jr$S<r%S=\\S>\\ 4S?jr&S@r'SAr(SBr)SCr*SDr+SEr,SFr-SGr.SHr/SIr0SJr1S$\ 4SKjr2SLr3g)MCache?N completion acompletion embedding aembeddingatranscription transcriptionatext_completiontext_completionarerankrerankTFztext-embedding-ada-002typemodehostportpassword namespacettldefault_in_memory_ttldefault_in_redis_ttlsimilarity_thresholdsupported_call_typess3_bucket_names3_region_names3_api_version s3_use_ssl s3_verifys3_endpoint_urls3_aws_access_key_ids3_aws_secret_access_keys3_aws_session_token s3_configs3_pathredis_flush_sizeredis_startup_nodesqdrant_api_baseqdrant_api_keyqdrant_collection_nameqdrant_quantization_configc! U[R:Xa[SUUUUUS.U!D6UlOU[R:Xa[ SUUUU UUS.U!D6UlOU[R :Xa[UUUU UU S9UlOwU[R:Xa[5UlOSU[R:Xa[SU U UUUUUUUUUS. U!D6UlO"U[R:Xa[US9UlS[R;a[RR!S5 S[R";a[R"R!S5 S[R$;a[R$R!S5 XlXlX`lUUlXplU=(d [0R2UlUR([R:Xa UbXlUR([R:XdUR([R:Xa U bXlUR*b<[7UR[5(aUR*URlggg) a Initializes the cache based on the given type. Args: type (str, optional): The type of cache to initialize. Can be "local", "redis", "redis-semantic", "qdrant-semantic", "s3" or "disk". Defaults to "local". # Redis Cache Args host (str, optional): The host address for the Redis cache. Required if type is "redis". port (int, optional): The port number for the Redis cache. Required if type is "redis". password (str, optional): The password for the Redis cache. Required if type is "redis". namespace (str, optional): The namespace for the Redis cache. Required if type is "redis". ttl (float, optional): The ttl for the Redis cache redis_flush_size (int, optional): The number of keys to flush at a time. Defaults to 1000. Only used if batch redis set caching is used. redis_startup_nodes (list, optional): The list of startup nodes for the Redis cache. Defaults to None. # Qdrant Cache Args qdrant_api_base (str, optional): The url for your qdrant cluster. Required if type is "qdrant-semantic". qdrant_api_key (str, optional): The api_key for the local or cloud qdrant cluster. qdrant_collection_name (str, optional): The name for your qdrant collection. Required if type is "qdrant-semantic". similarity_threshold (float, optional): The similarity threshold for semantic-caching, Required if type is "redis-semantic" or "qdrant-semantic". # Disk Cache Args disk_cache_dir (str, optional): The directory for the disk cache. Defaults to None. # S3 Cache Args s3_bucket_name (str, optional): The bucket name for the s3 cache. Defaults to None. s3_region_name (str, optional): The region name for the s3 cache. Defaults to None. s3_api_version (str, optional): The api version for the s3 cache. Defaults to None. s3_use_ssl (bool, optional): The use ssl for the s3 cache. Defaults to True. s3_verify (bool, optional): The verify for the s3 cache. Defaults to None. s3_endpoint_url (str, optional): The endpoint url for the s3 cache. Defaults to None. s3_aws_access_key_id (str, optional): The aws access key id for the s3 cache. Defaults to None. s3_aws_secret_access_key (str, optional): The aws secret access key for the s3 cache. Defaults to None. s3_aws_session_token (str, optional): The aws session token for the s3 cache. Defaults to None. s3_config (dict, optional): The config for the s3 cache. Defaults to None. # Common Cache Args supported_call_types (list, optional): List of call types to cache for. Defaults to cache == on for all call types. **kwargs: Additional keyword arguments for redis.Redis() cache Raises: ValueError: If an invalid cache type is provided. Returns: None. Cache is set as a litellm param )rArBrCrU startup_nodes)rArBrCrH use_asyncembedding_model)rWrXcollection_namerHquantization_configr^) rJrKrLrMrNrOrPrQrRrSrT)disk_cache_dircacheNr*)LiteLLMCacheTypeREDISrrbREDIS_SEMANTICrQDRANT_SEMANTICrLOCALrS3rDISKrrinput_callbackappendsuccess_callback_async_success_callbackrIr?rDrUrEr&r(r@ isinstance)"selfr?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTredis_semantic_cache_use_async$redis_semantic_cache_embedding_modelrUrVrarWrXrYrZ%qdrant_semantic_cache_embedding_modelkwargss" r#__init__Cache.__init__@sF@ #)) )$.%!!11 %  %DJ%44 4+!%98 D DJ%55 5, /- 6%9$> E DJ%++ +&DJ %(( ( ---%# /%9)A%9#  DJ%** *".ADJ '00 0  " " ) )' 2 '22 2  $ $ + +G 4 '99 9  + + 2 27 ;$8! " 0#;y';'; 99(.. .3H3T,H II)// /yy,;;;".+H >> %*TZZ*L*L#'>>DJJ +M %r0returnc BSnUR"S0UD6nUb[R"SU5 U$UR5n[nUHnXd;a4UR Xa5nUbU[ U5S[ U53- nM:M<Xe;dMC[RSLdMXXcM_XnU[ U5S[ U53- nM [R"SU5 [RU5nUR"U40UD6nUR"SSU0UD6 U$)z Get the cache key for the given arguments. Args: **kwargs: kwargs to litellm.completion() or embedding() Returns: str: The cache key generated from the arguments, or None if no cache key could be generated. z Returning preset cache key: %sz: Tz Created cache key: %spreset_cache_keyr*) !_get_preset_cache_key_from_kwargsrr'_get_relevant_args_to_use_for_cache_keyr_get_param_valuestrr3enable_caching_on_provider_specific_optional_paramsr2_get_hashed_cache_key!_add_redis_namespace_to_cache_key_set_preset_cache_key_in_kwargs) rors cache_keyrycombined_kwargslitellm_param_kwargsparam param_valuehashed_cache_keys r# get_cache_keyCache.get_cache_keysM  AAKFK  '  !CEU V# #FFH1E'-1-B-B5-Q *CJ >vg>r0rrclURS/5nU(aUHnX$;dM [U5s $ g)Ncaching_groups)rr})rorrrgroups r#rCache._get_caching_group5s7*26F)K ''u:%(r0cURS5nURS05nURS05nURS5=(d= [USS5=(d) URS5=(d URS5$)z_ Handles getting the value for the 'file' param from kwargs. Used for `transcription` requests rrr file_checksumnameN file_name)rgetattr)rorsrrrs r#rCache._get_file_param_value?s{zz&!::j"-$4b9 LL ) /tVT* /||K( /!!+.  r0c HU(aSU;aUSRSS5$g)z Get the preset cache key from kwargs["litellm_params"] We use _get_preset_cache_keys for two reasons 1. optional params like max_tokens, get transformed for bedrock -> max_new_tokens 2. avoid doing duplicate / repeated work rryN)r)rorss r#rz'Cache._get_preset_cache_key_from_kwargsMs, 6)./334FMMr0ryc 0U(aSU;aXSS'ggg)z Set the calculated cache key in kwargs This is used to avoid doing duplicate / repeated work Placed in kwargs["litellm_params"] rryNr*)roryrss r#r%Cache._set_preset_cache_key_in_kwargs[s( 6)?O'();<* r0cUR5nUR5nUR5nUR5nUR 5nUR 5nUR UUUU5nURU5nU$)z@ Gets the supported kwargs for each call type and combines them )-_get_litellm_supported_chat_completion_kwargs-_get_litellm_supported_text_completion_kwargs'_get_litellm_supported_embedding_kwargs+_get_litellm_supported_transcription_kwargs$_get_litellm_supported_rerank_kwargs%_get_kwargs_to_exclude_from_cache_keyunion difference)rochat_completion_kwargstext_completion_kwargsembedding_kwargstranscription_kwargs rerank_kwargsexclude_kwargsrs r#r{-Cache._get_relevant_args_to_use_for_cache_keygs"&!S!S!U!%!S!S!UGGI#OOQAAC CCE066 "     *44^Dr0c[[RR55R [[ RR555nU$)zT Get the litellm supported chat completion kwargs This follows the OpenAI API Spec )setr __annotations__keysrr )roall_chat_completion_kwargss r#r3Cache._get_litellm_supported_chat_completion_kwargs{sJ &) . > > C C E& %3CCHHJK L #*)r0c[[RR55R [[ RR555nU$)zT Get the litellm supported text completion kwargs This follows the OpenAI API Spec )r&TextCompletionCreateParamsNonStreamingrrr#TextCompletionCreateParamsStreaming)roall_text_completion_kwargss r#r3Cache._get_litellm_supported_text_completion_kwargssJ &) 2 B B G G I& %7GGLLNO P #*)r0cP[[RR55$)z) Get the litellm supported rerank kwargs )rr model_fieldsrros r#r*Cache._get_litellm_supported_rerank_kwargss=--22455r0cP[[RR55$)zN Get the litellm supported embedding kwargs This follows the OpenAI API Spec )rr rrrs r#r-Cache._get_litellm_supported_embedding_kwargss (88==?@@r0cP[[RR55$)zR Get the litellm supported transcription kwargs This follows the OpenAI API Spec )rr rrrs r#r1Cache._get_litellm_supported_transcription_kwargss ,<<AACDDr0c[S/5$)z. Get the kwargs to exclude from the cache key r)rrs r#r+Cache._get_kwargs_to_exclude_from_cache_keysJ<  r0rc[R"UR55nUR5n[R "SU5 U$)z Get the hashed cache key for the given cache key. Use hashlib to create a sha256 hash of the cache key Args: cache_key (str): The cache key to hash. Returns: str: The hashed cache key. zHashed cache key (SHA-256): %s)hashlibsha256encode hexdigestrr)r hash_objecthash_hexs r#rCache._get_hashed_cache_keys?nnY%5%5%78 ((*=xHr0rc URS05RS5=(d URnU(aUSU3n[R"SU5 U$)z If a redis namespace is provided, add it to the cache key Args: hash_hex (str): The hashed cache key. **kwargs: Additional keyword arguments. Returns: str: The final hashed cache key with the redis namespace. rredis_namespace:zFinal hashed key: %s)rrDrr)rorrsrDs r#r'Cache._add_redis_namespace_to_cache_keysRJJz2.223DEW #AhZ0H3X>r0c## Sn[S[U5U5H)nSSSXX2-S.0/0v [R"S5 M+ g7f)Nrchoicesdelta assistant)rolecontentg{Gz?)rangelentimesleep)ror chunk_sizeis r#generate_streaming_content Cache.generate_streaming_contents[ q#g, 3A$/'.1>'B"   JJt 4sAA  cached_resultmax_agecdUb[U[5(apSU;ajUSn[R"5nXC- nUbXR:agURS5n[U[5(aU$[R "U5nU$U$![ a [R"U5nU$f=f)z< Common get cache logic across sync + async implementations N timestampresponse) rndictrrjsonloadsr!ast literal_eval)rorrr current_time response_agecached_responses r#_get_cache_logicCache._get_cache_logics  %=$//},%k2I99;L(3L"|'=,// ;O Dot44# " '+jj''O # " D"%"2"2?"C" " DsB 0B B/.B/c UR"S 0UD6SLagURS/5nSU;aUSnOUR"S 0UD6nUbeURS05nURSURS[S555nURR X2S 9nUR XeS 9$g![a$ [S [R"535 gf=f) z Retrieves the cached result for the given arguments. Args: *args: args to litellm.completion() or embedding() **kwargs: kwargs to litellm.completion() or embedding() Returns: The cached result if it exists, otherwise None. TNmessagesrrb s-max-ages-maxageinf)rrrAn exception occurred: r*) should_use_cacherrfloatrb get_cacherr!r$ traceback format_exc)rorsrrcache_control_argsrrs r#rCache.get_caches $$.v.d:zz*b1Hf$";/  ..88 $%+ZZ%<",00!3!7!7 E%L!Q!% 4 4Y 4 R ,,"/- %  3I4H4H4J3KL M sB1BB11+CCc # UR"S 0UD6SLagURS/5 SU;aUSnOUR"S 0UD6nUbqURS05nURSURS[S555nURR "U40UD6IShvN nUR XTS 9$gN![a$ [S [R"535 gf=f7f) zL Async get cache implementation. Used for embedding calls in async wrapper TNrrrbrrrrrr*) rrrrrbasync_get_cacherr!r$rr)rorsrrrrs r#rCache.async_get_cache's $$.v.d: JJz2 &f$";/  ..88 $%+ZZ%<",00!3!7!7 E%L!Q'+jj&@&@&Uf&U U ,,"/- % !V  3I4H4H4J3KL M sEC2CC2BC*B?+C=C2?C+C/,C2.C//C2c SU;aUSnOUR"S0UD6nUb[U[5(aUR5nURbURUS'UR SS5n[U[ 5(a%UR5HupVUS:XdM XbS'M [R"5US.nX7U4$[S5e![anUeSnAff=f)z? Common implementation across sync + async add_cache functions rNrErb)rrzcache key is Noner*) rrnrmodel_dump_jsonrErritemsrr!) roresultrsr _cache_kwargskv cached_dataes r#_add_cache_logicCache._add_cache_logicDs f$";/  ..88 $fi00#335F88'$(HHF5M & 7D 9 mT22 - 3 3 5:,-5M!6-1IIKVL  v55 344 G s$B"C(#C C C(!C##C(c UR"S0UD6SLagUR"SSU0UD6up4nURR"X440UD6 g![a,n[ R "S[U535 SnAgSnAff=f)z Adds a result to the cache. Args: *args: args to litellm.completion() or embedding() **kwargs: kwargs to litellm.completion() or embedding() Returns: None TNr #LiteLLM Cache: Excepton add_cache: r*)rrrb set_cacher!r exceptionr}ror rsrrrs r# add_cacheCache.add_cachebs U$$.v.d:-1-B-B..!'. *IF JJ  B6 B U  $ $'J3q6(%S T T UsA 4A B"A>>Bc# UR"S0UD6SLagURS:Xa)URbUR"U40UD6IShvN gUR"SSU0UD6up4nUR R "X440UD6IShvN gNBN![a,n[R"S[U535 SnAgSnAff=f7f)z# Async implementation of add_cache TNredisr rr*) rr?rUbatch_cache_writerrbasync_set_cacher!rrr}rs r#async_add_cacheCache.async_add_cachews U$$.v.d:yyG#(=(=(I,,V>v>>>151F1F2!2%+2. jj00R6RRR ? S U  $ $'J3q6(%S T T UshC BC 3B BBC 7B B BC BB C "CC C  C c# UR"S 0UD6SLagURbURUS'/n[US5HYupEUR"S 00UESU0ED6nXbS'URUnUR "S SU0UD6upnUR X45 M[ URR"S SU0UD6IShvN gN![a,n [R"S[U 535 Sn A gSn A ff=f7f) zm Async implementation of add_cache for Embedding calls Does a bulk write, to prevent using too many clients TNrEinputrr  cache_listrr*) rrE enumeraterdatarrkrbasync_set_cache_pipeliner!rrr}) ror rsr!idxrryembedding_responserrrs r#async_add_cache_pipelineCache.async_add_cache_pipelines  U$$.v.d:xx# $u J#F7O4#'#5#5#O8N68N7A8N#O &6{#%+[[%5"151F1F2-22. !!9":;5**55VVvV V V U  $ $'J3q6(%S T T UsKDC DB*C C C D C D"C=8D=DDc UR[R:XagURSS5n[R "SX5 U(a+[ U[5(aURSS5SLagg)z Returns true if we should use the cache for LLM API calls If cache is default_on then this is True If cache is default_off then this is only true when user has opted in to use cache TrbNz(should_use_cache: kwargs: %s; _cache: %sz use-cacheF)r@r&r(rrrrnr)rors_caches r#rCache.should_use_cachesd 99 ,, ,GT*GX j..zz+u-5r0c# UR"SSU0UD6up4nURR"X440UD6IShvN gN7f)Nr r*)rrbr)ror rsrrs r#rCache.batch_cache_writes@)-)>)>)Wf)WPV)W& jj**9LVLLLs8AAAch# [URS5nU(aU"5IShvN $gN7f)Npingrrb)ro cache_pings r#r/ Cache.pings+TZZ0 #% %&s (202cj# [URS5nU(aU"U5IShvN $gN7f)Ndelete_cache_keysr0)rorcache_delete_cache_keyss r#r4Cache.delete_cache_keyss0")$**6I"J "066 67s )313c# [URS5(a#URR5IShvN ggN7f)N disconnect)hasattrrbr8rs r#r8Cache.disconnects4 4::| , ,**'') ) ) - )s9AAAcdUR(aUR[R:Xagg)z Internal method to check if the cache type supports async get/set operations Only S3 Cache Does NOT support async operations FT)r?rcrhrs r#_supports_asyncCache._supports_asyncs# 99&6&9&99r0)rbr@rDrUrIrEr?)4r+r,r-r.rcrgr&r(rr}rrCachingSupportedCallTypesboolr rintrtrrr|rrrrzrrr{rrrrrr staticmethodrrrrrrrrrr'rrr/r4r8r<r/r*r0r#r2r2?s ,<+A+A  """&#'#150404 K )-(,(,%)04)-.226.2#'!%',-E*..2)-(,0448.F_f2'(f2  f2 sm f2smf23-f2C=f2e_f2 (f2'uof2'uof2't,E'FGf26! 7f28! 9f2:! ;f2<TN=f2>E$),-?f2@"#Af2B'smCf2D#+3-Ef2F'smGf2HC=If2J#Kf2P#3-Qf2R&d^Sf2V"#Wf2X! Yf2Z!) [f2\%-SM]f2P, , \    # ?T?c?"+3C= #  D  S   Xc]  P PRV PS( *s3x * *s3x *6c#h6 ASAESXE!s3x! $#C" %}%%%N@:<U*U&"UH$M  *  r0r2r4r?rArBrCrIc $[S5 S[R;a[RRS5 S[R;a[RRS5 S[R ;a[R RS5 [R c[SUUUUUS.UD6[l[S[R 35 [S[[R 535 g)a Enable cache with the specified configuration. Args: type (Optional[Literal["local", "redis", "s3", "disk"]]): The type of cache to enable. Defaults to "local". host (Optional[str]): The host address of the cache server. Defaults to None. port (Optional[str]): The port number of the cache server. Defaults to None. password (Optional[str]): The password for the cache server. Defaults to None. supported_call_types (Optional[List[Literal["completion", "acompletion", "embedding", "aembedding"]]]): The supported call types for the cache. Defaults to ["completion", "acompletion", "embedding", "aembedding"]. **kwargs: Additional keyword arguments. Returns: None Raises: None zLiteLLM: Enabling CacherbNr?rArBrCrIz&LiteLLM: Cache enabled, litellm.cache=LiteLLM Cache: r*) r$rrjrkrlrmrbr2varsr?rArBrCrIrss r# enable_cacherGsJ+,g,,,%%g.g...  ''0g555''..w7}} !5     :7==/JKOD$7#89:r0c [S5 [SUUUUUS.UD6[l[S[R35 [S[ [R535 g)a Update the cache for LiteLLM. Args: type (Optional[Literal["local", "redis", "s3", "disk"]]): The type of cache. Defaults to "local". host (Optional[str]): The host of the cache. Defaults to None. port (Optional[str]): The port of the cache. Defaults to None. password (Optional[str]): The password for the cache. Defaults to None. supported_call_types (Optional[List[Literal["completion", "acompletion", "embedding", "aembedding"]]]): The supported call types for the cache. Defaults to ["completion", "acompletion", "embedding", "aembedding"]. **kwargs: Additional keyword arguments for the cache. Returns: None zLiteLLM: Updating CacherCz&LiteLLM: Cache Updated, litellm.cache=rDNr*)r$r2rrbrErFs r# update_cacherIsgF+,   1   GM:7==/JKOD$7#89:r0czSSKJn [S5 U"[5 [R R S5 [RR S5 [RR S5 SSS5 S[l [S[R35 g!,(df  N6=f)a* Disable the cache used by LiteLLM. This function disables the cache used by the LiteLLM module. It removes the cache-related callbacks from the input_callback, success_callback, and _async_success_callback lists. It also sets the litellm.cache attribute to None. Parameters: None Returns: None rsuppresszLiteLLM: Disabling CacherbNz'LiteLLM: Cache disabled, litellm.cache=) contextlibrLr$ ValueErrorrrjremoverlrmrbrKs r# disable_cacherPJs$,- * %%g.  ''0''..w7  GM;GMM?KL  s AB,, B:)rGrIrPr*r0r#rds]  88TG+#.2!!!*6#4  T ^ ^ D(8'='=" G 7; # $7; 3-7; 3-7;sm 7; #4(A#BC 7;v(8'='=" G -; # $-; 3--; 3--;sm -; #4(A#BC -;`Mr0