^hs("SSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSK r SSK J r \RR\R"55 Sr\R "SS5r\S:Xa\ "5 SSKJr Sr"SS\5rS \4S jrS rS r\ R4"5\ R6"S SSSS9\ R6"SSSSS9\ R6"SSSSS9\ R6"SSSS9\ R6"SSSS9\ R6"S S!SS"S9\ R6"S#SS$S9\ R6"S%SS"S9\ R6"S&SS'S9\ R6"S(S)\S*S+9\ R6"S,S-S)\S.S/S09\ R6"S1S-S)\S2S3S09\ R6"S4S-S)\S5S69\ R6"S7S\S8S99\ R6"S:S\S;S99\ R6"SS)S?S@9\ R6"SAS)SBS@9\ R6"SCSDSSES9\ R6"SFS\SGS99\ R6"SHS)\SIS99\ R6"SJS\SKS99\ R6"SLSMS-S)\SNS69\ R6"SOS)SPSQ9\ R6"SRS)SSSQ9\ R6"STS-S)SUSV9\ R6"SWS-S)SXSV9\ R6"SYSZ\S[S99\ R6"S\S-S)S]SV9\ R6"S^S-S)S_SV9\ R6"S`S\SaSbSc9\ R6"SdS\SeSfSc9\ R6"SgS)S-ShSi9Sj5555555555555555555555555555555555r\ Sk:Xa\"5 gg)lN) load_dotenvzlitellm.secrets LITELLM_MODEDEV)Enumc\rSrSrSrSrSrg)LiteLLMDatabaseConnectionPool <N)__name__ __module__ __qualname____firstlineno__database_connection_pool_limit database_connection_pool_timeout__static_attributes__r Q/home/james-whalen/.local/lib/python3.13/site-packages/litellm/proxy/proxy_cli.pyrrs%'"')$rrreturnc`SSKJn URSU35 URSU35 [R"U5n[R"UR 5nUR U5 [R"USS9n[R"URUS95nU$)Nr)verbose_proxy_loggerzurl: zparams: T)doseq)query) litellm._loggingrdebugurlparseparse_qsrupdate urlencode urlunparse_replace)urlparamsr parsed_url parsed_query encoded_query modified_urls rappend_query_paramsr)s5se}-&23""3'J$$Z%5%56L&&|4@M&&z':':':'OPL rcSS/n[[RS5n[R"XUS9 SSS5 g!,(df  g=f![ an[ SUS35 SnAgSnAff=f)Nollamaservew)stdoutstderrzd LiteLLM Warning: proxy started with `ollama` model `ollama serve` failed with Exceptionz). Ensure you run `ollama serve` )openosdevnull subprocessPopen Exceptionprint)commandr2es rrun_ollama_server9*sp  W% "**c "g   WW E# " "   eefdgh     s1A?A A  A A A3A..A3cSSKnURURUR5nURSU45S:HsSSS5 $!,(df  g=f)Nr localhost)socketAF_INET SOCK_STREAM connect_ex)portr<ss ris_port_in_userB8sA v~~v'9'9 :a||[$/0A5 ; : :s A  Az--hostz0.0.0.0z!Host for the server to listen on.HOST)defaulthelpenvvarz--portzPort to bind the server to.PORTz --num_workersz%Number of gunicorn workers to spin up NUM_WORKERSz --api_basez API base URL.)rDrEz --api_versionz2024-07-01-previewz$For azure - pass in the api version.z--modelz-mz)The model name to pass to litellm expectsz--aliaszThe alias for the model - use this to give a litellm model name (e.g. "huggingface/codellama/CodeLlama-7b-Instruct-hf") a more user-friendly name ("codellama")z --add_keyz --headerszheaders for the API callz--saveTzSave the model-specific config)is_flagtyperEz--debugFzTo debug the inputDEBUG)rDrKrLrErFz--detailed_debugzTo view detailed debug logsDETAILED_DEBUGz --use_queuez)To use celery workers for async endpoints)rDrKrLrEz --temperaturezSet temperature for the model)rDrLrEz --max_tokenszSet max tokens for the modelz--request_timeoutz+Set timeout in seconds for completion callsz --drop_paramszDrop any unmapped params)rKrEz--add_function_to_promptz5If function passed but unsupported, pass it as promptz--configz-cz]Path to the proxy configuration file (e.g. config.yaml). Usage `litellm --config config.yaml`z --max_budgetz`Set max budget for API calls - works for hosted models like OpenAI, TogetherAI, Anthropic, etc.`z --telemetryzZHelps us know if people are using this feature. Turn this off by doing `--telemetry False`z --log_configz&Path to the logging configuration filez --versionz-vzPrint LiteLLM versionz--healthz:Make a chat/completions request to all llms in config.yaml) flag_valuerEz--testz4proxy chat completions url to make a test request toz --test_asyncz9Calls async endpoints /queue/requests and /queue/response)rDrKrEz--iam_token_db_authz!Connects to RDS DB with IAM tokenz--num_requestsr z-Number of requests to hit async endpoint withz--run_gunicornzTStarts proxy via gunicorn, instead of uvicorn (better for managing multiple workers)z--run_hypercornz@Starts proxy via hypercorn, instead of uvicorn (supports HTTP/2)z--ssl_keyfile_pathz^Path to the SSL keyfile. Use this when you want to provide SSL certificate when starting proxySSL_KEYFILE_PATH)rDrLrErFz--ssl_certfile_pathz_Path to the SSL certfile. Use this when you want to provide SSL certificate when starting proxySSL_CERTFILE_PATHz--localzfor local debugging)rKrDrEc! ^^^j^k[5n!U(aSSKJn"Jn#Jn$Jn%Jn&Jn'Jn(J n) OSSKJn"Jn#Jn$Jn%Jn&Jn'Jn(J n) USLa:[RRS5n+[R "SU+S35 gU(aS U;a Tc [#5 SSKmjUSLGa3SSKn,SSKmkS US T3mUUjUk4S jn-Un./n/TkR)5n0U,R*R-U.S 9n1[/U.5H#n2U/R1U1R3U-55 M% SSS5 U,R*R5U/5 Sn3Sn4U/H7n5U5R75(dMU5R95bU3S- n3M2U4S- n4M9 TkR)5n6[;SU6U0- 35 [;S5 [;SU.35 [;SU335 [;SU435 gUSLaN[;S5 TjR=S US TS3S9n7[;[>R@"U7R?5SS95 gUSLGaU=(d Sn8[R "SU835 SSK!n9USLa S US T3mOUmU9RESTS9n:U:RFRHRKU8SSS./S S!9n7[R "S"U735 [;S#U835 U:RFRHRKU8SSS./SS$9n7U7Hn;[R "S%U;35 M [;S&5 U:RHRKU8SS'9n7[;U75 gU(a[>RL"U5nU)"S0S(U_S)U_S*T_S+U_S,U _S-U _S.U _S/U _S0U _S1U_S2U_S3U_S4U_S5U_S6U_S7U_S8U_6 SSK'n<[PRRS9:XaOSSK*n=S;n>S5nB[PR\"S?5nC[PR\"S@5nD[PR\"SA5nE[PR\"SB5nFUA"UBUCUDSC9nGSDUDS UGSEUBS UCSFUE3 nHUF(aWHSGWF3- nHWH[PR^SH'SI[PR^SJ'SSKK0J1nI [PR\"SLS5bN[PR\"SL5SI:Xa4WI"5nJUJRe5HunKnLUL[PR^UK'M UGbGSSK3nMU$"5nNWMRiUNRkUSN95nOUOR=SOS5nPUPb)SPWP;a#WPSPSLaSSK6nQSUQl7UQRq5 WOR=SQ05n@U@c0n@W@(aW@R=SRS5nRURbWRU#RrRt:XaU'"SSS9 OWRU#RvRt:XaU("SST9 OWRU#RxRt:XaSSUK=J>nS USRSSV9 OTWRU#RRt:XaU&"SSW9 O3WRU#RRt:XaSSXKBJCnT UT"5 O [SY5eW@R=SZS5nUUUbSSK6nQU""S0WUD6UQlEW@R=S[S5nVUVc[PR\"S>5nW[PR\"S\5nX[PR\"S]5nY[PR\"SA5nZUW(a6WX(a/WY(a(WZ(a!SDWXS WYSEWWSFWZ3nVUV[PR^SH'W@R=S^[RRt5n>U@R=S_[RRt5n?WV(aWVRS`5(ay[PR"5n[[RRS[PRRSa55 SSK6nQSSbK6JOn\ U\"WVSSc9nV[PR"U[5 WVb([WV[5(aWV[PR^SH'[PR\"SHS5c[PR\"SdS5GbSSeKRJSn] [PR\"SHS5b+U>U?Sf.n^W]"SHSSc9nV[UVU^5n_U_[PR^SH'[PR\"SdS5b:U>U?Sf.n^[PR\"Sd5nV[UVU^5n_U_[PR^Sd'[Rh"Sg/SSh9 Sn`W`(aSSiKWJXna SSjKYJZnb Ub"W@R=Sk55SLaWa"SSl9 O[/S5Hn2[PR"5n[[PRR[5nc[PRRUc5nd[PR"Ud5 [Rh"/SmQ5 [PR"U[5 O O [;Sq5 TSr:Xa'[T5(a[R"SsSt5mSSK6nQU SLaWQR5 SSuKdJn% U%UTSv.neU b[;SwU 35 U WeSx'O!WQRn(a[;Sy5 SWeSx'USLa:USLa5UbUb[;SzUS{US|35 UWeS}'UUeS~'USSSS./S.nTRSUS9nUR5nUSnTU3nTRU5nUR5n[S U5 US nUS :XaUS  g[S USUSU35 T R S5 Mq![ an[SU5 SnAgSnAff=f)N gpt-3.5-turbouserz!Write a short poem about the moonrolecontent)modelmessagesz!http://0.0.0.0:4000/queue/request)jsonr#z RESPONSE FROM POLLING JOBstatusfinishedresultz POLLING JOBz STATUS: z , Response g?zgot exception in polling)postrfgetr6sleepr5) dataresponser# polling_urlpolling_responsergr8api_basehttpxtimes r_make_openai_completion+run_server.._make_openai_completion6s(#0STDzz"EDzQH}}H"5/C%-Jse"4K',yy'=$'7'<'<'>$8:JK-h7F+(2%k]*VHNScRdeJJsO!4a8sAB6%B B=' B88B=) max_workerszElapsed Time: zLoad test Summary:zTotal Requests: zSuccessful Calls: zFailed Calls: Fz) LiteLLM: Health Testing models in configz/health)r#)indentr_zE LiteLLM: Making a test ChatCompletions request to your proxy. Model=z My API Key)api_keybase_urlr`z*this is a test request, write a short poemra)rdre max_tokensz LiteLLM: response from proxy zN LiteLLM: Making a test ChatCompletions + streaming r equest to proxy. Model=)rdrestreamz'LiteLLM: streaming response from proxy z$ making completion request to proxy)rdpromptrdaliasrq api_versionrdetailed_debug temperaturer|request_timeout max_budget telemetry drop_paramsadd_function_to_promptheaderssaveconfig use_queuentzLuvicorn, gunicorn needs to be imported. Run - `pip install 'litellm[proxy]'`dr )generate_iam_auth_token DATABASE_HOST DATABASE_PORT DATABASE_USER DATABASE_NAMEDATABASE_SCHEMA)db_hostdb_portdb_userz postgresql://@/z?schema= DATABASE_URLTrueIAM_TOKEN_DB_AUTH)decrypt_env_var USE_AWS_KMSz?yaml needs to be imported. Run - `pip install 'litellm[proxy]'`)config_file_pathlitellm_settings json_logsgeneral_settingskey_management_system)use_azure_key_vault)use_google_kms)AWSSecretsManagerV2)use_aws_secret_manager) use_aws_kms)GoogleSecretManagerz&Invalid Key Management System selectedkey_management_settings database_urlDATABASE_USERNAMEDATABASE_PASSWORDrdatabase_connection_timeoutz os.environ/z../..)get_secret_str) default_value DIRECT_URL) get_secret)connection_limit pool_timeoutprisma)capture_output)check_prisma_schema_diff)should_update_schemadisable_prisma_schema_update)db_url)rdbpushz--accept-data-losszError: )startstopzYUnable to connect to DB. DATABASE_URL found in environment, but prisma package not found.rGii)rV)rVhostr@zUsing log_config: log_configz,Using json logs. Setting log_config to None.z/LiteLLM Proxy: Using SSL with certfile: z and keyfile: z ssl_keyfile ssl_certfilec<>^\rSrSrSUU4SjjrSrSrSrU=r$))run_server..StandaloneApplicationic>U=(d 0UlXl[TU] 5 STS3nUS-n[ 5 [ S5 [ SUS35 [ S5 [ STS35 g) Nz curl --location 'http://0.0.0.0:z/chat/completions' \a --header 'Content-Type: application/json' \ --data ' { "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "what llm are you" } ] }' zLiteLLM: Test your local proxy with: "litellm --test" This runs an openai.ChatCompletion request to your proxy [In a new terminal tab] z8LiteLLM: Curl Command Test for your local proxy z  z;Docs: https://docs.litellm.ai/docs/simple_proxy z5See all Router/Swagger docs on http://0.0.0.0:)options applicationsuper__init__r6)selfrVr _endpoint_str curl_command __class__r@s rr2run_server..StandaloneApplication.__init__s#*=bDL'*$G$&;4&@UV"&  ! GtVWcVddno\RSWRXXbcrcpURbOURR5VVs0sH'upXRR;dM UcM%X_M) nnnO0nUR5H>upURcMURR UR 5U5 M@ gs snnfN)cfgritemssettingssetlower)rkeyvaluers r load_config5run_server..StandaloneApplication.load_configsxx+/3ll.@.@.B".B "hh&7&77'.StandaloneApplication.loads+++r)rrr) r rrrrrrr __classcell__)rr@s@rStandaloneApplicationrs$$L =,,rrz)LiteLLM Proxy: Starting server on z with z workers zuvicorn.workers.UvicornWorker-iXz+%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s)bindworkers worker_classpreload accesslogtimeoutaccess_log_formatcertfilekeyfile)rVr)r,)Configz using Hypercorn r )nlocals proxy_serverrSrTrUrVrWrXrYrZ ImportErrorstr importlibmetadataversionclickechor9rr concurrentrsfuturesThreadPoolExecutorrangeappendsubmitwaitdonerir6rkrfdumpsopenaiOpenAIchat completionscreateloadsuvicornr1namegunicorn.app.baser5 litellm.proxy.auth.rds_iam_tokenrgetenvenviron*litellm.secret_managers.aws_secret_managerrrasynciorun get_configr[r _turn_on_jsonAZURE_KEY_VAULTr GOOGLE_KMSAWS_SECRET_MANAGER-litellm.secret_managers.aws_secret_manager_v2rload_aws_secret_managerAWS_KMSGOOGLE_SECRET_MANAGER-litellm.secret_managers.google_secret_managerr ValueError_key_management_settingsrrr startswithgetcwdsyspathinsertabspathrchdir isinstancelitellm.secret_managers.mainrr)r3FileNotFoundError litellm.proxy.db.check_migrationrlitellm.proxy.db.prisma_clientr__file__dirnameCalledProcessErrorrlrandom randrangerBrandint_turn_on_debuglitellm.proxy.proxy_serverbaseBaseApplicationhypercorn.asyncior,hypercorn.configrrrr)lrr@rqrrdradd_keyrrrrrr|rrrrrrtestlocal num_workers test_asynciam_token_db_auth num_requestsrhealthr run_gunicorn run_hypercornssl_keyfile_pathssl_certfile_pathrargsrSrTrUrVrWrXrYrZr8 pkg_versionrrtconcurrent_callsr start_timeexecutor_successful_calls failed_callsfutureend_timern request_modelrclientchunkrgunicorndb_connection_pool_limitdb_connection_timeoutrrrrrdb_name db_schematoken_db_urlr new_env_varkvr  proxy_config_configrr[rrrrr database_hostdatabase_usernamedatabase_password database_name original_dirrrr$r(is_prisma_runnablerrrdname uvicorn_argsrgunicorn_optionsr,rrrrssl `` @@r run_serverr]?s B 8D      4$((00;  2;-rBC U"x'7TTF!D6* @(YY[    2 2(3 +,x/FGH-  ( F{{}}==?.$)$ A%L  99; x 2345 "$  !1 234 "#3"456 |n-. U :;9974&$w!?9@ djj34 50  TUbTc d   4< av.HH|hG;;**11#K 2    4XJ?@ ]^k]l m ;;**11#K 2  E JJ@H I 56%%,,(T-  h jj)G    $    *  $ " , "   $ $:   ! " # &  ww$( $' "  Pii0Gii0Gii0Gii0G "34I+'E &gYaway'!G9UGXi[11)0BJJ~ &.4BJJ* + O IImT * 6 -(F2*+K#))+1 ! 1 ,     '=Lkk,"9"96"9"RSG '{{+=tD  ,#33$[1T9$(!%%'&{{+=rB '#% (8(<(<+T)%)4-.>>DDE2dK.2E2P2P2V2VV't<-.AAGGH ,CC37D/2E2M2M2S2SS$6-.DDJJK,-()QRR&6&:&:)4' #'23H4-40,//EL# " / : $&II.A$B!$&II.A$B! " / : "))%&33D2EQGXFYYZ[hZiijkxjy#zL1=BJJ~.'7';';0-LLRR( $%5$8$8--NNTT% ! 7 7 F F!yy{ rwww/2-l$O &'J|S,I,I-9 >* IInd + 7yyt,8 +C99^T2>-E(=F$.nD#QL#6|V#LL1=BJJ~.99\40<-E(=F$&99\#:L#6|V#LL/;BJJ|,z$?%)""UO)(,,-KL -D9"1X(*yy{ "$''//("; " 8 3&NN N"HH\2'&*o 4>$.D T !  " " $ 3  ! &zl3 4)3L &    @ A)-L & 5 ]e%; ,1A1MHIZH[[ijzi{|EF/? ]+/@ ^, KK ', ' T !7 , (9(9(I(I7 ,r >tfAdV6R]Q^^op  &$(& ? %R  !,1A1MHIZH[[ijzi{|EF0A ,.> + !!1 ce d "  / / >tfAdVKde XF"V1TF+,FK ,1A1MHIZH[[ijzi{|EF#4!1 KKc6* +)#W 3q6)      H  Z ^  l !U X% +%*" +8 *<<J'!GA3-0 JJv'7'7aa'HII JJHH\2BHH\2s~o#(3p p23p29q 0B3q$q7# p-%pp p/2q q!$ q43q47s  6ss(s  s((t__main__)!rrfr1r&r3r urllib.parseparserrdotenvrrrrconfig_filenamer litellm_modeenumrrrrr)r9rBr7optionboolfloatintr]r r rrris    #yy/ 5M  *D*    6 i&IRXh+HQWX 0   lD? /  tT(S  k $Ok4.HIh46VW        &     4 T4SDs1O  6  ot2LM  @   h     k    e    1      E   ?    D     ,    8     _     K    i    j  iu;PQZ ,R            N XJ  @ Y|Z ,z zLr