i#S dSrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK J r SSK J r SSKJr SSKJrJr SSKJr SSKJr SS KJr SS KJrJrJrJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r&J'r' SSK(r(SSK)r)SS K*J+r+ \(aSS K,J-r- "S S5r.\."5r/\'"SS\#55r0S\&\0\\1/\244SS4Sjr3SYS\1S\\1/\!44Sjjr4S\$S\14Sjr5S\$S\64Sjr7\ S\S\S\4Sj5r8\"SS55r9SZS\S\!\9S \6S!\:S"\:S\4 S#jjr;SZS$\S\!\9S \6S!\:S"\:S\4 S%jjrS,\&\ \4S-\6S\ 4S.jr?S/\1S0\1S\14S1jr@S[S2\ S3\AS4\1S5\"\6S\ 4 S6jjrBS)\1S\64S7jrCS\1S\\1/\!\144S8jrDS\14S9jrES:\S\4S;jrFSS?S@SA.rHSBSCSDSESFSGSHSI.rIS\S2\!\1SJ\AS\\1\144SKjjrJS)\1SL\1S\14SMjrKS]S)\1SL\"\1SN\1SS4SOjjrLS\14SPjrMSQ\S\\SS44SRjrNSQ\S\\S44SSjrOS^ST\"\PSU\"\&\1\4SV\"\1SW\AS\4 SXjjrQg)_zGeneral utils functions.N)Error)contextmanager) dataclass)partialwraps)BytesIO)islice)Path) TYPE_CHECKINGAnyAsyncGeneratorCallableDict GeneratorIterableListOptionalProtocolSetTypeUnionruntime_checkable)urlparsePunktSentenceTokenizerc\rSrSr%SrSr\\\\ S'Sr \S\ S'Sr \\\ S'SS jr SS jr \S\\4S j5r\SS j5rS rg) GlobalsHelper+z?Helper to retrieve globals with asynchronous NLTK data loading.N _stopwordsr_punkt_tokenizer_nltk_data_dirreturncVSSKJn S[R;a+[ [ [RS55UlO\[ [RR[RR[555n[ US- 5Ul[RRUR 5(d[R"UR SS9 UR U;aURUR 5 UR5 g)zInitialize NLTK data download.rpath NLTK_DATAz_static/nltk_cacheT)exist_okN) nltk.datar%osenvironstrr r!dirnameabspath__file__existsmakedirsappend_download_nltk_data)self nltk_pathr%s P/home/james-whalen/.local/lib/python3.13/site-packages/llama_index/core/utils.pywait_for_nltk_check!GlobalsHelper.wait_for_nltk_check2s/ "** $"%d2::k+B&C"DD (ABCD"%d-A&A"BD ww~~d1122 KK++d ;   i /   T00 1   "c<SSKJn SSKJn U"SUR/S9 U"S UR/S9 g ![ a U"SURSS9 N2f=f![ a U"S URSS9 g f=f![ an[S U35 S nAg S nAff=f) z.Download NLTK data packages in the background.r)download)findzcorpora/stopwords)paths stopwordsT) download_dirquietztokenizers/punkt_tab punkt_tabzNLTK download error: N)nltkr:r(r;r! LookupError Exceptionprint)r3r: nltk_findes r5r2!GlobalsHelper._download_nltk_dataHs!/ / T-d6I6I5JK  T09L9L8MN  T43F3FdS T  T43F3FdS T / )!- . . /sJ5AAA9AA9A63A95A66A99 BBBcURc>UR5 SSKJn SSKJn UR S5UlU"5UlUR$)z+Get stopwords, ensuring data is downloaded.rr=renglish)rr6 nltk.corpusr= nltk.tokenizerwordsr r3r=rs r5r=GlobalsHelper.stopwords]sE ?? "  $ $ & - <'ooi8DO$:$UR5 SSKJn SSKJn U"5UlUR S5UlUR$)z1Get punkt tokenizer, ensuring data is downloaded.rrIrrJ)r r6rKr=rLrrMrrNs r5punkt_tokenizerGlobalsHelper.punkt_tokenizerlsI  (  $ $ & - <$:$ Y'  /0    , , 88 8    , ,,) 312 2 3s D!!D7dcT[[R"55nX;aU$M()z Get a new ID.)r+uuiduuid4r{new_ids r5 get_new_idrs' TZZ\" ?  M r8cb[R"S[R5nX;aU$M/)zGet a new integer ID.r)randomrandintsysmaxsizers r5get_new_int_idrs+ 3;;/ ?  M r8objrac+># UVs0sHo"[X5_M nnUR5Hup$[XU5 M Sv UR5Hup$[XU5 M gs snf!UR5Hup$[XU5 M f=f7f)z Temporary setter. Utility class for setting a temporary value for an attribute on a class. Taken from: https://tinyurl.com/2p89xymh N)getattritemssetattr)rrak prev_valuesvs r5temp_set_attrsrs066v!gco%vK6  %%'DA CA ( 7  %%'DA CA (s'BA.'BA3+B3'BBcJ\rSrSr%Sr\\\S'Sr\ \ \ /\ 4\S'Sr g) ErrorToRetryz Exception types that should be retried. Args: exception_cls (Type[Exception]): Class of exception. check_fn (Optional[Callable[[Any]], bool]]): A function that takes an exception instance as input and returns whether to retry. exception_clsNcheck_fnr[)rSrTrUrVrWrrCrXrrrr boolrZr[r8r5rrs-  ?"04Hhxt ,-4r8r lambda_fnerrors_to_retry max_triesmin_backoff_secsmax_backoff_secscU(d [S5eUVs0sHnURUR_M nn[UR 55nUnSn U"5$s snf!Uan [ R "5 U S- n X:aeURU R5n U (aU "U 5(de[R"U5 [US-U5nSn A OSn A ff=fM)a Execute lambda function with retries and exponential backoff. Args: lambda_fn (Callable): Function to be called and output we want. errors_to_retry (List[ErrorToRetry]): List of errors to retry. At least one needs to be provided. max_tries (int): Maximum number of tries, including the first. Defaults to 10. min_backoff_secs (float): Minimum amount of backoff time between attempts. Defaults to 0.5. max_backoff_secs (float): Maximum amount of backoff time between attempts. Defaults to 60. 0At least one error to retry needs to be providedrN ValueErrorrrtuplekeys traceback print_excget __class__timesleepmin) rrrrrerror_to_retry error_checksexception_class_tuples backoff_secstriesrFrs r5 retry_on_exceptions_with_backoffrs* KLL.-N $$n&=&==-#<#4#4#67#L E  C; & C    ! QJE!#'' 4H  JJ| $|a/1ABL C s!A A%%C*+A5C%%C*async_fnc# U(d [S5eUVs0sHnURUR_M nn[UR 55nUnSn U"5IShvN $s snfN !Uan [ R "5 U S- n X:aeURU R5n U (aU "U 5(de[R"U5 [US-U5nSn A OSn A ff=fM7f)a  Execute lambda function with retries and exponential backoff. Args: async_fn (Callable): Async Function to be called and output we want. errors_to_retry (List[ErrorToRetry]): List of errors to retry. At least one needs to be provided. max_tries (int): Maximum number of tries, including the first. Defaults to 10. min_backoff_secs (float): Minimum amount of backoff time between attempts. Defaults to 0.5. max_backoff_secs (float): Maximum amount of backoff time between attempts. Defaults to 60. rrNrrr) rrrrrrrrrrrFrs r5!aretry_on_exceptions_with_backoffrs* KLL.-N $$n&=&==-#<#4#4#67#L E  C!# #$% C    ! QJE!#'' 4H  JJ| $|a/1ABL C sLC;!A* C; A1%A/&A1)C;/A11C67A5C1,C;1C66C; retry_args retry_kwargsc4^^S[S[4UU4SjjnU$)zPReturn a decorator that retries with exponential backoff on provided exceptions.funcr"c>^[T5S[S[S[4UUU4Sjj5n[T5S[S[S[4UUU4Sjj5n[R"T5(aU$U$)N func_args func_kwargsr"c2>^^[UUU4Sj/TQ70TD6$)Nc>T"T0TD6$rcr[rrrsr5dget_retry_on_exceptions_with_backoff_decorator..decorator..wrapper..Msi7;7r8)r)rrrrrs``r5wrapperRget_retry_on_exceptions_with_backoff_decorator..decorator..wrapperJs#37:DHT r8cb>^^# S[4UUU4Sjjn[U/TQ70TD6IShvN $N7f)Nr"c0># T"T0TD6IShvN $N7frcr[rsr5foo`get_retry_on_exceptions_with_backoff_decorator..decorator..awrapper..fooRs!9< <<<.decorator..awrapperPsC =s = =; $0 s #/-/)rr asyncioiscoroutinefunction)rrrrrs` r5 decoratorAget_retry_on_exceptions_with_backoff_decorator..decoratorIs t  C C     t s 3 3    #66t<>> list(iter_batch([1,2,3,4,5], 3)) [[1, 2, 3], [4, 5]] rN)iterlistr r)rr source_iterbs r5 iter_batchres<x.K   * + q6Q;  +s AAAr,basenamec^XSS:waSOS- n[RRX5$)z Append basename to dirname, avoiding backslashes when running on windows. os.path.join(dirname, basename) will add a backslash before dirname if basename does not end with a slash, so we make sure it does. /)r)r%ru)r,rs r5 concat_dirsrts, bkS(sb0G 77<< **r8r show_progressdesctotalcVUnU(aSSKJn U"XUS9$U$![a U$f=f)z< Optionally get a tqdm iterable. Ensures tqdm.auto is used. r)tqdm)rr) tqdm.autorrt)rrrr _iteratorrs r5get_tqdm_iterablersC I  &6 6     s  ((c<[5nU"U5n[U5$rc)rzr)r_rftokenss r5 count_tokensrsI t_F v;r8c|SSKJn UR U5nUR $![a [S5ef=f)z Args: model_name(str): the model name of the tokenizer. For instance, fxmarty/tiny-llama-fast-tokenizer. r) AutoTokenizerzG`transformers` package not found, please run `pip install transformers`) transformersrrtrfrom_pretrainedtokenize)rnrrfs r5get_transformer_tokenizer_fnrsI . --j9I      U   s%;cS[R;a[[RS5nO[[R"S55nUR SSS9 [ U5$)zg Locate a platform-appropriate cache directory for llama_index, and create it if it doesn't yet exist. LLAMA_INDEX_CACHE_DIRrlT)parentsr')r)r*r platformdirsuser_cache_dirmkdirr+r$s r5 get_cache_dirrsU "**,BJJ678L// >? JJtdJ+ t9r8rc^[R"T5(de[T5S[S[S[4U4Sjj5nUTlT$)z Decorator for adding sync version of an async function. The sync version is added as a function attribute to the original function, func. Args: func(Any): the async function for which a sync variant will be built. r`kwdsr"cX>[R"5RT"U0UD65$rc)rget_event_looprun_until_complete)r`rrs r5_wrapper"add_sync_version.._wrappers'%%'::4;N;NOOr8)rrrr sync)rrs` r5add_sync_versionrsZ  & &t , ,, , 4[PPSPSPPDI Kr8a Context LLMs are a phenomenal piece of technology for knowledge generation and reasoning. They are pre-trained on large amounts of publicly available data. How do we best augment LLMs with our own private data? We need a comprehensive toolkit to help perform this data augmentation for LLMs. Proposed Solution That's where LlamaIndex comes in. LlamaIndex is a "data framework" to help you build LLM apps. It provides the following tools: Offers data connectors to ingest your existing data sources and data formats (APIs, PDFs, docs, SQL, etc.) Provides ways to structure your data (indices, graphs) so that this data can be easily used with LLMs. Provides an advanced retrieval/query interface over your data: Feed in any LLM input prompt, get back retrieved context and knowledge-augmented output. Allows easy integrations with your outer application framework (e.g. with LangChain, Flask, Docker, ChatGPT, anything else). LlamaIndex provides tools for both beginner users and advanced users. Our high-level API allows beginner users to use LlamaIndex to ingest and query their data in 5 lines of code. Our lower-level APIs allow advanced users to customize and extend any module (data connectors, indices, retrievers, query engines, reranking modules), to fit their needs. z38;2;237;90;200z38;2;90;149;237z38;2;11;159;203z38;2;155;135;227) llama_pink llama_bluellama_turquoisellama_lavender313233343536z38;5;200)redgreenyellowbluemagentacyanpinkuse_llama_index_colorsc U(a[nO[n[UR55n[ U5VVs0sHupEXSU[ U5-_M snn$s snnf)a/ Get a mapping of items to colors. Args: items (List[str]): List of items to be mapped to colors. use_llama_index_colors (bool, optional): Flag to indicate whether to use LlamaIndex colors or ANSI colors. Defaults to True. Returns: Dict[str, str]: Mapping of items to colors. )_LLAMA_INDEX_COLORS _ANSI_COLORSrr enumerater)rr color_palettecolorsiitems r5get_color_mappingrsU + $ -$$& 'F9B59I J9IgaDS[) )9I JJ JsAcolorcN0[E[EnX;aSUS3$X!nSUSUS3$)z Get the colored version of the input text. Args: text (str): Input text. color (str): Color to be applied to the text. Returns: str: Colored version of the input text. zzz[1;3;m)rr)r_r all_colorss r5_get_colored_textr sF9'8<8J 4&((  E ugQtfG ,,r8endc6Ub [X5OUn[X2S9 g)a Print the text with the specified color. Args: text (str): Text to be printed. color (str, optional): Color to be applied to the text. Supported colors are: llama_pink, llama_blue, llama_turquoise, llama_lavender, red, green, yellow, blue, magenta, cyan, pink. end (str, optional): String appended after the last character of the text. Returns: None N)r!)r rD)r_rr! text_to_prints r5 print_textr$'s7<6G%d2TM -!r8cWRR5nU(agURR R5(agg![a! SSKnURR5nN]f=f)z Infer the input to torch.device.rNcudampscpu)r& is_available NameErrortorchbackendsr')r+has_cudas r5infer_torch_devicer.:sf-::**,  ~~&&((  -::**,-sA(A:9A:xc## Uv g7fz A function that returns a generator of a single element. Args: x (Any): the element to build yield Yields: Any: the single element Nr[r/s r5unit_generatorr3Is  Gsc# U7v g7fr1r[r2s r5async_unit_generatorr5Ws  Gs  raw_bytesr%url as_base64cUb^[R"U5n[R"USS9nU(a[ [R "U55$[ U5$Ubc[ U[5(a [U5OUnUR5nU(a[ [R "U55$[ U5$UGbh[U5nURS:XaURnSU;a [S5eURSS5upURS5n U (aG[R"U 5n U(a[ [R "U 55$[ U 5$U(a.[ [R "U R!S555$[ U R!S55$S S 0n ["R$"X,S S 9n U R'5 U(a)[ [R "U R(55$[ U R(5$[S 5e![a UnGN3f=f![a UnGN0f=f)aX Resolve binary data from various sources into a BytesIO object. Args: raw_bytes: Raw bytes data path: File path to read bytes from url: URL to fetch bytes from as_base64: Whether to base64 encode the output bytes Returns: BytesIO object containing the binary data Raises: ValueError: If no valid source is provided T)validatedata,z0Invalid data URL format: missing comma separatorrz;base64zutf-8z User-AgentzOLlamaIndex/0.0 (https://llamaindex.ai; info@llamaindex.ai) llama-index-core/0.0)<r=)headerstimeoutz0No valid source provided to resolve binary data!)base64 b64decoderC BinasciiErrorr b64encoderir+r read_bytesrschemer%rsplitendswithrdrequestsrraise_for_statuscontent)r6r%r7r8 decoded_bytesr; parsed_url data_partmetadataurl_datais_base64_encoded decoded_datar>responses r5resolve_binaryrSes, &",,Y7M &#,,YFM 6++M:; ;}%% 'c22tDz  6++D12 2t} c]    &#I)# !STT!*a!8 H ( 1 1) <  %//9 "6#3#3L#ABB#<00"6#3#3HOOG4L#MNN#8??7#;<< k <<hG!!# 6++H,<,<=> >x''(( G HHA &%M & &%M &s"H?I? II I"!I")z gpt-3.5-turbo) g?gN@rc)T)Nr)NNNF)RrWrr@r)rrrrr}binasciirrB contextlibr dataclassesr functoolsrrior itertoolsr pathlibr typingr r r rrrrrrrrrrrrrH urllib.parserrLrrglobals_helperr]r+rrmrzrintrrrfloatrrrrrrrrrrrr SAMPLE_TEXTrrrr r$r.r3r5bytesrSr[r8r5rcs?  +%!$"!4N%N%bMMM6E)XseTk5J*J$K6PT6-c-#8M-@##ccsy$  5 5  5&!" -C-C,'-C-C -C  -C  -Cf!" -C-C,'-C-C -C  -C  -C`&) 4**** x23 3 8 ++++MQ $(03N5O"s"33( 6$#((       6:K 9K.2K #s(^K2-C---,"S"#"C""& C   c  iT48    #  .d*C  "&'+ ZIZI 5d# $ZI #ZI ZI  ZIr8