h.,SSKrSSKJr SSKJrJrJrJrJrJ r J r SSK r SSK r SSK J r SSKJr SSKJrJrJrJr SSKJr SS KJr SS KJrJr SS KJr S S KJr S SK J!r" S SK J#r# "SS5r$Sr%Sr&Sr'\RP"\$\&\'5 \$r)S/r*g)N)Path)DictIterableIteratorListOptionalSetUnion)ndarray)NumpyOps)IDSORTHSPACY intify_attr)copy_reg)Errors)SimpleFrozenList ensure_path)Vocab) SpanGroups)DOCBIN_ALL_ATTRS)Docc \rSrSrSr\S\"54S\\S\ S\\ SS4S jjr S\ 4S jr S \ SS4S jrS \S\\ 4SjrSSjrS\4SjrS\SS4SjrS\\\4SS4SjrS\\\4SS4SjrSrg)DocBinauPack Doc objects for binary serialization. The DocBin class lets you efficiently serialize the information from a collection of Doc objects. You can control which information is serialized by passing a list of attribute IDs, and optionally also specify whether the user data is serialized. The DocBin is faster and produces smaller data sizes than pickle, and allows you to deserialize without executing arbitrary Python code. The serialization format is gzipped msgpack, where the msgpack object has the following structure: { "attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE] "tokens": bytes, # Serialized numpy uint64 array with the token data "spans": List[Dict[str, bytes]], # SpanGroups data for each doc "spaces": bytes, # Serialized numpy boolean array with spaces data "lengths": bytes, # Serialized numpy int32 array with the doc lengths "strings": List[str] # List of unique strings in the token data "version": str, # DocBin version number } Strings for the words, tags, labels etc are represented by 64-bit hashes in the token data, and every string that occurs at least once is passed via the strings object. This means the storage is more efficient if you pack more documents together, because you have less duplication in the strings. A notable downside to this format is that you can't easily extract just one document from the DocBin. Fattrsstore_user_datadocsreturnNcUVs/sHn[U5PM nnSU;a\UVs/sHn[U5bMUPM nn[[RR SU[ R "5S95Se[U5nSUlUVs/sHoD[:wdMU[:wdMUPM snUl URRS[5 /Ul /Ul/Ul/Ul/Ul/Ul['5UlX lUHnUR-U5 M gs snfs snfs snf)aNCreate a DocBin object to hold serialized annotations. attrs (Iterable[str]): List of attributes to serialize. 'orth' and 'spacy' are always serialized, so they're not required. store_user_data (bool): Whether to write the `Doc.user_data` to bytes/file. docs (Iterable[Doc]): Docs to add. DOCS: https://spacy.io/api/docbin#init Nr)dictkeykeysz0.1r)rKeyErrorrE983formatrr%sortedversionrrrinserttokensspacescats span_groups user_dataflagssetstringsradd)selfrrr attr int_attrs non_validdocs Q/home/james-whalen/.local/lib/python3.13/site-packages/spacy/tokens/_serialize.py__init__DocBin.__init__4s49954[&5 9 9 */M%$;t3D%IM ""YSXXZ"P y! ',Out dduO  !T"%' %' " (*02!# !$ .C HHSM%:M Ps!D9D>D> E%E1Ec,[UR5$)z7RETURNS: The number of Doc objects added to the DocBin.)lenr,)r5s r:__len__DocBin.__len__Xs4;;r9c:URUR5n[UR5S:Xa UR URSS45nUR R U5 UR[5nURSURS:XdeUR URSS45nURR [R"U[S95 URR SUR05 UGHZnURRUR 5 URRUR"5 URRUR$5 URRUR&5 URR[)UR*55 URRUR,5 URRUR.5 URRUR05 URRUR25 GM] UR4R UR45 UR6(a9UR8R [:R<"UR855 UR>R UR@RC55 UR@RE5HupVUHnURRURF5 URHURJRLR;a%URRURN5 URPURJRLR;dMURRURR5 M M g)zAdd a Doc's annotations to the DocBin for serialization. doc (Doc): The Doc object to add. DOCS: https://spacy.io/api/docbin#add rrdtypehas_unknown_spacesN)*to_arrayrr>shapereshaper,appendrr-numpyasarrayboolr1rEr3r4texttag_lemma_norm_strmorphdep_ ent_type_ ent_kb_id_ent_id_r.rr0srsly msgpack_dumpsr/spansto_bytesitemslabel_kb_idr9vocabkb_id_idid_)r5r9arrayr-tokenr$groupspans r:r4 DocBin.add\s TZZ( u{{ q MM5;;q>1"56E 5!e${{1~a000a! 45 5==t<= /1G1GHIE LL  UZZ ( LL  UZZ ( LL  U\\ * LL  U[[ ) LL  S- . LL  UZZ ( LL  U__ - LL  U-- . LL  U]] + "    NN ! !%"5"5cmm"D E  2 2 45))//+JC   -::!7!77LL$$T[[177dhhnn444LL$$TXX. ,rAr^c## URHnX M URR[5n[ [ UR 55GHlnURUnUR UnURUnURS5(aSn[XSS2U4US9nURURU5nURUUl URU(aJURU[R:wa)UR R#URU5 OUR R%5 U[ UR&5:aMUR&Ub=[(R*"UR&USS9n UR&R-U 5 Uv GMo g7f)a1Recover Doc objects from the annotations, using the given vocab. Note that the user data of each doc will be read (if available) and returned, regardless of the setting of 'self.store_user_data'. vocab (Vocab): The shared vocab. YIELDS (Doc): The Doc objects. DOCS: https://spacy.io/api/docbin#get_docs rEN)wordsr-F)use_list)r3rindexrranger>r,r1r-getr from_arrayr.r/r _EMPTY_BYTESrY from_bytesclearr0rW msgpack_loadsupdate) r5r^stringorth_colir1r,r-r9r0s r:get_docsDocBin.get_docss\llF M#::##D)s4;;'(AJJqME[[^F(, AFyy-..e!X+#6vFC..V4Cyy|CH"t'7'7':j>U>U'U $$T%5%5a%89 !3t~~&&4>>!+<+H!//q0AER  $$Y/I#)sGG cbURUR:wa<[[RR SURURS95eUR UR :wa<[[RR SUR UR S95eUR RUR 5 URRUR5 URRUR5 URRUR5 URRUR5 URRUR5 URRUR5 g)a2Extend the annotations of this DocBin with the annotations from another. Will raise an error if the pre-defined attrs of the two DocBins don't match, or if they differ in whether or not to store user data. other (DocBin): The DocBin to merge into the current bin. DOCS: https://spacy.io/api/docbin#merge r)paramcurrentotherrN)r ValueErrorrE166r(rr,extendr-r3rrr.r/r1r0)r5r{s r:merge DocBin.merges1 :: $ ""$**EKK"X    5#8#8 8 ""+ 00//#  5<<( 5<<( EMM* $  1 12 %++& eoo.rAc URH-n[UR5S:XaMUR5e URVs/sHn[U5PM nnUR(a [R"UR5O[R "/5nUR (a [R"UR 5O[R "/5nURURURS5URS5[R "USS9RS5[[UR55URURURS. nUR (aUR"US'[$R&"[(R*"U55$s snf)zSerialize the DocBin's annotations to a bytestring. RETURNS (bytes): The serialized DocBin. DOCS: https://spacy.io/api/docbin#to_bytes r Cint32rC) r*rr,r-lengthsr3r.r1r/r0)r,r>rGrJvstackrKr-r*rtobyteslistr)r3r.r1r/rr0zlibcompressrWrX)r5r,rr-msgs r:rZDocBin.to_bytess:kkFv||$) 76<< 7)"-1[[9[63v;[9.2kkdkk*u}}R?P.2kkdkk*u}}R?P||ZZnnS)nnS)}}WG<DDSIF4<<01IIZZ++     #~~C }}U00566!:s G bytes_datacp[R"[R"U55nUSUl[US5Ul [R"USSS9n[R"US[S9n[R"USSS9nUR[UR5-[UR54nUR!U5nUR!URS 45n[#5R%XS5Ul[#5R%XC5UlUS UlUR-S UVs/sHnS PM sn5UlUR-S UVs/sHn0PM sn5UlSU;a[3US5UlOS/[U5-UlUR&H-n[UR65S:XaMUR65e U$![Ra [ [ R5ef=fs snfs snf)zDeserialize the DocBin's annotations from a bytestring. bytes_data (bytes): The data to load from. RETURNS (DocBin): The loaded DocBin. DOCS: https://spacy.io/api/docbin#from_bytes rr3rrrCr-r,uint64rr.r/rAr1r0Nr )rWrqr decompresserrorr|rE1014rr2r3rJ frombufferrLsizer>rHr unflattenr,r-r.rlr/r1rr0rG) r5rrr flat_spaces flat_tokensrG_r,s r:roDocBin.from_bytess +%%dooj&ABC\ 3y>* ""3y>A&&s8}DA &&s8}HE !!S_4c$**oF!))%0 !));+;+;Q*?@ j**;@ j**;@ K 77=2H132HIWWW7&;7ar7&;< # !#k"23DN"Vc$i/DNkkFv||$) 76<< 7)" +zz +V\\* * +3I&;s*G= H. 8 H3 =.H+pathc[U5nURS5nURUR55 SSS5 g![a [ [ R 5ef=f!,(df  g=f)zSave the DocBin to a file (typically called .spacy). path (str / Path): The file path. DOCS: https://spacy.io/api/docbin#to_disk wbN)ropenwriterZr|rE870r5rfile_s r:to_diskDocBin.to_disksa4  YYt_ . DMMO,_ . -- ._sA/A$A,,A// A=c[U5nURS5nURUR55 SSS5 U$!,(df  U$=f)zLoad the DocBin from a file (typically called .spacy). path (str / Path): The file path. RETURNS (DocBin): The loaded DocBin. DOCS: https://spacy.io/api/docbin#to_disk rbN)rrroreadrs r: from_diskDocBin.from_disk sD4  YYt_ OOEJJL ) _ s A A) rr.r1r-r/rr3r,r0r*)r{rr!N)__name__ __module__ __qualname____firstlineno____doc__ ALL_ATTRSrrrQrLrr;intr?r4rrrvrbytesrZror rrr__static_attributes__rAr:rrsB ) %.0 "}""sm "  "H  $/s$/t$/Le @/<7%76UxB .E#t), . . eCI. 8 rArcSnUH8nUcM[SS9RU5nUcUnM'URU5 M: UbUR5$g)NT)rrA)rrorrZ)binsmerged byte_stringdoc_bins r: merge_binsrsZ F  "T2==kJG~  W%   rAc0[UR544$N) unpickle_binrZ)rs r: pickle_binr's 7++-/ 00rAc4[5RU5$r)rro)rs r:rr+s 8  { ++rA)+rpathlibrtypingrrrrrr r rJrWr thinc.apir rrrrrcompatrerrorsrutilrrr^r _dict_proxiesrr9rrrrrrrpickleBinder__all__rrAr:rsu GGG 110%.AAH 1,  L1  *rA