hs# SSKrSSKJrJrJrJrJrJrJr SSK J r J r SSK J r Jr S\\S\\4SjrS\\S\\4S jrS\\S\\4S jrS\\S\\4S jrSS \ S \4SjjrS \ S\\4SjrSS \ S\\\\\\\44S \S\\4SjjrS \ S\\S\\4SjrS \ S\\S\\\\\\\444SjrS\\S\\\\\44SjrS\S\\\44SjrS\S\4Sjr\r\r\rg)N)DictIterableIteratorListTupleUnioncast)ErrorsWarnings)DocSpantagsreturnc/n[U5nU(a=UR[U55 UR[U55 U(aM=U$)N)listextend _consume_os _consume_ent)routs R/home/james-whalen/.local/lib/python3.13/site-packages/spacy/training/iob_utils.py iob_to_biluorsCC :D  ;t$% <%& $ Jc/nUHNnUcURU5 MURSSS5RSSS5nURU5 MP U$)NU-B-L-I-)appendreplace)rrtags r biluo_to_iobr#sT C ; JJsO++dD!,44T4CC JJsO  Jrc#|# U(a1USS:Xa'URS5v U(a USS:XaM%gggg7f)NrO)pop)rs rrrs5 47c>hhqk 47c>$>$s4<<cU(d/$URS5nSUSS-nSUSS-nSnU(a3USX#1;a)US- nURS5 U(a USX#1;aM)USSnUS:Xa;[U5S:Xa&[[RR US95eSU-/$SU-nS U-n[ SUS- 5Vs/sHnS U3PM n nU/U -U/-$s snf) NrIrLr r"rrrr)r&len ValueErrorr E177formatrange) rr" target_in target_lastlengthlabelstartend_middles rrr!s   ((1+Cc!"g IAB-K F 47y66!    47y66 GE { u:?V[[//C/89 9u ~u Ul(-a!(<=(<1Bug,(<=w3%''>sC#docmissingc [UURVs/sH%o"RURUR4PM' snUS9$s snf)Nr9)offsets_to_biluo_tagsents start_charend_charlabel_)r8r9ents rdoc_to_biluo_tagsrB7s?  ?BxxHx..#,, 3xH Hs,A cn[USS9n[U5Hup#URS:XdMSX'M U$)N-r;r r%)rB enumerateent_iob)r8r=itokens r_doc_to_biluo_tags_with_partialrI?s8 S# .DcN ==A DG# Krentitiesc <0nUVs0sHoDRUR_M nnUVs0sH%oDR[U5-UR_M' nnUVs/sHnSPM nnUHupn U (d UHn X:dM X:dMSXU 'M M-[X5HWn XR 5;a;[ [ RRX=SX=SX=S4XU 4S95eXU 4X='MY URU 5nURU 5nUcMUcMX:Xa SU 3X'MS U 3X'[US-U5H nS U 3UU'M S U 3X'M [5nUH)upn [X5HnURU5 M M+ UHMn[URUR[U5-5H nUU;dM M= X(UR'MO SU;aUS:wa[U5n[R"[R R[UR"5S :aUR"SS S -O UR"[U5S :aUSS S -OUS95 U$s snfs snfs snf)u\Encode labelled spans into per-token tags, using the Begin/In/Last/Unit/Out scheme (BILUO). doc (Doc): The document that the entity offsets refer to. The output tags will refer to the token boundaries within the document. entities (iterable): A sequence of `(start, end, label)` triples. `start` and `end` should be character-offset integers denoting the slice into the original string. missing (str): The label used for missing values, e.g. if tokenization doesn’t align with the entity offsets. Defaults to "O". RETURNS (list): A list of unicode strings, describing the tags. Each tag string will be of the form either "", "O" or "{action}-{label}", where action is one of "B", "I", "L", "U". The missing label is used where the entity offsets don't align with the tokenization in the `Doc` object. The training algorithm will view these as missing values. "O" denotes a non-entity token. "B" denotes the beginning of a multi-token entity, "I" the inside of an entity of three or more tokens, and "L" the end of an entity of two or more tokens. "U" denotes a single-token entity. EXAMPLE: >>> text = 'I like London.' >>> entities = [(len('I like '), len('I like London'), 'LOC')] >>> doc = nlp.tokenizer(text) >>> tags = offsets_to_biluo_tags(doc, entities) >>> assert tags == ["O", "O", 'U-LOC', "O"] rDr%rrr )span1span2Nrrrr2z...)textrJ)idxrGr+r/keysr,r E103r.getsetaddstrwarningswarnr W030rO)r8rJr9tokens_in_entsrHstartsendsr6biluor>r?r3s token_index start_token end_tokenrG entity_charsent_strs rr<r<Gs<CEN.1 2cUii cF 29< =IIE "EGG +D = #QS#E '/# e?q|'*E)$ %Z: "5"5"77$ ** . ;A > . ;A > . ;A ># $."? +  0:U.K+ ;!**Z0K*I&9+@++-eWE&+-eWE&";?I>%'w<a?)+E7|E$;(0>5L'/# ez,A   Q -(0uyy%))c%j"89AL :%%''N   e|3h- MM .1#((mb.@SXXcr]U*chh14W1B"- !  Lk3 = s J,J Jcv[U5n/nUH%upEn[XUS-US9nURU5 M' U$)aEncode per-token tags following the BILUO scheme into Span object, e.g. to overwrite the doc.ents. doc (Doc): The document that the BILUO tags refer to. tags (iterable): A sequence of BILUO tags with each tag describing one token. Each tag string will be of the form of either "", "O" or "{action}-{label}", where action is one of "B", "I", "L", "U". RETURNS (list): A sequence of Span objects. Each token with a missing IOB tag is returned as a Span with an empty label. rr3)tags_to_entitiesrr )r8r token_offsetsspansr3 start_idxend_idxspans rbiluo_tags_to_spansrlsG%T*M E%2!'CGaKu= T&3 Lrc[X5nUVs/sH%o3RURUR4PM' sn$s snf)aEncode per-token tags following the BILUO scheme into entity offsets. doc (Doc): The document that the BILUO tags refer to. tags (iterable): A sequence of BILUO tags with each tag describing one token. Each tags string will be of the form of either "", "O" or "{action}-{label}", where action is one of "B", "I", "L", "U". RETURNS (list): A sequence of `(start, end, label)` triples. `start` and `end` will be character-offset integers denoting the slice into the original string. )rlr>r?r@)r8rrhrks rbiluo_tags_to_offsetsrns7  *EFK Led__dmmT[[ 9e LL Ls,?c  /nSn[U5GHyup4UbURS5(aUbSnM&URSX345 M;URS5(aMSURS5(a;Uc6[[R R S[U5SUS-S95eMURS5(aURUS SX345 MURS 5(aUnMURS 5(aTUc6[[R R S [U5SUS-S95eURUS SX#45 SnGMV[[RR US 95e U$) ztNote that the end index returned by this function is inclusive. To use it for Span creation, increment the end by 1.NrDr%r(r)r4rUr Br)r*) rE startswithr r,r E067r.rE068)rrJr4rGr"s rrfrfsnH ED/ ;#..-- Q + ^^C  ^^C } KK&&StDz'AE7J&K^^C OOSWaO , ^^C E ^^C } KK&&StDz'AE7J&K OOSWe/ 0EV[[//C/89 95"6 Orr3c\[[[[4URSS55$NrDr)r rrVsplitres rsplit_bilu_labelrys" c3hS!!4 55rc,URSS5S$rw)rxres rremove_bilu_prefixr{s ;;sA q !!r)r%) rWtypingrrrrrrr errorsr r tokensr rrVrr#rrrBrIintr<rlrnrfryr{offsets_from_biluo_tagsspans_from_biluo_tagsbiluo_tags_from_offsetsrrrsEEE%x}cx}cd3iHSM (tCy(T#Y(,3cTWT T sCsCx'@!ABTMPT #YTnS $t*&M MSMM %S%S/) *+M" 8C= T%S# 2F-G F6C6E#s(O6"c"c" 0+/r