hH SSKrSSKrSSKrSSKJr SSKJr SSKJr SSK J r SSK J r SSK Jr SSKJrJrJrJr SS KJr \R,R/S 5S 5r\R,R/S 5\R,R3S S9\R,R5SS/SQ4S/SQ4S/SQ4S/SQ4S/SQ4S/SQ4S/SQ4/5S555r\R,R/S5S 5r\R,R/S!5S"5r\R,R3S#S9\R,R/S$5S%55r\R,R/S&5S'5r\R,R/S(5S)5r \R,R/S*5S+5r!\R,R/S,5S-5r"\R,R/S.5S/5r#\R,R3S0S9\R,R/S15S255r$\R,R/S35S45r%\R,R5S5S6S7/5\R,R/S85S955r&\R,R/S:5S;5r'\R,R/S<5S=5r(\R,R/S>5S?5r)\R,R3S@S9\R,R/SA5SB55r*\R,R5SCSD/SEQ4SFSF/4/5SG5r+SHr,\R,R5S5SI/5SJ5r-SKr.SLr/SMr0\R,R5S5/SNQ5SO5r1\R,R5S5SP/5SQ5r2\R,R5S5/SRQ5SS5r3STr4\R,R5SUSV/5SW5r5SXr6\R,R5SSISYSZ0SYS[0/4/5S\5r7\R,R5SSISYSZ0SYS]0/4SISZS^S_.SYS[0/4/5S`5r8\R,R5SSISZSaSb.SYS[0/4/5Sc5r9Sdr:Ser;SfrSir?Sjr@SkrASlrB\R,R/Sm5Sn5rCSorDg)pN)German)English)ORTH) Tokenizer)Doc)Example)compile_infix_regexcompile_prefix_regexcompile_suffix_regex ensure_path)Vocabic|[[5SS/5nUSn[U/5n[U5nUSULdeg)Nhelloworldr)rr setlist)doctokensitemss ^/home/james-whalen/.local/lib/python3.13/site-packages/spacy/tests/tokenizer/test_tokenizer.py test_issue743rsD eg) *C FE UG A GE 8u  i!zECan not be fixed unless with variable-width lookbehinds, cf. PR #3218)reasonz text,tokensz"deserve,"--and)"deservez,"--andzexception;--exclusive) exceptionz;-- exclusivezday.--Is)dayz.--Iszrefinement:--just) refinementz:--justz memories?--To)memoriesz?--TozUseful.=--Therefore)Useful.=-- Thereforez=Hope.=--Pandora)=Hoper'PandoracU"U5n[U5[U5:XdeUVs/sHoDRPM snU:Xdegs snf)z;Test that special characters + hyphens are split correctly.Nlentext) en_tokenizerr/tokensrts r test_issue801r3sF$ t C s8s6{ "" " CqFFC F ** * sAi%cSn[5RnU"U5nSUVs/sHo3RPM sn;deSUVs/sHo3RPM sn;deURS[S0/5 U"U5nSUVs/sHo3RPM sn;deSUVs/sHo3RPM sn;de[5RnURS[S0/5 U"U5nSUVs/sHo3RPM sn;deSUVs/sHo3RPM sn;degs snfs snfs snfs snfs snfs snf)z>Test special-case works after tokenizing. Was caching problem.zTI like _MATH_ even _MATH_ when _MATH_, except when _MATH_ is _MATH_! but not _MATH_.MATH_MATH_N)r tokenizerr/add_special_caser)r/r7rws rtest_issue1061r:5sC bD ##I D/C c*cffc* ** * C0CqFFC0 00 0 x4*:);< D/C ,1, ,, , #.#Q&&#. .. . ##I x4*:);< D/C ,1, ,, , #.#Q&&#. .. .+0-. -.s#EE E +EE$Eic4U"S5n[R"[U5S4SS9UlUR 5nUR USS5 SSS5 [U5S:XdeURR S :Xdeg!,(df  N<=f) z(Test that doc.merge() resizes doc.tensorza b c df)dtyperN)r@r<)numpyonesr.tensor retokenizemergeshape)r0r retokenizers rtest_issue1963rHKs~ y !CSXsO37CJ  [#a(#  s8q== ::  x '' '  s B  BzICan not be fixed without variable-width look-behind (which we don't want)ic [5nSnU"U5n[U5S:XdeUSRS:XdeUSRS:XdeUSRS:XdeUS RS :XdeUS RS:Xdeg ) z@Test that g is not split of if preceded by a number and a letterz e2g 2g 52gre2g2r?gr@52N)rr.r/)nlp testwordsrs rtest_issue1235rSVs )CI i.C s8q== q6;;%   q6;;#   q6;;#   q6;;$   q6;;#  ric[5nU"S5n[U5S:Xde[URSS/55n[US5S:Xde[US5S:Xdeg)NrrrL)rr.rpipe)rQrdocss rtest_issue1242rXgse )C b'C s8q== "g' (D tAw<1   tAw<1  ric[[5/SQS9n[[5/SQS9nUSUS:wdeUSUS:Xaeg)z#Test that tokens compare correctly.abcwords)r[r]erN)rr )doc1doc2s rtest_issue1257rcqsO uwo .D uwo .D 7d1g  Aw$q'!! !!ri_c[[5/SQS9n[R"[5 USR S5(deSSS5 USR S5R S:Xde[R"[5 USR S5(deSSS5 USR S5R S :Xdeg!,(df  N=f!,(df  ND=f) zBTest that token.nbor() raises IndexError for out-of-bounds access.)01rMr^rNrLrer?rM)rr pytestraises IndexErrornborr/)rs rtest_issue1375rlzs eg_ -C z "1v{{2 # q6;;r?  3 && & z "1v{{1~~~ # q6;;q>  # %% % # " # "sCC- C*- C;icD^^^^[R"S5m[R"S5m[R"S5m[R"S5mUUUU4Sjn[5nU"U5UlU"S5nUHnUR(aMe g)zBTest that tokenizer can parse DOT inside non-whitespace separatorsz[\[\("']z[\]\)"']z[-~\.]z ^https?://c >[UR0TRTRTRTRS9$)N) prefix_search suffix_searchinfix_finditer token_match)rvocabsearchfinditermatch)rQinfix_re prefix_re simple_url_re suffix_res r my_tokenizer$test_issue1488..my_tokenizers> II #**#**#,,%++   rzThis is a test.Nrecompilerr7r/)r{rQrrrwrxryrzs @@@@rtest_issue1488rs ?+I ?+Izz-(HJJ01M   )C %CM  Czzzzric^[R"S5mS/SQ4SSS/4S/SQ4/nU4S jn[5nU"U5UlUH-up4U"U5Vs/sHoURPM snU:XaM-e g s snf) z&Test if infix_finditer works correctlyz[^a-z]z token 123test)rrfrM3testz token 1testr1testz hello...test)r.rrrcB>[UR0TRS9$)N)rq)rrsru)rQrws r new_tokenizer%test_issue1494..new_tokenizersBx7H7HIIrNr}) test_casesrrQr/expectedrrws @rtest_issue1494rszz-(H :; '*+ 9:J J )C!#&CM$(+D 2 u 2h>>>%2sA:zJCan not be fixed without iterative looping between prefix/suffix and infixicJ[5nU"S5n[U5S:Xdeg)zJTest that checks that a dot followed by a quote is handled appropriately. z.First sentence."A quoted sentence" he said ... N)rr.rQrs rtest_issue2070rs& )C > ?C s8r>>rin cU"S5n[U5S:XdeUSRS:XdeUSRS:XdeUSRS:XdeUS RS :XdeUS RS:XdeUS RS :XdeUSRS:XdeUSRS:Xdeg)z\Test that the tokenizer correctly splits tokens separated by a slash (/) ending in a digit. z"Learn html5/css3/javascript/jqueryrLearnrLhtml5r?/r@css3rPrJ javascriptjqueryNr-) fr_tokenizerrs rtest_issue2926rs ; > q6;;#   q6;;&  q6;;%   q6;;&  q6;;#   q6;;%   q6;;%   q6;;&  q6;;$   q6;;&  r7<<3  ri cxU"S5nUSRS:XdeU"S5nUSRS:Xdeg)zFTest that words like 'a' and 'a.m.' don't get exceptional norm values.r[ramN)norm_)r0r[rs rtest_issue2754rsD SA Q4::   d B a5;;$  ri cJ[5nU"S5n[U5S:Xdeg)z;Test that the tokenizer doesn't hang on a long list of dotszW880.794.982.218.444.893.023.439.794.626.120.190.780.624.990.275.671 ist eine lange ZahlrJN)rr.rs rtest_issue3002rs* (C a C s8q==rz;default suffix rules avoid one upper-case letter before dotiy c[5nURS5 SnSnSnU"U5nU"U5nU"U5nUSRS:XdeUSRS:XdeUSRS:Xdeg)N sentencizerz>He gave the ball to I. Do you want to go to the movies with I?z?He gave the ball to I. Do you want to go to the movies with I?z>He gave the ball to I. Do you want to go to the movies with I?rJr)radd_piper/)rQtext1text2text3t1t2t3s rtest_issue3449rs )CLL LE ME ME UB UB UB a5::   a5::   a5::  rz text,wordszA'B C)A'BCzA-BcFU"U5n[R"USU05 g)Nr_)r from_dict)r0r/r_rs rtest_gold_misalignedrs# t C cGU+,rc6U"S5n[U5S:Xdeg)NrUrr.)r7r1s rtest_tokenizer_handles_no_wordr s r]F v;!  rloremc>U"U5nUSRU:Xdeg)Nrr/r7r/r1s r"test_tokenizer_handles_single_wordr%s! t_F !9>>T !! !rcSnU"U5n[U5S:XdeUSRS:XdeUSRS:XdeUSRS:XdeUSRS:wdeg) Nz Lorem, ipsum.rPrLoremrLrr?ipsumr-rs rtest_tokenizer_handles_punctr+sz D t_F v;!   !9>>W $$ $ !9>>S  !9>>W $$ $ !9>>W $$ $rc:SnU"U5n[U5S:Xdeg)NzLorem, (ipsum).rrrs r#test_tokenizer_handles_punct_bracesr5s" D t_F v;!  rcSS/nSnU"U5nUSRU;a<[U5S:XdeUSRS:XdeUSRS:Xdegg) NhubnzLorem ipsum: 1984.rrJrr@1984)lang_r.r/)r7 exceptionsr/r1s rtest_tokenizer_handles_digitsr;slJ D t_F ayj(6{aay~~(((ay~~''')r)z google.comz python.orgzspacy.ioz explosion.aizhttp://www.google.comc6U"U5n[U5S:XdegNrLrrs rtest_tokenizer_keep_urlsrFs t_F v;!  rz NASDAQ:GOOGc6U"U5n[U5S:Xdeg)Nr@rrs rtest_tokenizer_colonsrOs t_F v;!  r)zhello123@example.comzhi+there@gmail.itzmatt@explosion.aic6U"U5n[U5S:Xdegrrrs rtest_tokenizer_keeps_emailrUst_F v;!  rc:SnU"U5n[U5S:deg)NaLorem ipsum dolor sit amet, consectetur adipiscing elit Cras egestas orci non porttitor maximus. Maecenas quis odio id dolor rhoncus dignissim. Curabitur sed velit at orci ultrices sagittis. Nulla commodo euismod arcu eget vulputate. Phasellus tincidunt, augue quis porta finibus, massa sapien consectetur augue, non lacinia enim nibh eget ipsum. Vestibulum in bibendum mauris. "Nullam porta fringilla enim, a dictum orci consequat in." Mauris nec malesuada justo.rJrrs r test_tokenizer_handles_long_textr]s% ZDt_F v;??r file_namezsun.txtc[[5RU- nURSSS9nUR 5nSSS5 [ W5S:wdeU"U5n[ U5S:deg!,(df  N9=f)Nrutf8)encodingrd)r __file__parentopenreadr.)r7rlocinfiler/r1s r$test_tokenizer_handle_text_from_filerksm h  & & 2C # '6{{} ( t9>> t_F v;   ( 's A11 A?cSnSnU"U5nU"U5nUSRS:XdeUSRS:Xdeg)Nz2Lorem dolor sit amet, consectetur adipiscing elit.z8Lorem ipsum dolor sit amet, consectetur adipiscing elit.rrr)r7rrtokens1tokens2s r(test_tokenizer_suspected_freeing_stringsrusL @E FEGG 1:??g %% % 1:??g %% %rorthloremcURX5 U"U5nUSRUSS:XdeUSRUSS:Xdeg)NrrrLr8r/)r7r/r1rs rtest_tokenizer_add_special_caser~sW t, D/C q6;;&)F+ ++ + q6;;&)F+ ++ +rr~r)rtagc[R"[5 URX5 SSS5 g!,(df  g=f)N)rhri ValueErrorr8rs r$test_tokenizer_validate_special_casers* z """40 # " "s 6 ALO)rnormc [5n[U0SSS5nURX5 U"U5nUSRUSS:XdeUSRUSS:XdeUSRUSS:Xdeg)NrrrrL)r rr8r/r)r/r1rsr7rs r#test_tokenizer_add_special_case_tagrs GE%T46I t, D/C q6;;&)F+ ++ + q6<<6!9V, ,, , q6;;&)F+ ++ +rcSnURSSS0/5 URSSS0/5 U"U5nUVs/sHo3RPM sn/SQ:Xdegs snf)Nz(((_SPECIAL_ A/B, A/B-A/B") _SPECIAL_rA/B) (rrrrrr-rr)rr7r/rrs r)test_tokenizer_special_cases_with_affixesrsj (D {fk-B,CD u&78 D/C$' (C5JJC ( -    (sAc[5Rn0UlSnURSSS0/5 U"U5RU:XdeURSSS0SS0/5 SnU"U5RU:Xdeg)Nz''a'' z''rabr[r\zab ab ab ''ab ab'' ab'' ''ab)rr7rulesr8r/)r7r/s r8test_tokenizer_special_cases_with_affixes_preserve_spacyr s ##IIO D tvtn%56 T?  4 '' 'tvsmfc]%CD )D T?  4 '' 'rcSnURSSS0/5 U"U5nUVs/sHo3RPM snSS/:Xdegs snf)Nz _SPECIAL_.rrrrrs r(test_tokenizer_special_cases_with_periodr sO D {fk-B,CD D/C$' (C5JJC ([#,> >> > (sAcSnURSSS0SS0/5 U"U5nUSRS:XdeUSRS :Xdeg) Nz the _ID'X_z_ID'X_r_IDz'X_rLrPr?r)r8idx)r7r/rs r test_tokenizer_special_cases_idxrsU D x65/FE?)KL D/C q6::?? q6::??rcU"S5Vs/sHoRPM sn/SQ:XdeURSSS0/5 U"S5Vs/sHoRPM snS/:Xdegs snfs snf)Nza b crZr)r/r8)r7r2s r#test_tokenizer_special_cases_spacesrsp%g. /.qFF. /? BB B w&'):(;<%g. /.qFF. /G9 << < 0 /s A* A/c [R"S5n[UURS9nU"S5Vs/sHo3RPM snSS/:XdeSUlU"S5Vs/sHo3RPM snS/:Xdegs snfs snf)N[\.]$)rpza.r[r)r~rrrtr/rp)en_vocabrzr7r2s rtest_tokenizer_flush_cachers 8$I&&I&dO ,OqFFO ,c :: :"I%dO ,OqFFO , 66 6 - ,s B&B c2[R"S5nSSS0/0n[UURUS9nU"S5Vs/sHoDRPM snSS/:Xde0UlU"S5Vs/sHoDRPM sn/SQ:Xdegs snfs snf)Nrza ar)rprza a.r)r[r[r)r~rrrtr/r)rrzr tokenizer1r2s rtest_tokenizer_flush_specialsrs 8$I fe_% &E&&J 'v. /.qFF. /E3< ?? ?J&v. /.qFF. /? BB B 0 /s B.Bc:S/nSS/n[U5n[U5n[UURURS9nU"S5Vs/sHofRPM nnU/SQ:XdeUR S5Vs/sHofSPM nnXx:Xdegs snfs snf)Nza(?=.)z (?<=\w)\.z (?<=a)\d+\.)rorpza10.)r[10rrL)r r rrtr/explain) rprefixessuffixesrxrzr7r2r1explain_tokenss r/test_tokenizer_prefix_suffix_overlap_lookbehindrszHn-H$X.I$X.I&&&&I (/ 0/ff/F 0 % %% %$-$5$5f$=>$=qd$=N>  ## #1>s B;Bc8S/nS/n[U5n[U5n[UURURS9nU"S5Vs/sHofR PM nnUSS/:XdeUR S5Vs/sHofSPM nnXx:Xdegs snfs snf)N±%)rqrpu±10%u±10rL)r r rrurtr/r) rinfixesrrwrzr7r2r1rs rtest_tokenizer_infix_prefixr$sfGuH"7+H$X.I((&&I (0 10ff0F 1 fc] "" "$-$5$5g$>?$>qd$>N?  ## #2?s B:Bif'c.SnSUlU"U5nSUVs/sHo3RPM sn;deSUVs/sHo3RPM sn;deSUlU"U5nSUVs/sHo3RPM sn;degs snfs snfs snf)z5Test special case works when part of infix substring.z No--don't seeFzn'tdoTzdon'tN)faster_heuristicsr/)r0r/rr9s rtest_issue10086r(s D&+L" t C S)SVVS) )) ) C(CqFFC( (( (&*L" t C s+s!vvs+ ++ + *( ,sBB *Bc[U[R"S5RSSS0SS0/0S9nU"S5Vs/sHo"RPM nnUR S5Vs/sHo"SPM nnX4:Xdegs snfs snf)Nz^id$idrid)rrrrL)rr~rrvr/r)rr7r2r1rs r+test_tokenizer_initial_special_case_explainr-#sJJv&,, FC=63-0 I(o .offoF .$-$5$5d$;<$;qd$;N<  ## #/&B)Er~rArh spacy.lang.der spacy.lang.enr spacy.symbolsrspacy.tokenizerr spacy.tokensrspacy.trainingr spacy.utilr r r r spacy.vocabr markissuerskip parametrizer3r:rHrSrXrcrlrrrrrrrrrrrrrrrrrrrrrrrrrr r rrrrrr$r(r-rrrr:s !%" 33 R ;< "CD )* ;< 34  ?@ => +  +4//*4(( V4  44""4&&404??$ W4  4 # #   C c4 4&44VW4 X G12UUG4DE--  '+"," % ( U  -12  N yk23&'VTNVUO3T)U(VW,X, FD>FD>23 D-?@11 Wd;fe_MNO,,( ( ?= 7 C$"$"5 , , $r