h1$SSKrSSKJr SSKJr SSKJrJrJrJ r J r SSK r SSK J r SSKJr SSKJr SS KJrJr SS KJr SS KJr SS KJr SS KJr SSKJrJr SSK J!r! SSKJ"r"J#r#J$r$ SSK%J&r& SSK'J(r( SSK)J*r* SSK+J,r, SSK-J.r. SSK/J0r0 Sr1S7S\ \24Sjjr3"SS\"5r4"SS\5r5"SS \5r6\6RoS!S"S#/\S$S$S%S&0S'.S(S(SS).S*9S+\S,\ S-\2S.\8S/\8S0\ \4 S1j5r9\"S2/S3Q5r:S8S4jr;S5rg):N) namedtuple)Path)AnyCallableDictOptionalUnion)Model)util)Errors) BaseDefaultsLanguage) Morphologizer)DEFAULT_MORPH_MODEL)Scorer)POS)Doc MorphAnalysis)validate_examples)DummyTokenizerload_config_from_strregistry)Vocab) STOP_WORDS)SYNTAX_ITERATORS)TAG_BIGRAM_MAP)TAG_MAP) TAG_ORTH_MAPzU [nlp] [nlp.tokenizer] @tokenizers = "spacy.ja.JapaneseTokenizer" split_mode = null split_modec^U4SjnU$)Nc,>[URTS9$)Nr!)JapaneseTokenizervocab)nlpr!s P/home/james-whalen/.local/lib/python3.13/site-packages/spacy/lang/ja/__init__.pyjapanese_tokenizer_factory4create_tokenizer..japanese_tokenizer_factory$s zBB)r!r)s` r(create_tokenizerr-#sC &%r+c\rSrSrSS\S\\SS4SjjrSrS\S\ 4S jr SS \ 4S jjr S r S rS\\\44Sjr04S\\\4SS4SjjrS\4SjrS\SS4SjrS\\\4SS4SjrS\\\4SS4SjrSrg)r%*Nr&r!returncXlX l[UR5UlUSL=(d US:H(+Ulg)NA)r&r!try_sudachi_import tokenizerneed_subtokens)selfr&r!s r(__init__JapaneseTokenizer.__init__+s6 $+DOO<#-#5#Js9JKr+c>[URUR44$N)r%r&r!r6s r( __reduce__JapaneseTokenizer.__reduce__2s 4::t"???r+textc URRU5nURU5n[X15up4U(a[ U6O//S-upVpxpn [ U 5n [ URXTS9n Sn [[ X55GHununnURUl U (a Xl Sn OB[URURUS-[U5:aXnS-OS5uUl n UR(a URO UR Ul0nUR$(aUR$US'UR&UlUR*(a%[,R."SSUR*5US'[1URU5UlGM" UR4(aXR6S'U $) N)wordsspacesr Inflectionz[=|]_Reading sub_tokens)r4tokenize _get_dtokensget_dtokens_and_spacesziplistrr& enumeratetagtag_pos resolve_posorth_lenlemmasurfacelemma_infnormnorm_readingresubrmorphr5 user_data)r6r>sudachipy_tokensdtokensrBrAtags inflectionslemmasnormsreadingssub_tokens_listdocnext_posidxtokendtokenr\s r(__call__JapaneseTokenizer.__call__5su>>2248##$450?%CM2$( K[%?/$**E9$-c#.?$@ C%EJ$ &1KKJJ%(1Ws4y%8DqMd'# 8 ,2<<6< F2 F2AF7+A4F>#F>- F7cUR(dg/nUGH)nURURRR5n[ U5S:XaUR S5 MUURS:Xa$UR URUS5/5 MURURRR5n[ U5[ U5:Xa&URUS5nUR Xf/5 MUR URUS5URUS5/5 GM, U$)NrBF) r5splitr4 SplitModer2rRappendr!rHr)r6r^rerisub_asub_br_s r(rt!JapaneseTokenizer._get_sub_tokens}s ""%EKK 8 8 : :;E5zQ&&t,C'&&(9(9%(G'HI DNN$<$<$>$>?u:U+"//u=G#**G+=>#** --eU; --eU;&$r+cF[US5 [R"U5$)NzJapaneseTokenizer.score)rrscore_tokenization)r6exampless r(scoreJapaneseTokenizer.scores($=>((22r+cSUR0$Nr!r$r;s r( _get_configJapaneseTokenizer._get_configsdoo..r+configc2URSS5Ulgr)getr!)r6rs r( _set_configJapaneseTokenizer._set_configs **\48r+c B^SU4Sj0n[R"U/5$)NcfgcL>[R"TR55$r:)srsly json_dumpsrr;sr(,JapaneseTokenizer.to_bytes..se&6&6t7G7G7I&Jr+)r to_bytes)r6kwargs serializerss` r(rJapaneseTokenizer.to_bytessJK }}["--r+datac z^SU4Sj0n[R"X/5 [TR5TlT$)NrcN>TR[R"U55$r:)rr json_loads)br6s r(r.JapaneseTokenizer.from_bytes..s$*:*:5;K;KA;N*Or+)r from_bytesr3r!r4)r6rr deserializerss` r(rJapaneseTokenizer.from_bytess1 OP  R0+DOO< r+pathc p^[R"U5nSU4Sj0n[R"X/5 g)NrcN>[R"UTR55$r:)r write_jsonrpr6s r(r+JapaneseTokenizer.to_disk..s(8(8DTR[R"U55$r:)rr read_jsonrs r(r-JapaneseTokenizer.from_disk..s(8(89K(Lr+)r r from_diskr3r!r4rs` r(rJapaneseTokenizer.from_disksA%LM  t"-+DOO< r+)r5r!r4r&r:)T)__name__ __module__ __qualname____firstlineno__rrstrr7r<rrkboolrHrtrrrrrbytesrrr rrr__static_attributes__r,r+r(r%r%*sLeL#L$L@&S&S&P d @43/T#s(^/469$sCx.9$9.E.u3F ,E#t),,4, eCI.=Pr+r%c8\rSrSr\"\5r\r\ r SSSS.r Sr g)JapaneseDefaultsltrF) directionhas_case has_lettersr,N) rrrrrDEFAULT_CONFIGrr stop_wordsrsyntax_iteratorswriting_systemrr,r+r(rrs# !. 1FJ'#(eERNr+rc\rSrSrSr\rSrg)Japanesejar,N)rrrrlangrDefaultsrr,r+r(rrs DHr+r morphologizerz token.morphz token.posTz@scorerszspacy.morphologizer_scorer.v1)model overwriteextendscorerg?)pos_acc morph_micro_fmorph_per_feat)assignsdefault_configdefault_score_weightsr'rnamerrrc .[URXX4US9$)N)rrr)rr&)r'rrrrrs r(make_morphologizerrs.  5)6 r+ru)rTrMrVrSrWrYrFcSSKJnJn URRR URRR URRR URRRS.UnUR5RUS9nU$![a [S5Sef=f)zSudachiPy is required for Japanese support, so check for it. It it's not available blow up and explain how to fix it. split_mode should be one of these values: "A", "B", "C", None->"A".r) dictionaryr4)Nr2rC)modezJapanese support requires SudachiPy and SudachiDict-core (https://github.com/WorksApplications/SudachiPy). Install with `pip install sudachipy sudachidict_core` or install spaCy with `pip install spacy[ja]`.N) sudachipyrr4 Tokenizerrr2rr Dictionarycreate ImportError)r!rr4toks r(r3r3s3%%//11$$..00$$..00$$..00    ##%,,*,=  :   s B(B++CcU[;a[UnX;aX0S4$U(a0X4nU[;a#[UupVUc[U[U4$XV4$[U[S4$)aIf necessary, add a field to the POS tag for UD mapping. Under Universal Dependencies, sometimes the same Unidic POS tag can be mapped differently depending on the literal token or its context in the sentence. This function returns resolved POSs for both token and next_token by tuple. N)r rrr)orthrMnext_tagorth_map tag_bigram current_posrgs r(rPrPs l$  >4' ']  '$2:$> !K"CL% #,, 3< d ""r+c UVs/sHo3RPM nnSRSRU5R55SRUR55:wa&[[R R XS95e/n/nSn[U5S:XaXV4$[UVs/sHoR5(aMUPM sn5S:Xa-UR5(de[XSXSS5/nS/nXV4$[[X@55Hun upUR5(aMXSRU5n U S:a9XX{-n UR[XSXSS55 URS5 X{- nURU 5 URS5 U[U5- nU S-[U5:dMX S-RS:XdMSUS'US- nM U[U5:a3XSn UR[XSXSS55 URS5 XV4$s snfs snf![a( [[R R XS95Sef=f) N)r>rArFr T)rTrvr ValueErrorr E194formatrRr{rurLrJindexr) r_r>gap_tagxrA text_dtokens text_spacestext_poswordirj word_startws r(rIrI sS ' (1YYE ( wwrwwu~##%&"''$**,*??+++CDDLKH 5zQ(( u ;utLLNdu ; < A||~~~%dRT4PQ g (('s5':;>D <<>>  Si..t4J > 56A    a"aD$ O P   u %  "H F#5!CI q53w< GEN$:$:c$A"KO MH/<4#d) OM!b!dKL5!  $$a )< SV[[//T/GHd R SsI2III  2I<r:)r2)rs)?rZ collectionsrpathlibrtypingrrrrr r thinc.apir rr errorsr languagerrpipelinerpipeline.morphologizerrrrsymbolsrtokensrrtrainingrrrrr&rrrrrtag_bigram_maprtag_mapr tag_orth_mapr rrr-r%rrfactoryrrrur3rPrI__all__r,r+r(rsL "77 .%9()BB".*&&#&KK\S|S x   K ($>?            X   X 0#D2%j ,r+