h SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJ r J r SSK J r SSK Jr SSKJrJr SSKJr SSKJr SSKJrJrJrJrJrJrJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r& SSK'r'SSK(r(SS K)J*r* SS K+J,r,J-r-J.r.J/r/ SS K0J1r1 S S K2J3r3J4r4J5r5 S SK6J7r7 S SK8J9r9J:r: S SK;Jr>J?r?J@r@ S SKAJBrBJCrC S SKDJErE S SKFJGrGJHrHJIrI S SKJJKrKJLrLJMrMJNrNJOrO S SKPJQrQ S SKRJSrS S SKTJUrU S SKVJWrW S SKXJYrYJZrZ S SK[J\r\J]r] S SK5J^r^J_r_J`r`JaraJbrbJcrcJdrdJereJfrf S SKgJhrh S SKiJjrjJkrk \\U/\U4rl\"\m5RS- ro\5R"\o5rq\"\m5RS - rr\#"S!5rs"S"S#5rtS$\S%/\S44S&jruS'rv"S(S%5rwS)\xS$\x4S*jry\"S+S,55rz"S-S.\{5r|S/\\YS$\\Y4S0jr}S1\\$\~\U\4\s/\U4S2\\S3\\U4S4\"\x\x\x4S$S4S5jr"S6S75r"S8S95r\"5rg):N) ExitStackcontextmanager)deepcopy) dataclass)chaincycle)Path) default_timer)AnyCallableDictIterableIteratorListNoReturnOptionalPatternSequenceSetTupleTypeVarUnioncastoverload)Pool)ConfigCupyOps Optimizerget_current_ops)convert_recursive)abouttyutil)Literal)ErrorsWarnings) GIT_VERSION)TOKENIZER_INFIXESTOKENIZER_PREFIXESTOKENIZER_SUFFIXES)BASE_EXCEPTIONS URL_MATCH) load_lookups) analyze_pipesprint_pipe_analysisvalidate_attrs) ConfigSchemaConfigSchemaInitConfigSchemaNlpConfigSchemaPretrainvalidate_init_settings)Scorer Tokenizer)Doc) Underscore)Examplevalidate_examples) init_tok2vec init_vocab) _DEFAULT_EMPTY_PIPESCONFIG_SECTION_ORDERSimpleFrozenDictSimpleFrozenList_pipecombine_score_weights raise_errorregistrywarn_if_jupyter_cupy) BaseVectors)Vocab create_vocabzdefault_config.cfgzdefault_config_pretraining.cfg _AnyContextcH\rSrSr%Sr\"\S9r\\S'\ r \ \ \ \4\S'\r\\\\ \4\S'\r\\\\ \4\S'\r\\\\ \4\S'S r\\\S '\r\\\S '0r\ \ \4\S '0r\ \\\ /\ 44\S '\!"5r"\#\ \S'SSSS.r$Sr%g ) BaseDefaultsXzLanguage data defaults, available via Language.Defaults. Can be overwritten by language subclasses by defining their own subclasses of Language.Defaults.  section_orderconfigtokenizer_exceptionsprefixessuffixesinfixesN token_match url_matchsyntax_iteratorslex_attr_getters stop_wordsltrT) directionhas_case has_letters)&__name__ __module__ __qualname____firstlineno____doc__rrArR__annotations__r,rSr strrdictr*rTrrrrr+rUr)rVrWr r-rXrYrZintr setr[rwriting_system__static_attributes__r`H/home/james-whalen/.local/lib/python3.13/site-packages/spacy/language.pyrNrNXs *>?FF?2A$sDJ/A8JHhxc7l 345J8JHhxc7l 345J7HGXhuS'\23 4H&*K(#*$-Ix!-,.d3=).8:d3# 445:5JC #(d4PNrmrNreturnLanguagec SSS[4SjnU$)zRegistered function to create a tokenizer. Returns a factory that takes the nlp object and returns a Tokenizer instance using the language detaults. nlprproc &URRnURRnURRnU(a [R "U5R OSnU(a [R"U5R OSnU(a [R"U5ROSn[URURRUUUURRURRS9$)N)rules prefix_search suffix_searchinfix_finditerrWrX)DefaultsrTrUrVr$compile_prefix_regexsearchcompile_suffix_regexcompile_infix_regexfinditerr9vocabrSrWrX)rrrTrUrVrurvrws rntokenizer_factory+create_tokenizer..tokenizer_factoryps<<((<<((,,&&FN11(;BBTX FN11(;BBTX GN11':CCTX II,,33'') 00ll,,  rmr8)rs rncreate_tokenizerrks  z i " rmcX[RRSU5 [XS9nU$)Nz+Loading lookups from spacy-lookups-data: %s)langtables)r$loggerdebugr.)rrlookupss rnload_lookups_datars&KKCVL4G Nrmc^\rSrSr%Sr\rSr\\ \ S'\ r \ "\RS9r0r\\ S4\ S'SS 0SSS S .S \\\4S \S\\ \4S\\S/\\ /\44S\\S/\4S\SS4SjjjrU4Sjr\S5r\S\\ \44Sj5r\R@S\\ \4SS4Sj5r\S\!4Sj5r"\"R@S\!SS4Sj5r"\S\#\ 4Sj5r$\S\#\ 4Sj5r%\S\#\&\ \'44Sj5r(\S\#\ 4Sj5r)\S\#\&\ \'44S j5r*\S\#\ 4S!j5r+\S\\ \ 44S"j5r,\S\\ \#\ 44S#j5r-\.S$\ S\4S%j5r/\.S$\ S\ 4S&j5r0\.S$\ SS4S'j5r1\.S$\ SSSS4S(j5r2S$\ SS4S)jr3S$\ S\!4S*jr4\.\ "5\5"5\5"5S+\ "5SS,.S$\ S-\\ \4S.\6\ S/\6\ S0\S1\\ \\74S2\\S\4S3jj5r8\.\5"5\5"5S+SS4.S$\ S.\6\ S/\6\ S0\S2\\'S\S5\44 S6jj5r9/S7QS+S8.S9\#\ S:\S\\\ \44S;jjr:S$\ S\'4S<jr;S\ "5SSS=.S>\ S$\\ S?\\ \4S@\\!SA\S\'4 SBjjjr\ S$\\ SG\\\ \4SH\\\ \4SI\\SJ\\SD\SS?\\ \4S@\\!SA\S\'4SKjjjr>SSG\\\ \4SH\\\ \4SI\\SJ\\S\4 SLjjr?S$\ S\4SMjr@\ "5SSN.S$\ S>\ S?\\ \4SA\S\'4 SOjjrASP\ SQ\ SS4SRjrBS$\ S\&\ \'44SSjrCS$\ SS4STjrDS$\ SS4SUjrE\5"5SSV.SW\\ \4SX\6\ SY\\\ \\ \44S\4SZjjrFSS\jrGSSS].SX\\\ \6\ 4S^\\\ \6\ 4SS[4S_jjrHSW\ S\4S`jrISa\\ \\J4S\4SbjrKSa\\ \\J4Sc\LS\4SdjrMSSeSSS\5"5\5"5Sf.Sg\6\NSh\\Si\7Sj\\OSk\\\ \74SY\\\ \\ \44Sl\6\ Sm\6\ 4SnjjjrPSSS\5"5So.Sg\6\NSj\\OSk\\\ \74SY\\\ \\ \44Sl\6\ S\\ \744 SpjjrQSSSq.Sr\\/\6\N4Sj\\OS\O4SsjjjrRSSSq.Sr\\/\6\N4Sj\\OS\O4StjjjrSSSq.Sj\\OS\O4SujjrTSv\\ \'\#\\U/\V44SwjrWSSSSS+Sx.Sg\6\NS\\Sy\\XSY\\\ \\ \44Sz\\\ \4S{\S\\ \44S|jjrYS}rZ\[S~\\\4Sj5r]\^S5S5S5S5S5S.S\6\\ \4S\_S+S\\SX\6\ SY\\\ \\ \44S\S\`\4Sjj5ra\^S5S5S5S5S5S.S\6\&\\ \4\L4S\_SS\\SX\6\ SY\\\ \\ \44S\S\`\&\\L44Sjj5raS+S\5"5SSS.S\\6\\ \4\6\&\\ \4\L44S\S\\SX\6\ SY\\\ \\ \44S\S\\`\\`\&\\L444SjjraSX\6\ 4SjrbS\6\\ \4S\6\S5\`\4S\S\S\`\4 SjrcSSjrd\.04S\e\e\e\ "5SSS.S?\\\ \4\!4S \\\4SX\\ \6\ 4S^\\ \6\ 4Sl\\ \6\ 4S\\ \4S\SA\SS4Sjjj5rfS\ S\ S\6\ SS4Sjrg\[SS\\hS\`\h4Sjj5ri\5"5S.S\\ \j4Sl\6\ SS4Sjjrk\lSX\\ \6\ 4S^\\ \6\ 4S\6\ S\&\ S544Sj5rm\5"5\ "5S.S\\ \j4Sl\6\ S\\ \4SS4Sjjrn\5"5S.Sl\6\ S\J4Sjjro\5"5S.S\JSl\6\ SS4SjjrpSrqU=rr$)rpa:A text-processing pipeline. Usually you'll load this once per process, and pass the instance around your application. Defaults (class): Settings, data and factory methods for creating the `nlp` object and processing pipeline. lang (str): IETF language code, such as 'en'. DOCS: https://spacy.io/api/language Nrerror FactoryMeta _factory_metaTi@Bi) max_lengthmetarcreate_vectors batch_sizer~rrrrrJrroc SSKJn U"5 [RRR 5 [ RUR5Ul [U5Ul SUl SUl 0Ul0Ul[!U["5(d9USLa4[%[&R(R+U[-["5S95eUSLaUR/S05R/S5n [1UR2UR4U S9nU(d-SURS S0n [R6"U 5SnU"U5UlOwUR2(afUR2(aUUR2UR2:wa;[%[&R:R+UR2UR2S 95eXlUR2cUR<R2Ul/Ul[A5Ul!X l"U(d-S URS S 0n [R6"U 5S nU"U5Ul#X`l$[JUl&g) aInitialise a Language object. vocab (Vocab): A `Vocab` object. If `True`, a vocab is created. meta (dict): Custom meta data for the Language class. Is written to by models to add model meta data. max_length (int): Maximum number of characters in a single text. The current models may run out memory on extremely long texts, due to large internal allocations. You should segment these texts into meaningful units, e.g. paragraphs, subsections etc, before passing them to spaCy. Default maximum length is 1,000,000 charas (1mb). As a rule of thumb, if all pipeline components are enabled, spaCy's default models currently requires roughly 1GB of temporary memory per 100,000 characters in one text. create_tokenizer (Callable): Function that takes the nlp object and returns a tokenizer. batch_size (int): Default batch size for pipe and evaluate. DOCS: https://spacy.io/api/language#init r!)register_factoriesNT)r~ vocab_typevectorsname) vectors_namerr)rrr~ tokenizer)'pipeline.factoriesrr$rG_entry_point_factoriesget_allDEFAULT_CONFIGmergedefault_config_configrh_meta_path _optimizer _pipe_meta _pipe_configs isinstancerJ ValueErrorr&E918formattypegetrKrrxresolverE150r~ _componentsrj _disabledrrrrFdefault_error_handler) selfr~rrrrrkwargsrr vectors_cfg tokenizer_cfgs rn__init__Language.__init__s< ; ,,446%++D,?,?@ $Z  /34602%''E,=V[[//eU /TU U D=88Ir266v>L DMM UE!($,,u*=i*HI !)!1!1+!>y!I*51EM ejjtyyEJJ/F !3!3 !3!TUU! 99  DI;=#&5$($,,u*=k*JKM'// >{K )$/$%0"rmc >[TU]"S0UD6 [RURR 5UlURUR SS'g)Nrrrr`)super__init_subclass__rrrxrRrr)clsr __class__s rnrLanguage.__init_subclass__sI !+F++11#,,2E2EF,/HH5!&)rmcUR$N)rrs rnpath Language.paths zzrmc[R"[R5nURR (a1UR RSURR 5 O&UR RSUR 5 UR RSS5 UR RSS5 UR RSU5 UR RSS5 UR RS S5 UR RS S5 UR RS S5 UR RS S5 UR RS [5 URR[URR5URRRURRRURRRS.UR S'[UR 5UR S'[#UR$5UR S'[#UR&5UR S'[#UR(5UR S'UR $)zCustom meta data of the language class. If a model is loaded, this includes details from the model's meta.json. RETURNS (Dict[str, Any]): The meta. DOCS: https://spacy.io/api/language#meta rrpipelineversionz0.0.0 spacy_version descriptionauthoremailurllicensespacy_git_version)widthrkeysrmoderlabels componentsdisabled)r$get_minor_version_ranger" __version__r~rr setdefaultr(vectors_lengthlenrn_keysrrrh pipe_labelslist pipe_namescomponent_namesr)rrs rnr Language.metas44U5F5FG ::?? JJ ! !&$**// : JJ ! !&$)) 4 fj1 i1 o}= mR0 h+ gr* eR( i, 1;?ZZ..4::--.JJ&&--JJ&&++JJ&&++ !  9 $D$4$45 8"&doo!6 :#'(<(<#= < !%dmm!4 :zzrmvaluecXlgr)rrrs rnrrs rmcURRS05 URRS05 URURSS'0n/nURHenUR U5nUR U5nSUR 0UEX'UR(dMJURUR5 Mg [UR5URSS'[UR5URSS'XRS'URSRS05n[X&5nXpRSS'[R"UR5(d0[[ R"R%URS 95eUR$) zTrainable config for the current language instance. Includes the current pipeline components, as well as default training config. RETURNS (thinc.api.Config): The config. DOCS: https://spacy.io/api/language#config rrtrainingrfactoryrrr score_weightsrR)rrrr get_pipe_metaget_pipe_configrdefault_score_weightsappendrrrrEsrslyis_json_serializablerr&E961r)rrr pipe_name pipe_meta pipe_config prev_weightscombined_score_weightss rnrRLanguage.configsx r*  B/&*ii UF# --I**95I..y9K#,i.?.?"O;"OH ...$$Y%D%DE . +/t/C/C*D UJ'*.t}}*= UJ'%- \"||J/33ORH !6}!S4J Z 1))$,,77V[[//t||/DE E||rmcXlgr)rrs rnrRr:s rmcURVVs/sHupXR;dMUPM nnn[U[RR SS9S9$s snnf)zYGet the names of all disabled components. RETURNS (List[str]): The disabled components. rattrrrrrCr&E926r)rr_namess rnrLanguage.disabled>sP&*%5%5P%5'$9O%5PV[[-?-?Z-?-PQQQs AAc^[URR55n[U5$)zOGet names of all available factories. RETURNS (List[str]): The factory names. )r factoriesrrCrrs rn factory_namesLanguage.factory_namesIs& T^^((*+&&rmc`[UR[RR SS9S9$)z_Get all (name, component) tuples in the pipeline, including the currently disabled components. rrr)rCrr&rrrs rnrLanguage.componentsRs.    FKK$6$6L$6$I  rmcURVVs/sHupUPM nnn[U[RR SS9S9$s snnf)zGet the names of the available pipeline components. Includes all active and inactive pipeline components. RETURNS (List[str]): List of component name strings, in order. rrr)rrCr&rrrrrrs rnrLanguage.component_names[sH04/?/?@/?|y/?@V[[-?-?EV-?-WXXAAcURVVs/sHupXR;dMX4PM nnn[U[RR SS9S9$s snnf)zThe processing pipeline consisting of (name, component) tuples. The components are called on the Doc in order as it passes through the pipeline. RETURNS (List[Tuple[str, Callable[[Doc], Doc]]]): The pipeline. rrrr)rnppipess rnrLanguage.pipelineesT%)$4$4P$4DA8O!$4PV[[-?-?Z-?-PQQQs AAcURVVs/sHupUPM nnn[U[RR SS9S9$s snnf)zsGet names of available active pipeline components. RETURNS (List[str]): List of component name strings, in order. rrr)rrCr&rrrs rnrLanguage.pipe_namespsB 04}}=}|y}=V[[-?-?\-?-RSS>rc0nURH"up#URU5RX'M$ [U5$)zGet the component factories for the available pipeline components. RETURNS (Dict[str, str]): Factory names, keyed by component names. )rrrrB)rrrpipes rnpipe_factoriesLanguage.pipe_factoriesys?  #//OI#'#5#5i#@#H#HI  0 **rmc0nURHQup#[US5(aURSLaM'[US5(dM:[UR5X'MS [ U5$)zGet the labels set by the pipeline components, if available (if the component exposes a labels property and the labels are not hidden). RETURNS (Dict[str, List[str]]): Labels keyed by component name. hide_labelsTr)rhasattrrrrrB)rrrrs rnrLanguage.pipe_labelssb**JDt]++0@0@D0HtX&&#DKK0 +  ''rmrc~URU5nU[R;=(d U[R;$)z=RETURNS (bool): Whether a factory of that name is registered.)get_factory_namerGrrr internal_names rn has_factoryLanguage.has_factorys5,,T2 x)))P]h>P>P-PPrmcBURcU$URSU3$)zGet the internal factory name based on the language subclass. name (str): The factory name. RETURNS (str): The internal factory name. .)r)rrs rnrLanguage.get_factory_names( 88 K((1TF##rmcURU5nX R;aURU$XR;aURU$[[RR SUS95e)zGet the meta information for a given factory name. name (str): The component factory name. RETURNS (FactoryMeta): The meta for the given factory name. rrr)rrrr&E967rrs rnget_factory_metaLanguage.get_factory_metasm,,T2 -- -$$]3 3 $$ $$$T* *+++FGGrmc>X RURU5'g)zSet the meta information for a given factory name. name (str): The component factory name. value (FactoryMeta): The meta to set. N)rr)rrrs rnset_factory_metaLanguage.set_factory_metas9>#..t45rmcXR;a'[[RR SUS95eURU$)zGet the meta information for a given component name. name (str): The component name. RETURNS (FactoryMeta): The meta for the given component name. componentr )rrr&r!rrrs rnrLanguage.get_pipe_metas<  &V[[//[t/LM Mt$$rmcXR;a&[[RR US95eURUnU$)zGet the config used to create a pipeline component. name (str): The component name. RETURNS (Config): The config used to create the pipeline component. r)rrr&E960r)rrrs rnrLanguage.get_pipe_configsC )) )V[[//T/:; ;((. rmF)rassignsrequires retokenizesrfuncrr/r0r1rr2c^^^^^^^[T[5(d&[[RR SS95eST;a&[[R R TS95e[T[5(d3[RR ST[T5S9n[U5eS[S[4UUUUUUU4S jjn UbU "U5$U $) aRegister a new pipeline component factory. Can be used as a decorator on a function or classmethod, or called as a function with the factory provided as the func keyword argument. To create a component and add it to the pipeline, you can use nlp.add_pipe(name). name (str): The name of the component factory. default_config (Dict[str, Any]): Default configuration, describing the default values of the factory arguments. assigns (Iterable[str]): Doc/Token attributes assigned by this component, e.g. "token.ent_id". Used for pipeline analysis. requires (Iterable[str]): Doc/Token attributes required by this component, e.g. "token.ent_id". Used for pipeline analysis. retokenizes (bool): Whether the component changes the tokenization. Used for pipeline analysis. default_score_weights (Dict[str, Optional[float]]): The scores to report during training, and their default weight towards the final score used to select the best model. Weights should sum to 1.0 per component and will be combined and normalized for the whole pipeline. If None, the score won't be shown in the logs or be weighted. func (Optional[Callable]): Factory function if not used as a decorator. DOCS: https://spacy.io/api/language#factory r decoratorrr,zdefault configstylercfg_type factory_funcroc >TRT 5nU[R;ac[RRU5n[R "X5(d)[ RRT X S9n[U5e[R"U5nSU;dSU;a&[[ RRT S95e[RRXS9 [T T[T5[T 5[T R!55T T S9nTR#T U5 [%[RR'5[ R(S9TlU$)N)rr2new_funcrrrr,)r2)rrr/r0scoresrr1r)rrGrrr$ is_same_funcr&E004rr get_arg_namesE964registerrr1rrr%rBrE957) r9r existing_funcerr arg_names factory_metar/rrrrr0r1s rn add_factory%Language.factory..add_factorysA006M 2 22 !) 2 2 6 6} E ((EE ++,,! -C%S/)**<8II%y)@ !3!3!3!>??    ' ' ' I&-&w/'116689&;'L  | 4 -""**,FKKCM rm) rrgrr&E963rE853rhE962rr ) rrrr/r0r1rr2rDrGs ``````` rnrLanguage.factorysF$$$V[[//)/DE E $;V[[//T/:; ;.$//++$$&TD^[T [5(a&[[RR T S95eS[ S[4U4SjjnTRT 5nU[R;aq[RRU5nURnU(aUVs/sHoURPM snSOSn[R"UT5(aUnTR!T TT T US9 T$s snf)Nr,rroc>T$rr`)rrrrOs rnr9?Language.component..add_component..factory_funcQs %%rmrrM)rrrr&E965rrg PipeCallablerrGrr __closure__ cell_contentsr$r=r)rOr9rrCclosurecwrappedr/rcomponent_namer2rr0r1s` rn add_component)Language.component..add_componentMs$%% !3!3!3!HII & &  & 006M 2 22!) 2 2 6 6} E '33CJG??!%!1t7K7KD7Q ", "8 " ">   & &rm)r/r0r<r1)rprettyrr_c8[XS9nU(a [X1S9 U$)aRAnalyze the current pipeline components, print a summary of what they assign or require and check that all requirements are met. keys (List[str]): The meta values to display in the table. Corresponds to values in FactoryMeta, defined by @Language.factory decorator. pretty (bool): Pretty-print the results. RETURNS (dict): The data. )r)r/r0)rrr_analysiss rnr/Language.analyze_pipesps!1   4rmcURHup#X!:XdM Us $ [[RR XR S95e)zGet a pipeline component for a given component name. name (str): Name of pipeline component to get. RETURNS (callable): The pipeline component. DOCS: https://spacy.io/api/language#get_pipe ropts)rKeyErrorr&E001rr)rrrr(s rnget_pipeLanguage.get_pipesH%)$4$4 I   %5v{{))t:N:N)OPPrm)rR raw_configvalidate factory_namerRrjrkcFUbUOUn[U[5(d3[RR SU[ U5S9n[ U5e[R"U5(d&[ [RR US95eURU5(dc[RR USRUR5S[R"U5UR S9n[ U5eUR#U5nUR$(a$['UR$5R)U5nUR+U5nU[,R.;aUnXS.UES U0EnX0n [,R0"XS 9n [,R2"S X0US 9S n ['U 5n XS 'U R5S S5 U R5S S5 U R5SS5 U(aU R)U5n XR6U'X$)aCreate a pipeline component. Mostly used internally. To create and add a component to the pipeline, you can use nlp.add_pipe. factory_name (str): Name of component factory. name (Optional[str]): Optional name to assign to component instance. Defaults to factory name if not set. config (Dict[str, Any]): Config parameters to use for this component. Will be merged with default config, if available. raw_config (Optional[Config]): Internals: the non-interpolated config. validate (bool): Whether to validate the component config against the arguments and types expected by the factory. RETURNS (Callable[[Doc], Doc]): The pipeline component. DOCS: https://spacy.io/api/language#create_pipe NrRr6r, create_pipe)rremethodr lang_code)rrrz @factories)rkcfgrrrr)rrhr&rKrrrrrrrE002joinrr$r]rr"rrrrrGrrfillpopr) rrlrrRrjrkrDrrrrresolvedfilleds rnroLanguage.create_pipes0't\&$''++$$8$f$VCS/ !))&11V[[//v/>? ? --++$$!YYt112$))$/)) %CS/ !)),7   # #I445;;FCF--l;   2 2 2(MSvS|]S$##C;s'89HMeT(y <&  5$ 64  \\*-F#)4 %%rm source_namesourcec [U[5(d0[[RR U[ U5S95eURRURR:wa1[R"[RR US95 XR;a`[[RR UUR SSUR S3SR#UR5S95eUR%U5n['US5(aX4lUR*R-5n[.R0"USU5nX`R2U'URR4URR4:waBURR4H(nURR4R7U5 M* XFS 4$) aMCreate a pipeline component by copying it from an existing model. source_name (str): Name of the component in the source pipeline. source (Language): The source nlp object to copy from. name (str): Optional alternative name to use in current pipeline. RETURNS (Tuple[Callable[[Doc], Doc], str]): The component and its factory name. )rr{r,rrrrn)rmodelrerr)rrprr&E945rrr~rwarningswarnr'W113rrfE944rrtrhrrrR interpolater$ copy_configrstringsadd)rrzr{rr source_configrss rncreate_pipe_from_source Language.create_pipe_from_sources&(++V[[//[f/VW W ::  !5!5 5 MM(--..K.@ A 44 4 ""$#[[016;;v3F2GH6#9#9:# {+ 4 I 113 &&}\'B;'OP #.4 ::  !5!5 5\\)) ""&&q)*+++rm)beforeafterfirstlastr{rRrjrkrrrrc[U[5(d3[U5n [RR XS9n [ U 5eUbUOUnX R;a0[ [RR X RS95eSU;a@[R"[RR URS5S95 UbURXUS9upOURUUUU U S9n UR!X4XV5nUR#U5UR$U'UR&R)XU 45 UR+5 U $)aCAdd a component to the processing pipeline. Valid components are callables that take a `Doc` object, modify it and return it. Only one of before/after/first/last can be set. Default behaviour is "last". factory_name (str): Name of the component factory. name (str): Name of pipeline component. Overwrites existing component.name attribute if available. If no name is set and the component exposes no name attribute, component.__name__ is used. An error is raised if a name already exists in the pipeline. before (Union[str, int]): Name or index of the component to insert new component directly before. after (Union[str, int]): Name or index of the component to insert new component directly after. first (bool): If True, insert component first in the pipeline. last (bool): If True, insert component last in the pipeline. source (Language): Optional loaded nlp object to copy the pipeline component from. config (Dict[str, Any]): Config parameters to use for this component. Will be merged with default config, if available. raw_config (Optional[Config]): Internals: the non-interpolated config. validate (bool): Whether to validate the component config against the arguments and types expected by the factory. RETURNS (Callable[[Doc], Doc]): The pipeline component. DOCS: https://spacy.io/api/language#add_pipe r(rrdr)name_in_configr,)rrRrjrk)rrgreprr&E966rrrE007rrr'W119rvrro_get_pipe_indexr"rrinsert_link_components)rrlrrrrrr{rRrjrkbad_valrDpipe_component pipe_indexs rnadd_pipeLanguage.add_pipesMP,,,<(G++$$w$BCS/ !'t\ '' 'V[[//T@T@T/UV V V  MM(--..fjj>P.Q R  ,0+G+G4,H, (NL"--%! .N))&E $ 5 5l C  >,BC rmcXX4S.n[SXX4455S:a0[[RR XPR S95eU(d[ SX1U455(d[UR5$U(ag[U[5(aZXR ;a0[[RR XR S95eUR RU5$[U[5(a]X R ;a0[[RR X R S95eUR RU5S-$[U5[:XaTU[UR5:dUS:a3[RR S XR S 9n[U5eU$[U5[:XaWU[UR5:dUS:a3[RR S X R S 9n[U5eUS-$[[RR XPR S95e) aDetermine where to insert a pipeline component based on the before/ after/first/last values. before (str): Name or index of the component to insert directly before. after (str): Name or index of component to insert directly after. first (bool): If True, insert component first in the pipeline. last (bool): If True, insert component last in the pipeline. RETURNS (int): The index of the new pipeline component. )rrrrc3(# UHoSLv M g7frr`).0args rn +Language._get_pipe_index..WsG*F3$*F)argsrec3(# UHoSLv M g7frr`)rrs rnrr[sQ:P,:Prrrdr!r)diridxrer)sumrr&E006rranyrrrrgrgindexrriE959)rrrrrall_argsrDs rnrLanguage._get_pipe_indexFs %uS G6%*FG G1 L ""7K7K"L  sQ5%:PQQQt''( (   $ $111 KK&&F9M9M&N''--f5 5 s # #000 KK&&E8L8L&M''--e4q8 8&\S T--..&1*kk(( f3G3G)!o%M %[C D,,--kk((U1E1E)!o%19 ++@T@T+UVVrmcXR;$)zCheck if a component name is present in the pipeline. Equivalent to `name in nlp.pipe_names`. name (str): Name of the component. RETURNS (bool): Whether a component of the name exists in the pipeline. DOCS: https://spacy.io/api/language#has_pipe )rr)s rnhas_pipeLanguage.has_pipe}s&&rm)rRrkc XR;a0[[RR XR S95e[ US5(a2[RR [U5US9n[U5eURRU5nURU5 [UR5(aU[UR5:XaURX!X4S9$URUUUUUS9$)aReplace a component in the pipeline. name (str): Name of the component to replace. factory_name (str): Factory name of replacement component. config (Optional[Dict[str, Any]]): Config parameters to use for this component. Will be merged with default config, if available. validate (bool): Whether to validate the component config against the arguments and types expected by the factory. RETURNS (Callable[[Doc], Doc]): The new pipeline component. DOCS: https://spacy.io/api/language#replace_pipe rd__call__r)rrRrk)rrrRrk)rrr&rgrrrE968rr remove_piperrr)rrrlrRrkrDrs rn replace_pipeLanguage.replace_pipes( ++ +V[[//T/PQ Q < , ,++$$tL/A$MCS/ !))//5  4##$$ c$:J:J6K(K==! ==!! ! rmold_namenew_namecXR;a0[[RR XRS95eX R;a0[[R R X RS95eURR U5nX RUS4URU'URRU5URU'URRU5URU'XRSS;a5URSSRU5nX@RSSU'UR5 g)zRename a pipeline component. old_name (str): Name of the component to rename. new_name (str): New name of the component. DOCS: https://spacy.io/api/language#rename_pipe rdr! initializerN) rrr&rgrrrrrrvrrr)rrriinit_cfgs rn rename_pipeLanguage.rename_pipes; // / ""7K7K"L  ++ + ""7K7K"L   & &x 0')9)9!) rmcXR;a0[[RR XRS95eUR R URRU55nURR U5 URR U5 URRS05R US5 XRSS;a!URSSR U5 XR;aURRU5 UR!5 U$)zRemove a component from the pipeline. name (str): Name of the component to remove. RETURNS (Tuple[str, Callable[[Doc], Doc]]): A `(name, component)` tuple of the removed component. DOCS: https://spacy.io/api/language#remove_pipe rd_sourced_vectors_hashesNrr)rrr&rgrrrvrrrrrrrrremover)rrremoveds rnrLanguage.remove_pipes ++ +V[[//T@T@T/UV V""&&t';';'A'A$'GH D! t$ /488tD << -l; ; LL &| 4 8 8 > == NN ! !$ ' rmcXR;a0[[RR XRS95eUR R U5 g)zDisable a pipeline component. The component will still exist on the nlp object, but it won't be run as part of the pipeline. Does nothing if the component is already disabled. name (str): The name of the component to disable. rdN)rrr&rgrrrr)s rn disable_pipeLanguage.disable_pipesE ++ +V[[//T@T@T/UV V 4 rmcXR;a0[[RR XRS95eXR ;aUR RU5 gg)zEnable a previously disabled pipeline component so it's run as part of the pipeline. Does nothing if the component is already enabled. name (str): The name of the component to enable. rdN)rrr&rgrrrrr)s rn enable_pipeLanguage.enable_pipesT ++ +V[[//T@T@T/UV V == NN ! !$ ' !rm)disable component_cfgtextrrc URU5nUc0nURHupVXR;aM [US5(d0[[R R [U5US95eURn[US5(aUR5nU"U40URU05D6n[U[5(aM[[R R U[U5S95e U$![a,n[[RR US95UeSnAf[anU"XVU/U5 SnANSnAff=f)aTApply the pipeline to some text. The text can span multiple sentences, and can contain arbitrary whitespace. Alignment into the original string is preserved. text (Union[str, Doc]): If `str`, the text to be processed. If `Doc`, the doc will be passed directly to the pipeline, skipping `Language.make_doc`. disable (List[str]): Names of the pipeline components to disable. component_cfg (Dict[str, dict]): An optional dictionary with extra keyword arguments for specific components. RETURNS (Doc): A container for accessing the annotations. DOCS: https://spacy.io/api/language#call Nrrget_error_handlerr,)r returned_type) _ensure_docrrrr&E003rrrrrrfE109 Exceptionrr:E005) rrrrdocrproc error_handleres rnrLanguage.__call__s5*t$  M--JD4,, !3!3d4jt!3!TUU 66Mt011 $ 6 6 8  43>-"3"3D""=> c3'' !3!3TRUY!3!WXX!("  G !3!3!3!>?QF 4d3%33 4s$D E 'D33 E EE DisabledPipesc[R"[R[5 [ U5S:Xa#[ US[[45(aUSnURUS9$)a.Disable one or more pipeline components. If used as a context manager, the pipeline will be restored to the initial state at the end of the block. Otherwise, a DisabledPipes object is returned, that has a `.restore()` method you can use to undo your changes. This method has been deprecated since 3.0 r!r)r) rrr'W096DeprecationWarningrrrtuple select_pipesrs rn disable_pipesLanguage.disable_pipes'sS  hmm%78 u:?z%(T5MBB!HE   //rm)renablercUcUc[[R5e[U[5(aU/nUbv[U[5(aU/nUR Vs/sH o3U;dM UPM nnUb6X:wa1[[R RX!UR S95eUnUceUVs/sHoUUR;dMUPM nn[X5$s snfs snf)aDisable one or more pipeline components. If used as a context manager, the pipeline will be restored to the initial state at the end of the block. Otherwise, a DisabledPipes object is returned, that has a `.restore()` method you can use to undo your changes. disable (str or iterable): The name(s) of the pipes to disable enable (str or iterable): The name(s) of the pipes to enable - all others will be disabled DOCS: https://spacy.io/api/language#select_pipes )rrr) rr&E991rrgrE992rrr)rrrr to_disableds rnrLanguage.select_pipes4s >goV[[) ) gs # #iG  &#&& +/??Q?4&>P$?JQ"w'< KK&&%doo' !G"""&Ag$..)@1gAT++RBs! C$.C$:C)C)c[U5UR:a:[[RR [U5URS95eUR U5$)zcTurn a text into a Doc object. text (str): The text to process. RETURNS (Doc): The processed doc. )lengthr)rrrr&E088rr)rrs rnmake_docLanguage.make_docZsQ t9t & ""#d)"P ~~d##rmdoc_likecL[U[5(aU$[U[5(aURU5$[U[5(a$[UR 5R U5$[[RR[U5S95e)z|Create a Doc if need be, or raise an error if the input is not a Doc, string, or a byte array (generated by Doc.to_bytes()).)r) rr:rgrbytesr~ from_bytesrr&E1041rr)rrs rnrLanguage._ensure_docfsy h $ $O h $ $==* * h & &tzz?--h7 7,,$x.,ABBrmcontextc4URU5nX#lU$)z>Call _ensure_doc to generate a Doc and set its context object.)r_context)rrrrs rn_ensure_doc_with_context!Language._ensure_doc_with_contextqsx(  rmg)dropsgdlossesrexclude annotatesexamplesrrrrrrc Ub[[R5eUc0n[U[5(a[ U5S:XaU$[ US5 [U5nUc.URcUR5UlURnUc0n0n [UR5H[un upURU 05 [Xk5X'XkRSU5 XRSUR5 M] URHupX;a)[U S5(aU R "U4SUS.XkD6 US;aVX;aQ[U ["R$5(a2U R&(a!U R(S ;aU R+U5 X;dM[-[/S U5U U UR0XS 9U5H upXlM M [5U5$) aUpdate the models in the pipeline. examples (Iterable[Example]): A batch of examples _: Should not be set - serves to catch backwards-incompatible scripts. drop (float): The dropout rate. sgd (Optimizer): An optimizer. losses (Dict[str, float]): Dictionary to update with the loss, keyed by component. component_cfg (Dict[str, Dict]): Config parameters for specific pipeline components, keyed by component name. exclude (Iterable[str]): Names of components that shouldn't be updated. annotates (Iterable[str]): Names of components that should set annotations on the predicted examples after updating. RETURNS (Dict[str, float]): The updated losses dictionary DOCS: https://spacy.io/api/language#update NrzLanguage.updaterrupdaterr)NF)TFNc38# UHoRv M g7fr predictedregs rnr"Language.update..s9")rrrr)rr&E989rrrr=_copy_examplesrcreate_optimizer enumeraterrrrrrr#TrainableComponent is_trainabler} finish_updateziprDrr_replace_numpy_floats)rrrrrrrrr pipe_kwargsrrrrr s rnrLanguage.updateys: =V[[) ) >F h % %#h-1*<M($56!(+ ;&"&"7"7"9//C  M (7OA|  $ $T2 . ()< =K    * *64 8   ( (t G 8 --JD"wtX'>'> HU$vUATU-''"4)>)>??)) *==&&s+ "99!!.2.H.H*0  GC$'L (0%V,,rm)rrrrc ^ Uc0n[U[5(a[U5S:XaU$[US5 Uc.URcUR 5UlURn[UR 5n[R"U5 Uc0n0m U 4SjnURUl URUl URUl UHDupX;d[U S5(dM0m U R"U4XsS.URU05D6 MF T R5Hun upU"XU 5 M U$)aMake a "rehearsal" update to the models in the pipeline, to prevent forgetting. Rehearsal updates run an initial copy of the model over some data, and update the model so its current predictions are more like the initial ones. This is useful for keeping a pretrained model on-track, even if you're updating it with a smaller set of examples. examples (Iterable[Example]): A batch of `Example` objects. sgd (Optional[Optimizer]): An optimizer. component_cfg (Dict[str, Dict]): Config parameters for specific pipeline components, keyed by component name. exclude (Iterable[str]): Names of components that shouldn't be updated. RETURNS (dict): Results from the update. EXAMPLE: >>> raw_text_batches = minibatch(raw_texts) >>> for labelled_batch in minibatch(examples): >>> nlp.update(labelled_batch) >>> raw_batch = [Example.from_dict(nlp.make_doc(text), {}) for text in next(raw_text_batches)] >>> nlp.rehearse(raw_batch) DOCS: https://spacy.io/api/language#rehearse rzLanguage.rehearsec>X4TU'X4$rr`)keyWdWgradss rn get_grads$Language.rehearse..get_gradssE#J5Lrmrehearser)rrrr=rrrrandomshuffle learn_rateb1b2rrritems)rrrrrrr rrrrrrrs @rnrLanguage.rehearses*> >F h % %#h-1*<M($78 ;&"&"7"7"9//CT]]#u  M  #~~ vv vv JDgdJ&?&?E MM ' :G:K:KDRT:U   "KKMLC! O* rmr get_examplescr[R"[R[5 UR XS9$)Nr')rrr'W089rr)rr(rs rnbegin_trainingLanguage.begin_trainings(  hmm%78|55rmc^Uc;[RRS5 [UR/SQS9mU4Sjn[ US5(d2[ RRS[U5S9n[U5eURR5n[R"US[S 9nUS nUbU"U5 [!XS US US S9 URR&R(SS:a/[+5nURR&R-U5 [ UR.S5(aB[1UR.R2USSSS9nUR.R2"U4SU0UD6 UR4Hfup[7U [8R:5(dM&USR=U 05n [1U R2U SU S9n U R2"U4SU0U D6 Mh UR=S5n U (a%[R"U [>S 9n [AX U5 URC5 X l"UbX l"O"URDcURG5Ul"USnUbU"U5 URD$!["a* [#[ R$RUS S95ef=f)aInitialize the pipe for training, using data examples if available. get_examples (Callable[[], Iterable[Example]]): Optional function that returns gold-standard Example objects. sgd (Optional[Optimizer]): An optimizer to use for updates. If not provided, will be created using the .create_optimizer() method. RETURNS (thinc.api.Optimizer): The optimizer. DOCS: https://spacy.io/api/language#initialize zUNo 'get_examples' callback provided to 'Language.initialize', creating dummy examples)xyz)wordsc4>[R"T05/$r)r< from_dict)rsrnr()Language.initialize..get_examples#s))#r233rmrzLanguage.initialize)rpobjr)schema before_init vocab_datarr)datarr)rr!r)sectionrrrr pretraining after_init)$r$rrr:r~rr&E930rr TypeErrorrRrrGrr3r?IOErrorE884rshaperto_opsrr6rrrr#InitializableComponentrr5r>rrr)rr(rrDrRIr7ops tok_settingsrr p_settings pretrain_cfgPr<rs @rnrLanguage.initialize s   KK  g djj8C 4|Z00++$$,$|2D%CC. ((*   VL1:J K &  "   D \?AiL!I,  ::   # #A &! +!#C JJ   % %c * 4>>< 0 01))+# L NN % %l M M M--JD$ 9 9::|_00r: 3OOZD  E$E*E (zz-0   6JKA ! $  ?!O __ $"335DO|_  ! t C D&++,,Qy\,BC C Ds J""4Kc[5nURRRSS:a%URRR U5 UR H2up4[ US5(dM[UR5Ul M4 UbXl UR$URcUR5Ul UR$)aContinue training a pretrained model. Create and return an optimizer, and initialize "rehearsal" for any pipeline component that has a .rehearse() method. Rehearsal is used to prevent models from "forgetting" their initialized "knowledge". To perform rehearsal, collect samples of text you want the models to retain performance on, and call nlp.rehearse() with a batch of Example objects. RETURNS (Optimizer): The optimizer. DOCS: https://spacy.io/api/language#resume_training r!_rehearsal_model) rr~rrArBrrrr}rLrr)rrrErrs rnresume_trainingLanguage.resume_trainingYs ::   # #A &! + JJ   % %c *--JDt/00(0(<%( ?!O__ $"335DOrmrcXlURH)up#[US5(dMURU5 M+ g)aSet an error handler object for all the components in the pipeline that implement a set_error_handler function. error_handler (Callable[[str, Callable[[Doc], Doc], List[Doc], Exception], NoReturn]): Function that deals with a failing batch of documents. This callable function should take in the component's name, the component itself, the offending batch of documents, and the exception that was thrown. DOCS: https://spacy.io/api/language#set_error_handler set_error_handlerN)rrrrP)rrrrs rnrPLanguage.set_error_handlerrs6&3"--JDt011&&}5(rm)rscorerr scorer_cfg per_componentrRrSrTc"[U5n[US5 [U5nUc URnUc0nUc0nUc([ U5nUR SU5 [ S0UD6n[5nUH(n URU RR5 M* URSU5UUS9n [X5H upXl M [5n URXS9n [SU55nXU- - U S'[!U 5$) axEvaluate a model's pipeline components. examples (Iterable[Example]): `Example` objects. batch_size (Optional[int]): Batch size to use. scorer (Optional[Scorer]): Scorer to use. If not passed in, a new one will be created. component_cfg (dict): An optional dictionary with extra keyword arguments for specific components. scorer_cfg (dict): An optional dictionary with extra keyword arguments for the scorer. per_component (bool): Whether to return the scores keyed by component name. Defaults to False. RETURNS (Scorer): The scorer containing the evaluation results. DOCS: https://spacy.io/api/language#evaluate zLanguage.evaluaterrc38# UHoRv M g7frrrs rnr$Language.evaluate..s -Hb\\Hr )rr)rTc3L# UHn[UR5v M g7fr)rrrs rnrrWs;(Bc",,''(s"$speedr`)rr=r rrhrr7timerr referencerrrrscorerr)rrrrRrrSrTr start_timer docsrend_timeresultsn_wordss rnevaluateLanguage.evaluates6>($78!(+  J  M  J >*%F   eT *%f%FW B MM",,++ ,yy -H -!'  8*GBL+7,,x,E;(;;"&;<$W--rmc\SURSS0n[R"U5S$)zCCreate an optimizer, usually using the [training.optimizer] config. optimizerr)rRrGr)r subconfigs rnrLanguage.create_optimizers0 $++j"9+"FG  *;77rmparamsc#z# U(dSv gURVVs/sH<up#[US5(dM[US5(dM+URU5PM> nnnUHn[U5 M Sv UHn[U5 M gs snnf![a M@f=f![a M8f=f7f)aReplace weights of models in the pipeline with those provided in the params dictionary. Can be used as a contextmanager, in which case, models go back to their original weights after the block. params (dict): A dictionary of parameters keyed by model ID. EXAMPLE: >>> with nlp.use_params(optimizer.averages): >>> nlp.to_disk("/tmp/checkpoint") DOCS: https://spacy.io/api/language#use_params N use_paramsr})rrrjnext StopIteration)rrhrrcontextsrs rnrjLanguage.use_paramss #'--"/JD4.(3:43I('"/ $M$ #M$% %scB;BB B!B;* B5 B; B* B; B'#B;&B''B;* B84B;7B88B;) as_tuplesrrr n_processtextsrorpcgrr`rrqrorrrrps rnr Language.pipe rmcgrr`rss rnrrtrurmr!c #^# U(ar[[[[[[ 4[ 4U5nU4SjU5nTRUUUUUS9nUHn U Rn SU lX4v M g[[[[[ 4U5nUS:Xa[R"5nUc0nUc TRn/n TRHgupX;aM URU 05nURSU5 [R "["U U UTR$S9nU R'U5 Mi US:waMTR)U5(a$[*R,"[.R05 TR3XXc5nOU4SjU5nU H nU"U5nM UHn U v M g7f) aProcess texts as a stream, and yield `Doc` objects in order. texts (Iterable[Union[str, Doc]]): A sequence of texts or docs to process. as_tuples (bool): If set to True, inputs should be a sequence of (text, context) tuples. Output will then be a sequence of (doc, context) tuples. Defaults to False. batch_size (Optional[int]): The number of texts to buffer. disable (List[str]): Names of the pipeline components to disable. component_cfg (Dict[str, Dict]): An optional dictionary with extra keyword arguments for specific components. n_process (int): Number of processors to process texts. If -1, set `multiprocessing.cpu_count()`. YIELDS (Doc): Documents in the order of the original text. DOCS: https://spacy.io/api/language#pipe c3J># UHupTRX5v M g7fr)r)rrrrs rnr Language.pipe.. s%"RW--d<# UHnTRU5v M g7fr)r)rrrs rnrrySs=utD$$T**us!)rrrrrgr:rLrrmp cpu_countrrrr functoolspartialrDrr_has_gpu_modelrrr'W114_multiprocessing_pipe)rrqrorrrrpdocs_with_contextsr^rrr rrrfrs` rnrrts8 %c3h(D"EFNE"RW" 99"%#+ D,,# n$ XeCHo.6 ? I  M  J  --JD"&&tR0F   lJ 7!!&*&@&@ A LLO( >""7++ hmm,--eIRD>u=DDzCIsGGc>URHup#[US5=(a URnX!;dU(dM2[US5(dME[URS5(dMb[ URR [ 5(dM g g)Nrr}rETF)rrrr}rrEr)rrrrrs rnrLanguage._has_gpu_modelYsr--JD"48NT=N=NLltW%%'$**e*D*DTXT^T^TbTbdkIlIl (rmr c ## S[[[[4S[[[[[ 4[ 44SjnU"U5n[R"U5up[U5Vs/sHn[R"5PM n n[[U5Vs/sHn[R"S5PM sn6up[R"X5n [!XUS9n U R#5 U R#5 [X5VVs/sH@up[R$"[&UR(UUU[*R,"54S9PMB nnnUHnUR/5 M U HnUR15 M [2R4"S[7U 555n[9[UU5S5HunununnnUb0[UR:5R=U5nUUlUv OQUbN[@RB"U5nURESSS[G[HRJRMUS 955 UU-S :XdMU RO5 M U H(nURQ[R5 UR15 M* U HnUR15 M UHnURU5 M [WS U55(d%[XRZ"[\R^5 ggs snfs snfs snnf!U H(nURQ[R5 UR15 M* U HnUR15 M UHnURU5 M [WS U55(d%[XRZ"[\R^5 ff=f7f) Nrqroc3# UHZn[U[5(a.UR5[[UR 54v MFU[[S54v M\ g7fr)rr:to_bytesrrLr)rqrs rn prepare_input5Language._multiprocessing_pipe..prepare_inputksQ "h,,#,,.[(BSBS0TUU#T+t%<== "sA"A$F) chunk_size)targetrc3@# UHoR5v M g7fr)recv)rrs rnr1Language._multiprocessing_pipe..s* $;DIIKK$;sr!rrc3># UHoRS:Hv M g7f)rN)exitcode)rrs rnrrsE#s(O, > eE#u*-{:; < >%2%$8!$==)BC7C#*CLI+!// ;E..dD*V[[5G5Ge5G5T*Uz>Q&KKM<&)* & & RUUS9n[ U5eSU;a&URS5nURAUUUUUS9 MSU;deUcURBRESS/S9nUSn U U;a$[RF"U URBS/S9UU 'URS U5n!S!n"S"U;aZUU RI5 UU RJH4un#n$U![MU$S#/5;dMUU ROU#U!US"5 S$n"M6 [PRR"5 [PRT"S%S&S'9 URAU!UU US(9 SSS5 U U;a3[WUU RBRXRES/S95UU 'S)URZ;a0URZS)'UU URZS)U'U"(dGMjUU GMp UbURBR]U5 [)U[^5(aU/n[)U[^5(aU/n[)T[^5(aT/m[aU5[a[b5:wavUSRS*/5n%[eU%5(aQ[gU%5RiU5(d2[PRj"[lRnRUU%S+95 URq[s1UkUSRS,/5k5UUSS5n&[gU4S-jU&55Ul:USS.Ul;U(aU OUUl<UbIU"U5n[)UU5(d0[ [ R2RS/[+U5S95eU$!,(df  GND=f)0aCreate the nlp object from a loaded config. Will set up the tokenizer and language data, add pipeline components etc. If no config is provided, the default config of the given language is used. config (Dict[str, Any] / Config): The loaded config. vocab (Vocab): A Vocab object. If True, a vocab is created. disable (Union[str, Iterable[str]]): Name(s) of pipeline component(s) to disable. Disabled pipes will be loaded but they won't be run unless you explicitly enable them by calling nlp.enable_pipe. enable (Union[str, Iterable[str]]): Name(s) of pipeline component(s) to enable. All other pipes will be disabled (and can be enabled using `nlp.enable_pipe`). exclude (Union[str, Iterable[str]]): Name(s) of pipeline component(s) to exclude. Excluded components won't be loaded. meta (Dict[str, Any]): Meta overrides for nlp.meta. auto_fill (bool): Automatically fill in missing values in config based on defaults and function argument annotations. validate (bool): Validate the component config and arguments against the types expected by the factory. RETURNS (Language): The initialized Language class. DOCS: https://spacy.io/api/language#from_config rPrrrrz@vectorszspacy.Vectors.v1rN) bad_lang_coderqrrr;)rkr6rbefore_creationafter_creationafter_pipeline_creation)r)r~rrrcreation)rrrrnrdrr{)rrR)rrRrkrjrrr)r~rr(Freplace_listenerslistening_componentsTignorez\[W113\])message)r{rrenabled)rrrc36># UHoT;dM Uv M g7frr`)rr rs rnr'Language.from_config..sJ~!'9IAA~s  rpipeline_creation)=rrrArrr&E985rrrE958r$r]rrvrGrur2rr4rr issubclassE943rHE942is_interpolatedrrtrE956E984rr~r load_modelrrgetattrrrcatch_warningsfilterwarningshashrrrrgidr@rrjissubsetrr'W123_resolve_component_statusrrrrR)'rrRr~rrrrrrk config_lang orig_pipelineorig_pretrainingrx resolved_nlprrrrrlang_clsrr interpolatedr source_nlpssource_nlp_vectors_hashesvocab_brrepipe_cfgrjrDrr}rzlisteners_replacedrrrdisabled_pipess' ` rn from_configLanguage.from_configsF ""2FeFm   V[[//v/>? ? F5M )(24F'GF5M) $Um''/  "{hh'> """(-"7!hh--c2# !$u f !!&) <4 !::mT:!| ]]6\RFF,|,|  '$4= !$4= !'' 5MH_ ( 4%i0&'89%&67"./H"I  &&s+Hx..!(C003& !3!3$x.!3!IJJ  -)    % %Cc3'' !3!34PS9!3!UVV4:3I3Iv))+v ##L"5 $&!z2I(yy1 !3!3!3!NOO''(;@!:;2%8HH67!*)'.Q3T   II  ) gs # #iG fc " "XF gs # #iG f:01 1Um'' 26G7||CL$9$9&$A$A MM((% ')66 ?7?VE]..z2>? @  5M* %  J~JJ |4(Vf " .)#.Cc3'' KK&&,?tCy&Q {32s (*[ [ tok2vec_namer listenerscXR;aD[RRUUUSR UR5S9n[ U5eX R;aD[RRUUUSR UR5S9n[ U5eUR U5nURU5n[U[R5(d0[ [RRU[U5S95eURnURRU/5nUR U5n UR Un U(Ga["R$R'SU5 [)[+U55[)U5:wa4[R,RUUU[)U5S9n[ U5eUHgn ["R."X5 USn SUR4;a!UR4Sn U "USU SS 5n ["R6"XU 5 Mi UHnUR95nUR4RS 5nUbn[)[:R<"U5R>5nUS :Xa U"U5nO6US :Xa U"XU5nO&[ [R@RUS95e["RB"U RX5 UREX5 M g g ![0a* [R2RX!U S9n[ U5ef=f)a%Find listener layers (connecting to a token-to-vector embedding component) of a given pipeline component model and replace them with a standalone copy of the token-to-vector layer. This can be useful when training a pipeline with components sourced from an existing pipeline: if multiple components (e.g. tagger, parser, NER) listen to the same tok2vec component, but some of them are frozen and not updated, their performance may degrade significantly as the tok2vec component is updated with new data. To prevent this, listeners can be replaced with a standalone tok2vec layer that is owned by the component and doesn't change if the component isn't updated. tok2vec_name (str): Name of the token-to-vector component, typically "tok2vec" or "transformer". pipe_name (str): Name of pipeline component to replace listeners for. listeners (Iterable[str]): The paths to the listeners, relative to the component config, e.g. ["model.tok2vec"]. Typically, implementations will only connect to one tok2vec component, [model.tok2vec], but in theory, custom models can use multiple listeners. The value here can either be an empty list to not replace any listeners, or a complete (!) list of the paths to all listener layers used by the model. DOCS: https://spacy.io/api/language#replace_listeners rn)tok2vecrunknownre)rrz%Replacing listeners of component '%s')rrpaths n_listeners)rrrr}replace_listener_cfgrreplace_listenerNr!) num_params)#rr&E889rrtrrhrrr#rE888rr}rrrr$rrrrE887 dot_to_objectrfE886attrsset_dot_to_objectcopyinspect signature parametersE1055replace_model_noderemove_listener)rrrrrDr tok2vec_cfg tok2vec_modelpipe_listenersrr listener_path new_config replace_funclistener new_modelreplace_listener_funcrs rnrLanguage.replace_listenerss:  .++$$$$YYt/ %C S/ ! OO +++$$$!YYt/ %C S/ !-- -**<8 '2#9#9::V[[//\W /VW W  --11)R@}}Y'%%i0  KK  Ey Q4 ?#s>'::kk(("(# #N 3 ) !o%"+ *&&x? )1 )]-@-@@#0#6#67M#NL!-#G,hw.? .J"J&&x K"+"+)..0 (5(;(;(?(?@R(S%(4"%))*?@KK"J"Q$9)$D #q$9)w$W ()<)< )<)STT'' HH''<#+A & * ++,,&=-C%S/) *s #L##4Mmemc #"# Uc [5n[5nURURR U55/n[ UR S5(a9URURUR R U555 URHGupE[ US5(dMURURUR U555 MI Uv SSS5 g!,(df  g=f7f)a:Begin a block where all resources allocated during the block will be freed at the end of it. If a resources was created within the memory zone block, accessing it outside the block is invalid. Behaviour of this invalid access is undefined. Memory zones should not be nested. The memory zone is helpful for services that need to process large volumes of text with a defined memory budget. Example ------- >>> with nlp.memory_zone(): ... for doc in nlp.pipe(texts): ... process_my_doc(doc) >>> # use_doc(doc) <-- Invalid: doc was allocated in the memory zone N memory_zone) rr enter_contextr~rrrrr)rrstackrmrrs rnrLanguage.memory_zone2s$ ;&C[E++DJJ,B,B3,GHIHt~~}55 3 3DNN4N4Ns4S TU==4//OOE$7$78H8H8M$NO)I[[s#DB C>>7C>5 D> D Drrc ^^[R"U5n0nU4SjUS'U4SjUS'U4SjUS'TRH(upEUT;aM [US5(dM U4SjX4'M* UU4S jUS '[R"XT5 g ) a1Save the current state to a directory. If a model is loaded, this will include the model. path (str / Path): Path to a directory, which will be created if it doesn't exist. exclude (Iterable[str]): Names of components or serialization fields to exclude. DOCS: https://spacy.io/api/language#to_disk c:>TRRUS/S9$Nr~r)rto_diskr rs rn"Language.to_disk..`s!T^^-C-C y.D. rmrcX>[R"U[TR55$r)r write_jsonrrrs rnrr csU-=-= $TYY/. rm meta.jsonc:>TRRU5$r)rRrrs rnrr fsdkk.A.A!.Drm config.cfgrc$URUS/S9$r)rr rs rnrr lsT\\!gY\5Wrmc8>TRRUTS9$Nr)r~r)r rrs rnrr ms););Aw);)Ormr~N)r$ ensure_pathrrr)rrr serializersrrs` ` rnrLanguage.to_diskRs% $ K $ K %E L!**JDw4++/3 WK  + P G T0rmrcT[U[5(aU/nUnU(ax[U[5(aU/n1UVs/sH oDU;dM UPM snkUkn[[U5U-5(a&[ [ R RXS95e[U5$s snf)a&Derives whether (1) `disable` and `enable` values are consistent and (2) resolves those to a single set of disabled components. Raises an error in case of inconsistency. disable (Union[str, Iterable[str]]): Name(s) of component(s) or serialization fields to disable. enable (Union[str, Iterable[str]]): Name(s) of pipeline component(s) to enable. pipe_names (Iterable[str]): Names of all pipeline components. RETURNS (Tuple[str, ...]): Names of components to exclude from pipeline w.r.t. specified includes and excludes. )rr) rrgrrjrr&E1042rr)rrrrrs rnr"Language._resolve_component_statusps$ gs # #iG &#&& -7SZ F;R)ZSJ 3v;+,, !4!4F!4!TUUZ  Ts B% B%)r overridesr0c^^^S[SS4U4SjjnS[SS4UU4Sjjn[R"U5n0n[US- 5R5(a UU4SjUS'XFS'XVS 'U4S jUS 'TRH(upxUT;aM [ US 5(dM U4S jXg'M* US - R5(dS T;a[ T5S /-m[R"XT5 UTlTR5 T$)auLoads state from a directory. Modifies the object in place and returns it. If the saved `Language` object contains a model, the model will be loaded. path (str / Path): A path to a directory. exclude (Iterable[str]): Names of components or serialization fields to exclude. RETURNS (Language): The modified `Language` object. DOCS: https://spacy.io/api/language#from_disk rroNc>UR5(al[R"U5nTRR U5 UR S05R S5TR RlggNrr) existsr read_jsonrrrr~rr)rr9rs rndeserialize_meta,Language.from_disk..deserialize_metas]{{}}t,   &+/((9b*A*E*Ef*M ""' rmcf>UR5(aTRRUTS9 ggr))r4r~ from_disk)rrrs rndeserialize_vocab-Language.from_disk..deserialize_vocabs*{{}} $$T7$;rmr%c:>TRRUSTS9$)NF)rr0)rRr9)r r0rs rnr$Language.from_disk..s!DKK4I4Iu 5J5rmr#r~c:>TRRUS/S9$r)rr9rs rnrr=s!t~~/G/G y0H0 rmrr9c$URUS/S9$r)r9r's rnrr=st~~G98F8rm) r r$r*r4rrrr9rr) rrrr0r6r: deserializersrrs ` `` rnr9Language.from_disks$ N4 ND N TRRTS9$r))r~r)rrsrnr#Language.to_bytes..stzz':':7':'Krmr~c8>TRRS/S9$r)rrrsrnrrDs4>>+B+BG9+B+UrmrcV>[R"[TR55$r)r json_dumpsrrrsrnrrDs5+;+; !$)) ,, rmr#c8>TRR5$r)rRrrsrnrrDsDKK,@,@,Brmr%rc"URS/S9$r)r)rs rnrrDs$-- -2Rrm)rrr$r)rrr+rrs`` rnrLanguage.to_bytess79 K G#U K $ K %C L!**JDw4,,,0 RK  + }}['22rm bytes_datac^^U4Sjn0nU4SjUS'X4S'UU4SjUS'U4SjUS'TRH(upVUT;aM [US 5(dM U4S jXE'M* [R"XT5 TR 5 T$) zLoad state from a binary string. bytes_data (bytes): The data to load from. exclude (Iterable[str]): Names of components or serialization fields to exclude. RETURNS (Language): The `Language` object. DOCS: https://spacy.io/api/language#from_bytes c>[R"U5nTRRU5 UR S05R S5TR R lgr3)r json_loadsrrrr~rr)br9rs rnr6-Language.from_bytes..deserialize_metasO##A&D II  T "'+hhy"&=&A&A&&IDJJ   #rmc8>TRRUSS9$)NF)r)rRrrOrs rnr%Language.from_bytes..s 0F0F 51G1 rmr%r#c8>TRRUTS9$r))r~r)rOrrs rnrrSs4::+@+@G+@+Trmr~c:>TRRUS/S9$r)rrrRs rnrrSs!t~~/H/H y0I0 rmrrc$URUS/S9$r)r)rOrs rnrrS stG98G8rm)rrr$rr)rrKrr6r@rrs` ` rnrLanguage.from_bytess J<> ' l#&6k"!T g& k"**JDw4..15#M  +  7;  rm)rrrrrrrrrrrrrr~)Tr)NNNN)rorroN)srarbrcrdrerNrxrrrgrfrrrBr&rBrrr rrJboolrir r r:rIrrpropertyrrsetterrrRrrrrrTrrrrrr classmethodrrr"r%rrrCrfloatrr(r/rhrorrrrrrrrrrrrrrrrLrr<rrrr+rrMrrrPr7rbrrrhrjrr%rrrrrr@rrrrr r staticmethodrr9rrrl __classcell__)rs@rnrprpsHD(3-#N v{{3I.0M4]*+0%)F1 !SWEIF1UD[!F1 F1 38n F1 #8ZL(C5#::N,N#OP F1!7)[*@!ABF1F1 F1P5 #d38n##J [[$sCx.T   D ]]FtR$s)RR'tCy'' DsL'8!9:  YcYYR$uS,%678RRTDITT+S#X++ (T#tCy.1 ( (QsQtQQ $C$C$$ HC HM H H>C> >$>>%#%-% C F  *:);!1!3"2"4!7#>i > c5j)* >  S$sCx.%8 9: >#> c5j >DCG6$( 6xHW,=(=>?6i 6  6CGJ$( JxHW,=(=>?Ji J  JX=Ahy&9Y26lDIy I8 ST6,%)#'=A/3#9.7#9.SM 9.  9.  S$sCx.%8 9: 9.T#s(^,9.9. c3h9.v8 !$!!F %($'!$=@  c3h(  5>  SM  #   S$sCx.%8 9:     #     $'$'!$=@  eCHo{:;<  4=  SM  #   S$sCx.%8 9:     %[() *    $ $(!1!3=AU U38_ %xeCHo{6R0S'T T U  USMU#U S$sCx.%8 9:UU x}huS+-='>?? @Un hsm f-c3h(f-hsm!345f- f-  f- # f-P0,13e%)-A,@-A/1ed38nf,-eUD[! e sHSM)* e c8C=() esHSM)*e38neee eeNf=f=f=C= f=  f=Px~$@CSBT1#t)$12:3-1 1< !sHSM)* !c8C=() !SM ! sCx ! !L"2!3$4$6 7CI7# 7 S> 7  7r4D3E38C=3%30>N=O&&-5c]& &&rm meta_dictc2[SS[U55$)Nc6[U[R5$r)rnumpyfloatingvs rnr'_replace_numpy_floats.. s*Q/rmc[U5$r)r]res rnrrg s58rm)r rh)r`s rnrr s /1CT)_ rmc\rSrSr%Sr\\S'Sr\\ \\ 4\S'\ "5r \ \\S'\ "5r\ \\S'Sr\\S '\ "5r\ \\S 'Sr\\ \\\4\S 'S rg) ri aDataclass containing information about a component and its defaults provided by the @Language.component or @Language.factory decorator. It's created whenever a component is defined and stored on the Language class for each component instance and factory instance. rNrr/r0Fr1r<rr`)rarbrcrdrergrfrrr r rr/rr0r1rYr<rr]rlr`rmrnrr s} L/3NHT#s(^,3"WGXc]$#gHhsm%K!GFHSM#BF8Dhuo)=$>?FrmrcJ\rSrSrSrS\S\\SS4SjrSr S r S S jr S r g) ri# z)Manager for temporary pipeline disabling.rrrroNcXlX lURHnURRU5 M [R U5 UR UR5 gr)rrrrrrextend)rrrrrs rnrDisabledPipes.__init__& sH JJD HH ! !$ ' d DJJrmcU$rr`rs rn __enter__DisabledPipes.__enter__. s rmc$UR5 gr)restore)rrs rn__exit__DisabledPipes.__exit__1 s  rmcURH]nXRR;a&[[R R US95eURRU5 M_ /USS&g)zARestore the pipeline to its state when DisabledPipes was created.r,N)rrrrrr&E008rrr)s rnrrDisabledPipes.restore4 sZJJD88333 !3!3!3!>?? HH  &Qrm)rrrrX) rarbrcrdrerprrgrrorsrrrlr`rmrnrr# s23 H T#Y 4 rmrrcUVs/sH1n[URR5UR5PM3 sn$s snf)zMake a copy of a batch of examples, copying the predicted Doc as well. This is used in contexts where we need to take ownership of the examples so that they can be mutated, for instance during Language.evaluate and Language.update. )r<r.rr/)rr s rnr r = s1 2: :2GBDDIIK & :: :s8A ensure_docr .underscore_statec^[R"U5 UR5n[U[5(a!UR 5 UR 5 gU4SjU5nUH nU"U5nM UVs/sHoR 5URS4PM! n nS/[U5[U 5- -n X-n URU 5 Ms snf![aF SS[R"[R"554/n S/[W5S- -n X-n Nhf=f![a# UR 5 UR 5 gf=f)aWorker for Language.pipe ensure_doc (Callable[[Union[str, Doc]], Doc]): Function to create Doc from text or raise an error if the input is neither a Doc nor a string. pipes (Iterable[Pipe]): The components to apply. receiver (multiprocessing.Connection): Pipe to receive text. Usually created by `multiprocessing.Pipe()` sender (multiprocessing.Connection): Pipe to send doc. Usually created by `multiprocessing.Pipe()` underscore_state (Tuple[dict, dict, dict]): The data in the Underscore class of the parent. Nc38># UHupT"X5v M g7frr`)rrrrys rnr_apply_pipes..d sGU2C( 8--~s)NNNr!)r; load_staterr_WorkDoneSentinelrrrrrr msgpack_dumps traceback format_excrBrokenPipeError) ryr receiverrrztexts_with_ctxr^rr byte_docspaddingr9 error_msgs ` rnrrF sU&*+  '%\\^N.*;<<  GUDDzJNN#,,.#,,=IN)*c..AC N.RSG#   KK 7  O  'e&9&9):N:N:P&QRSI)*c..AA.EFG&D '  LLN NN    s=AC! !C!&C'!C! D4C!!A D10D14*E! E!cf\rSrSrSrS\\S\\RS\ SS4Sjr S S jr S S jr S rg) ri zAUtil for sending data to multiprocessing workers in Language.piper9queuesrroNcp[U5Ul[[U55UlX0lSUlg)Nr)iterr9rrrcount)rr9rrs rnr_Sender.__init__ s+J 5=) $ rmc[R"[UR[ UR 55UR 5HupURU5 M g)z1Send chunk_size items from self.data to channels.N)rislicerr9rrrr)ritemrs rnr _Sender.send sC ''  5- . GD EE$K  rmcU=RS- slURUR:aSUlUR5 gg)z^Tell sender that comsumed one item. Data is sent to the workers after every chunk_size calls. r!rN)rrrrs rnr _Sender.step s6 a :: (DJ IIK )rm)rrr9rrX)rarbrcrdrerr rr|rrirrrrlr`rmrnrr s@KSM+/>GJ rmrc\rSrSrSrg)ri r`N)rarbrcrdrlr`rmrnrr srmr)r~rrmultiprocessingr|r rr contextlibrrrr dataclassesrrrpathlibr timeitr rZtypingr r r rrrrrrrrrrrrrrcr cymem.cymemr thinc.apirrrr thinc.utilr rr"r#r$compatr%errorsr&r'git_infor(lang.punctuationr)r*r+lang.tokenizer_exceptionsr,r-rr. pipe_analysisr/r0r1schemasr2r3r4r5r6rRr7rr9tokensr:tokens.underscorer;rr<r=training.initializer>r?r@rArBrCrDrErFrGrHrrIr~rJrKrT__file__parentDEFAULT_CONFIG_PATH load_configrDEFAULT_CONFIG_PRETRAIN_PATHrLrNrrrprhrrrrr rgrrrrrr`rmrnrs 0!")& AA($!WWA!MM )09   !&s # 8n++.BB!!"56 $H~447WWm$ QQ&(J<#:;2 !!DDTd   G G  G D4;Xg.;4=;6%S%0+>CD6 HS(3-/0 16 D$,- 6  6r:  ()rm