hqXSSKrSSKrSSKrSSKJr SSKJrJr SSKJ r SSK J r SSK J r SSKJr SSKJr SS KJr SS KJr SS KJr SS KJrJr SS KJrJrJr SSKJ r SSKJ!r! SSS/04SSSS/04/r"\RFS5r$\RFS5r%\RFS5r&\RFS5r'\RFS5r(\RFS5r)\RTRWSS/5\RTRYS5S 55r-\RTRYS!5S"5r.\RTRYS#5S$5r/\RTRYS%5S&5r0\RTRYS'5S(5r1S)r2\RTRYS*5S+5r3\RTRYS,5S-5r4S.r5S/r6S0r7S1r8S2r9\RTRuS3S49S55r;\RTRuS3S49S65rS9r?S:r@S;rAS<rBS=rCS>rDS?rE\RTRWS@SASB/5SC5rFSDrGSErHSFrISGrJSHrKSIrL\R"SJ5"SKSL55rNg)MN) assert_equal)registryutil)ENT_IOB)English)Italian)Language)Lookups)EntityRecognizer) BiluoPushDown)DEFAULT_NER_MODEL)DocSpan)Example iob_to_biluosplit_bilu_labelVocab) make_tempdirWho is Shaka Khan?entitiesPERSONzI like London and Berlin.)r LOC)rcg)N non_entitiesr#U/home/james-whalen/.local/lib/python3.13/site-packages/spacy/tests/parser/test_ner.pyneg_keyr&s r$c[5$Nrr#r$r%vocabr)!s 7Nr$c[U/SQS9$)N)CaseywenttoNewYork.words)r)r)s r%docr3&s uG HHr$czUSSnUSSnURURS4URURS4/$)NrrGPE) start_charend_char)r3caseynys r% entity_annotsr=+sH !HE QqB   5>>84  U+ r$c f[[UVVVs/sHupo3PM snnn55$s snnnfr()sortedset)r=selabels r% entity_typesrD5s( #-@-!u-@A BB@s,cV[R"US9n[URU5$)NrD)r get_actionsstrings)r)rDactionss r%tsysrJ:s#''\BG  00r$rCz U-JOB-NAMEic [5n0nURSUS9n[R"[ UR S/S9S/S/S/S/S/U/S.5nS UR RU/S 9S ;deg) Nnerconfigwordr1rtagdep)idsr2tagsheadsdepsrzJOB-NAME)examplesr5)r create_piper from_dictrr)movesrG)rCnlprNrLexamples r%test_issue1967r\@s *C F //%/ /C CIIfX&3XGSG   G .. .B1E EE Er$icn[5nURS5nURS5 UR5 [5nURS5 [ UR S5R 5S:XdeUR S5RnURS"X0R S5RR5 URUR55 SUR S5R;deUR S5R S:Xdeg)zGTest that spurious 'extra_labels' aren't created when initializing NER.rL CITIZENSHIPr resize_output extra_labels)r^N)radd_pipe add_label initializelenget_pipelabelsmodelattrsrYn_moves from_bytesto_bytescfg)rZrLnlp2rgs r%test_issue2179rnTs )C ,,u CMM- NN 9DMM% t}}U#** +q 00 0 MM% & &E KK  U(;(A(A(I(IJOOCLLN# u!5!9!9 99 9 ==  & &*: :: :r$iQ cSn[U5/SQ:XdeSn[U5/SQ:XdeSn[U5/SQ:XdeSn[U5/SQ:Xdeg ) z9Test that IOB tags are correctly converted to BILUO tags.) B-BRAWLER I-BRAWLERrq)rprqz L-BRAWLER)I-ORGrrB-ORG)rsL-ORGzU-ORG)B-PERSONzI-PERSONru)ruL-PERSONU-PERSON)B-MULTI-PERSONzI-MULTI-PERSONrx)rxzL-MULTI-PERSONzU-MULTI-PERSONN)r)tags1tags2tags3tags4s r%test_issue2385r}esh 4E  "I II I 'E  "= == = 0E  "F FF F BE  "X XX Xr$i c [5n/nUR[R"UR S5S/05/5 [ S5Vs/sHn[ U5PM nnURS5n[U5HnURU5 M UR5n[ S5H6n0n[R"U5 UHnURU/XgSS9 M M8 gs snf) z\Test issue that arises when too many labels are added to NER model. Used to cause segfault. z One sentencerirLg?)sgdlossesdropN)rextendrrXmake_docrangestrralistrbrcrandomshuffleupdate) rZ train_datairDrL entity_type optimizerrr[s r%test_issue2800rvs )CJ   3<<7*b9I JK%*$K0KqCFKL0 ,,u CL)  k"* I 2Yz"!G JJyiSJ I" 1sC3i c[5nURS5nURS5 UR5 /SQnURU:Xde[5nURS5nUR nUR S"XQRR5 URUR55 URU:Xdeg)zTest issue that occurred in spaCy nightly where NER labels were being mapped to classes incorrectly after loading the model, when the labels were added using ner.add_label(). rLANIMAL)OzB-ANIMALzI-ANIMALzL-ANIMALzU-ANIMALr_N) rrarbrc move_namesrgrhrYrirjrk)rZrLrrmner2rgs r%test_issue3209rs )C ,,u CMM(NNFJ >>Z '' ' 9D == D JJE KK  (9(9:OOCLLN# ??j (( (r$c[5nURS5nURS5 UR5 /SQnS1nURU:Xde[ UR 5U:Xdeg)zBTest that labels are inferred correctly when there's a - in label.rLz LARGE-ANIMAL)rzB-LARGE-ANIMALzI-LARGE-ANIMALzL-LARGE-ANIMALzU-LARGE-ANIMALN)rrarbrcrr@rf)rZrLrrfs r%test_labels_from_BILUOrsf )C ,,u CMM.!NNJ F >>Z '' ' szz?f $$ $r$ic$[5nURS5nURS5 UR5 SUR;deU"S5nUR S5(deUHnUR S:XaMe SSS./nURS 5nURU5 S UR;deSUR;deU"S5nUR S5(deUHnUR S:XaMe g ) zDTest that running an entity_ruler after ner gives consistent resultsrLPEOPLEhirrSOFTWAREspacyrCpattern entity_rulerN)rrarbrc pipe_nameshas_annotationent_iob add_patterns)rZrLdoc1tokenpatternsrulerdoc2s r%test_issue4267rs )C ,,u CMM(NN CNN "" " t9D   y ) )) )}}!!!%9:H LL (E x S^^ ++ + CNN "" " t9D   y ) )) )}}!!!r$icSnSn[5nUUS.nURSUS9nURS5 UR5 U"S5n[ UR 5S:XdeSUR ;de[ US S S S 9n[UR5U/-UlU/nURUS XS9 [ UR 5S:XdeS UR ;deg)z:This should not crash or exit with some strange error code-C6? beam_width beam_densitybeam_nerrM SOME_LABELzWhat do you think about Apple ?r5r7MY_ORGrC)rrrrN) rrarbrcrdrfrrents beam_parse)rrrZrNrLr3 apple_entdocss r%test_issue4313rsJL )C $F ,,z&, 1CMM,NN / 0C szz?a   3:: %% %S!Qh/ICHH~ +CH 5DNN4cjNT szz?a   szz !! !r$c[R"USU05nURUSS9nUVs/sHoPRU5PM nnU/SQ:Xdegs snf)NrF)_debug)rwrrB-GPEL-GPEr)rrXget_oracle_sequenceget_class_name)rJr3r=r[ act_classesactnamess r%test_get_oracle_movesrs_j-%@AG**75*AK1< =#  %E = A AA A >sAcX RS'[USS/S9nSS/n[R"USU05n[ UR SSS S 9[ UR SS S S 9/UR R U'URU5nUVs/sHopRU5PM nnU(deUSS :wdeUSS :wdeUSS:wdegs snf)zTest that we don't get stuck in a two word input when we have a negative span. This could happen if we don't have the right check on the B action. r&ABr1Nrrr5rrrrrurv rlrrrXryspansrr rJr)r&r3r=r[rrrs r%$test_negative_samples_two_word_inputrs"HHY eC: &C4LMj-%@AG WYY1C( WYY1H- GIIOOG**73K1< =#  %E = L5 8s?? 8z !! ! 8z !! ! >sCcX RS'[U/SQS9n/SQn[R"USU05n[ UR SSSS 9[ UR SS S S 9/UR R U'URU5nUVs/sHopRU5PM nnU(deUSS:wdeUSS :wdeg s snf)HTest that we exclude a 2-word entity correctly using a negative example.r&)rrCr1)NNNrrr5rrrrruNrrs r%&test_negative_samples_three_word_inputrs!HHY e? +C&Mj-%@AG WYY1C( WYY1H- GIIOOG**73K1< =#  %E = L5 8s?? 8z !! ! >sC cX RS'[US/S9nS/n[R"USU05n[ UR SSSS 9[ UR SSS S 9/UR R U'URU5nUVs/sHopRU5PM nnU(deUSS:wdeUSS :wdegs snf) rr&rr1Nrrr5rrrrwrrs r%test_negative_samples_U_entityrs!HHY eC5 !CFMj-%@AG WYY1C( WYY1H- GIIOOG**73K1< =#  %E = L5 8s?? 8z !! ! >sC c[R"US9n[URUSS9nURSS:Xdeg)NrFr")incorrect_spans_keyr&)r rGrHrl)r)rDrIrJs r%%test_negative_sample_key_is_in_configr*s;''\BG ^ TD 88I . 00 0r$zNo longer supported)reasonc8/SQn/SQn[XS9n[R"X1US.5n[UR5nSnUHnUcMUS:Xa#UR UR S5S5 M1[U5upUR UR S5U 5 UR UR S 5U 5 UR UR S 5U 5 UR UR S 5U 5 M URU5 g) N)r52Bomber)NNz L-PRODUCTr1)r2rMrILUrrrrrr rrrXr rH add_actionindexrr en_vocabr2 biluo_tagsr3r[rY move_typesrPactionrCs r%test_oracle_moves_missing_Br2s !E*J h $C*%MNG (** +E/J ;  CZ   Z--c2B 7,S1MF   Z--c2E :   Z--c2E :   Z--c2E :   Z--c2E : g&r$cr/SQn/SQn[XS9n[R"USU05n[UR5nSnUH_nUcMUS:Xa#UR UR S5S5 M1[U5upUR UR U5U 5 Ma URU5 g)N) production ofNorthroprzCorp.rz'sradar) rrrrsNrrrtrrr1rrrrrrs r%test_oracle_moves_whitespacerLs VEKJ h $Cj*%=>G (** +E/J ;  CZ   Z--c2B 7,S1MF   Z--f5u = g&r$cT[5nU"S5n0nURSUS9nUVs/sHoDRPM sn/SQ:XdeUVs/sHoDRPM sn/SQ:XdeURR SS5 UR S5 URRU/5SnURRUS 5 URRUS 5 URRUS 5 URRUS 5(de[5nU"S5n0nURSUS9nUR/US S/S S 9 UVs/sHoDRPM sn/SQ:XdeUVs/sHoDRPM sn/SQ:XdeURR SS5 URR SS5 UR S5 URRU/5Sn URRU S 5 URRU S 5 URRU S 5 URRU S 5(aeURRU S5(deURRU S5 URRU S 5(aeURRU S5(degs snfs snfs snfs snf)z5Test succesful blocking of tokens to be in an entity.I live in New YorkrLrMrrrrrr7rr8rrrr6 unmodifiedblockeddefault)rrrrrzU-N) rrWent_iob_ ent_type_rYrrb init_batchapply_transitionis_validset_ents) nlp1rrNner1rstate1rmrrstate2s r%test_accept_blocked_tokenras 9D $ %D F   E&  1D(, -uNN -1E EE E)- .OO .2F FF F JJ!R NN5 ZZ " "D6 *1 -FJJ,JJ,JJ, ::  vw / // / 9D $ %D F   E&  1D MM"tAayknUR[R"UR US5US55 M@ UR SSS 9nUR S 5 UR5 [S 5H2n0n[R"US S 9nUHnURXS9 M M4 g)z7Test that training an empty text does not throw errors.rrrrrr5rLTlastrrsizerN) rappendrrXrrarbrcrr minibatchr rrZtrain_examplestrLitnrbatchesbatchs r%test_train_emptyrs  ->,?@A j" J )CN g// QqT0BAaDIJ ,,u4, (CMM(NNQx..a8E JJuJ ,r$c SSS/04/n[5n/nUH>nUR[R"UR US5US55 M@ UR SSS9nUR S 5 UR5 [S 5HUn0n[R"US S 9nUH5n[R"[5 URXS 9 SSS5 M7 MW g!,(df  MM=f)zFTest that the deprecated negative entity format raises a custom error.rr)rrz!PERSONrr5rLTrrrrrr N)rr rrXrrarbrcrrr pytestraises ValueErrorrr s r%test_train_negative_deprecatedrs  -?,@ABJ )CN g// QqT0BAaDIJ ,,u4, (CMM(NNQx..a8Ez* 5 0+*+*s C// C> c[5nURS5 UR5 U"S5nUVs/sHo"RPM sn/SQ:XdeUVs/sHo"RPM sn/SQ:Xde0nUR SUS9nUR RSS5 URS5 UR RU/5S nUR RUS 5(deUR RUS 5(deUR RUS 5 UR RUS 5(deUR RUS 5(degs snfs snf)NrLr)rrrrrrrMr7rr8rrzU-GPEzI-GPEr) rrarcrrrWrYrrbrrr)rZr3rrNrstates r%test_overwrite_tokenrsL )CLLNN " #C(+ ,uNN ,0I II I), -OO -1E EE E F ??5? 0DJJ!R NN5 JJ ! !3% ( +E ::  ug . .. . ::  ug . .. .JJw/ ::  ug . .. . ::  ug . .. . - -s FFc[5nURS5nURS5 UR5 U"S5n/SQnUVs/sHoDRPM snU:Xdegs snf)NrLMY_LABELz3John is watching the news about Croatia's elections) rrrrrrrrr)rrarbrcr)rZrLr3resultrs r%test_empty_nerrs_ )C ,,u CMM*NN C DC :F(+ ,uNN , 66 6 ,s A,c[5nSSS./nURS5nURS5nURS5 UR5 UR U5 U"S5n/SQn/S QnUVs/sHowR PM snU:XdeUVs/sHowR PM snU:Xdeg s snfs snf) zLTest that an NER works after an entity_ruler: the second can add annotationsTHINGThisrrrLr*This is Antti Korhonen speaking in Finlandrrrrrrrr rrrrrrNrrarbrcrrr)rZrr untrained_nerr3 expected_iobsexpected_typesrs r%test_ruler_before_nerr)s )C"f56H LL (ELL'MJ'NN x : ;C7M6N(+ ,uNN , == =), -OO - ?? ? - -s 9B:B?czSS0nS[0n[R"USS9Sn[X40UD6 [X5 g)Nupdate_with_oracle_cut_sizedrgT)validate)r rresolver )rrNrlrgs r%test_ner_constructorr/sG%sF % &C   S4 0 9EX//X%r$c[5nURSSS9nURS5 UR5 SSS./nURS5nUR U5 U"S 5n/S Qn/S QnUVs/sHowR PM snU:XdeUVs/sHowR PM snU:Xdeg s snfs snf) zTTest that an entity_ruler works after an NER: the second can overwrite O annotationsrLuner)namerr r!rrr"r#r$Nr%)rZr&rrr3r'r(rs r%test_ner_before_rulerr3s )CLLVL4MJ'NN"f56H LL (E x : ;C7M6N(+ ,uNN , == =), -OO - ?? ? - -s 8B9B>c^[5nURSSSS.S9 URS5nURS5 UR5 U"S5n/S Qn/S QnUVs/sHoURPM snU:XdeUVs/sHoUR PM snU:Xdeg s snfs snf) zITest functionality for blocking tokens so they can't be in a named entityblockerrr7)startendrMrLrz,This is Antti L Korhonen speaking in Finland)rrrrrrrr)rrrrrrrrN)rrarbrcrr)rZr&r3r'r(rs r%test_block_nerr8s )CLLQq#9L:LL'MJ'NN < =CI like London.r5rLondonr has_upperRANDOM_NEW_LABEL)zJust a sentence.z$Then one more sentence about London.zHere is another one.r>zI like London and London.r6i)rCkb_id)rra TRAIN_DATAr rrXrgetrbrcrrrrdtextlabel_rto_diskrload_model_from_pathrergrhpipeto_arrayrrrrB)r9rZrLr rE annotationsentrrr test_textr3rtmp_dirrmrents2rdoc3ents3texts batch_deps_1 batch_deps_2 no_batch_depss r%test_overfitting_IOrVs )C ,,ug Y/G%H, ICN'g// T0BKPQ??:.C MM#a& !/( I 2Y > @ %=7 "" "!I i.C 88D t9>> 7<<8 ## # 7>>U "" " 7 G((1I 5zQQx}}(((Qx%'''}}U#zz , 999 )*I 5zQQx}}(((Qx%'''! & E 8;xxGLL'+LG7:xxGLL'+LGOT8UutTu8UV8U\\7),8UMV|,}-,I ,,y !CS!Qe489CH 88D t9>> 7<<8 ## # 7>>U "" " 7==D  ,,u c "C 88D t9>> 7<<8 ## # 7>>U "" " 7==D  7<<8 ## # 7>>U "" " 7==A  c 2HG8UVs%+DP/QQ+Q Q/ P>cSnSn[5nUUS.nURSUS9n/n[HfupgUR[R "UR U5U55 URS5HnURUS5 M Mh UR5n 0n URXYU S9 S n UR U 5n U /n URU 5nURU5S n[[U 55H;nURH(nUUUS -U4nS nS U- Us=::a S U-::aM%e e M= g) NrrrrrMrrr<r>rr5r=)rrarCr rrXrrDrbrcrpredict scored_entsrrdrf)rrrZrNrLr rErKrLrrrMr3rbeams entity_scoresjrCscoreepss r%test_beam_ner_scoresr_lsJJL )C $F ,,z&, 1CN'g// T0BKPQ??:.C MM#a& !/( IFJJ~VJ<!I ,,y !C 5D KK EOOE*1-M 3s8_ZZE!1a!eU"34ECs7e.q3w. .. .. r$c[5nSnSnUUUS.nURSUS9n/n[HfupxUR[R "UR U5U55 URS5Hn URU S5 M Mh UR5n [S5Hn 0n URXjU S 9 M W SS:deS n UR U 5/nURU5nURU5S nUS S :XdeUSS:Xde[U"U 5R5S:Xde[!5nUR#U5 [$R&"U5nUR U 5/nUR)S5nURU5nURU5S nUS S :XdeUSS:XdeSSS5 UR U 5n[ UU5n[+USSS5/UR,R.U'U/n[S5Hn 0n URUXS 9 M [U"U 5R5S :Xdeg!,(df  N=f)NrrrrrrrMrrr;r<z I like Londonr)rr6rg?)rr6rrr5r6rr)rrarCr rrXrrDrbrcrrrXrYrdrrrGrrHrer referencer)r&rZrrrNrLr rErKrLrrrrMrrZr[rNrmdocs2rbeams2entity_scores2neg_docneg_exneg_train_exampless r%test_beam_overfitting_IOris{ )CJL $&F ,,z&, 1CN'g// T0BKPQ??:.C MM#a& !/( I2Y > @ *  && & I LL # $D KK EOOE*1-M  '3 .. . ) *c 11 1 s9~"" #q (( ( 7 G((1y)*}}Z(e$))&1!4m,333./3666 ll9%G Wg &F'+GQ5'A&BF7#  2Y %9 D s9~"" #q (( (+ s BI++ I9c[5nSnSnUUUS.nURSUS9nSnURU5nURS5 URS5 [R "US S /05n[ URS S S5[ URS S S5[ URSS S5/URRU'UR5n [S 5Hn 0n URU/XS9 M g)zCheck that the NER update works with a negative annotation that is a different label of the correct one, or partly overlapping, etcrrrarrMrrORGrrrrr6r5r<N) rrarrbrrXrrbrrcrr r&rZrrrNrL train_textrfr[rrrs r%test_neg_annotationrns )CJL $&F ,,z&, 1C%Jll:&GMM(MM%*7H6I)JKG W  1e, W  1h/ W  1h/(GG$  I 1X G9) ;r$c [5nSnSnUUUS.nURSUS9nSnURU5nURS5 URS5 [R "US S /05n[ URS S S5/URRU'[URR5S :XdeURRSRS:XdeURRSRS:Xde[URRU5S :XdeURRUSRS:XdeURRUSRS:XdeUR5n [S 5H9n 0n [R "["5 UR%U/XS9 SSS5 M; g!,(df  MM=f)NrrrarrMrrrrrrrr5rz Shaka Khanr<)rrarrbrrXrrbrrdrrErFrcrrrrrrls r%test_neg_annotation_conflictrps )CJL $&F ,,z&, 1C%Jll:&GMM(MM%*7H6I)JKG(,W->->1h(O'PGG$ w  %% &! ++ +    ! !! $ ) )\ 99 9    ! !! $ + +x 77 7 w  &&w/ 0A 55 5    " "7 +A . 3 3| CC C    " "7 +A . 5 5 AA A I 1X ]]: & JJyiJ ?' & & &s G44 H c[5nSnSnUUUS.nURSUS9 /SQn/SQn[URUS9n[R "US U05n[ URS S S 5n U /URRU'UR5n [S 5Hn 0n URU/XS9 M SW ;deg)z/Regression test for previously flakey behaviourrrrarrM)5FEDERALNATIONALMORTGAGE ASSOCIATION(FannieMaez):Postedyieldson30yearmortgage commitmentsfordeliverywithinr|daysrvpricedatpar)z9.75%,standard conventionalfixed-rate mortgages;z8.70rrz6/2rcappedonerr} adjustablerrr0Source:TelerateSystemszInc.)5rsrrrrrtrrsrtrrrrB-DATEL-DATErrrrrrrrrrrr B-PERCENT L-PERCENTrrrrrrrrrrrz U-CARDINALrrrzI-DATErrrrrrrrrrr1rLr;5rkr7r<N) rrarr)rrXrrbrrcrr) r&rZrrrNtokensiobr3r[neg_spanrrrs r%test_beam_valid_parsers )CJL $&F LLFL+VF xC ciiv &CeS\2GG%%r2u5H(0zGG$ I 1X G9) ;   r$c"[5nUR[R;de[ 5UR l[UR R 5(aeURS5 UR[R5 UR5 SUR;deSSS5 UR5 UR R RS5 SUR R R!S5S'UR[R5 UR5 SUR;deSSS5 g!,(df  N=f!,(df  g=f)NrLW033 lexeme_normra)rlangrLEXEME_NORM_LANGSr r)lookupsrdraat_levelloggingDEBUGrcrEclear add_table get_table)caplogrZs r%test_ner_warns_no_lookupsr"s )C 88t-- -- - CII399$$%%% %LL  ' $$$ ( LLNII .69CII .s3  ' V[[((( ( ' ( ' ( 's#E/#F/ E= Fr5c$\rSrSrSSjrSrSrg)BlockerComponent1i3c(X lX0lX@lgr()r6r7r2)selfrZr6r7r2s r%__init__BlockerComponent1.__init__5s  r$cVUR/XRUR/SS9 U$)Nrr)rr6r7)rr3s r%__call__BlockerComponent1.__call__:s) R#jj488"rs  & !!#!+>0"BBJ):(;<= :/P"QR IICC11 <.14F2F$4 ; ; 4 Y Y 4JJ*4))&%$4""24""4B","&"&1./'0'2./'0'(.-b-*1*/*7@(&@* @tUm4J5JZ /F7)t<:@> :)" )r$