bCiV SSKrSSKrSSKrSSKrSSKrSSKJrJr SSKJ r SSK J r SSK J r JrJrJr SSKJrJr SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SSK J!r! SSK"J"r" \"5(aSSK#r$SSK#J%r% SSK#J&r' "SS\%5r(\ RR"5RTr+\ "\,5RZRZq.[\RZRZq/S\+S3Ra5r1"SS5r2S\\3\Rh4S\3S\34Sjr5S\2S\3S\3S \6\7\3\844S!jr9S"\3S#\6\34S$jr:S%\3S\3S\34S&jr;S%\3S\3S \6\7\3\844S'jrS(\3S\2S+\3S)\7\3\3\644S,jr?S\2S\3S \6\7\3\84S)\34S-jr@S\2S \6\7\3\844S.jrAS\2S\3S\3S \6\7\3\84S/\84 S0jrBS=S1\3S2\\3S3\\S4\\3S)\ 4 S5jjrCS6\3S)\84S7jrDS8rES9\4S:jrF"S;S<\!5rGg)>N)ArgumentParser Namespace)date)Path)AnyCallableOptionalUnion)CONFIG_MAPPING_NAMESMODEL_NAMES_MAPPING)FEATURE_EXTRACTOR_MAPPING_NAMES)IMAGE_PROCESSOR_MAPPING_NAMES)PROCESSOR_MAPPING_NAMES)TOKENIZER_MAPPING_NAMES)VIDEO_PROCESSOR_MAPPING_NAMES)is_libcst_available)BaseTransformersCLICommand)add_fast_image_processor) CSTVisitor)matchersc\rSrSrSrSrS\RSS4SjrS\R4Sjr S\R4S jr S r g) ClassFinder*z3 A visitor to find all classes in a python module. c./Ul/UlSUlgNF)classespublic_classes is_in_classselfs b/home/james-whalen/.local/lib/python3.13/site-packages/transformers/commands/add_new_model_like.py__init__ClassFinder.__init__/s!#DL(*D $D nodereturnNcpURRURR5 SUlg)zwRecord class names. We assume classes always only appear at top-level (i.e. no class definition in function or similar)TN)rappendnamevaluer r"r's r#visit_ClassDefClassFinder.visit_ClassDef4s$ LL    0#D r&cSUlgr)r r-s r#leave_ClassDefClassFinder.leave_ClassDef9s $D r&c[R"[R"[R"[R"5S9/S9/S9nUR (d[R "X5(aURSRSRRnUS:XaOURSRRnUVs/sHoURRPM snUl ggggs snf)z:Record all public classes inside the `__all__` assignment.)target)targets)bodyr__all__N) mSimpleStatementLineAssign AssignTargetNamer matchesr6r5r4r,elementsr)r"r'!simple_top_level_assign_structureassigned_variabler>elements r#visit_SimpleStatementLine%ClassFinder.visit_SimpleStatementLine<s010E0Ehhaffh(G'HIJ1 -## $(Z(Z$(IIaL$8$8$;$B$B$H$H!$ 1#yy|11::HNV*Wh7==+>+>h*WD'2)[#+XsD)rr r) __name__ __module__ __qualname____firstlineno____doc__r$cstClassDefr.r1r9rB__static_attributes__r&r#rr*sG  %  $s|| $ $  %s|| % X#2I2I Xr&rz # coding=utf-8 # Copyright aN the HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. c&\rSrSrSrS\4SjrSrg) ModelInfos^zA Retrieve the basic information about an existing model classes. lowercase_namecUR5RSS5RSS5UlUR[;aURRSS5 UR[;a[ US35e[ URUl[URUlURRSS5UlUR[;aD[URuUl Ul URS:XaSO URUl OSuUl Ul [R"URS5uUlUl[ R"URS5Ul[$R"URS5Ul[(R"URS5Ulg) N _-z is not a valid model nameConfigPreTrainedTokenizerFast)NN)lowerreplacerPr ValueErrorr paper_name config_classcamelcase_namertokenizer_classfast_tokenizer_classrgetimage_processor_classfast_image_processor_classrvideo_processor_classrfeature_extractor_classrprocessor_class)r"rPs r#r$ModelInfos.__init__cs,224<UVZViVi>j ;D $";115NNTXTmTm  %?I ;D $";FcFgFg   G C"D$C&C%F%FtGZGZ\`%a"'F'J'J4K^K^`d'e$6::4;N;NPTUr&) r]r\rbr_rdrarPr[rer^rcN)rDrErFrGrHstrr$rKrLr&r#rNrN^sVsVr&rN file_name new_content add_afterc[USSS9nUR5nSSS5 WRUS5upVXR-U-U-n[USSS9nURU5 SSS5 g!,(df  NR=f!,(df  g=f)a? A utility to add some content inside a given file. Args: file_name (`str` or `os.PathLike`): The name of the file in which we want to insert some content. new_content (`str`): The content to add. add_after (`str`): The new content is added just after the first instance matching it. rutf-8encodingNrw)openreadsplitwrite)rhrirjf old_contentbeforeafters r#add_content_to_filerys~ iw /1ffh  0 %%i3MF${2U:K iw /1   0 / 0 / 0 /sA)A:) A7: Bold_model_infosnew_lowercase_namenew_model_paper_namefilenames_to_addc SRSURSS5RS555nURnURnUSSVVs/sHupxURUS5U4PM nnn/n UHKup[ R "SU 5(aU S upX=(d U4U S 'M9U RX45 MM [[S - S- S - S US US3SS9 [[S - S- S - S US US3SS9 U HupxU(dMURSS5n[[S - S- U- 5n U R5n SSS5 [ R"SUS3W [ R5nUH9n[[S - S- U- URXQ5RXd5US9 M; M gs snnf!,(df  N=f)a Add a model to all the relevant mappings in the auto module. Args: old_model_infos (`ModelInfos`): The structure containing the class information of the old model. new_lowercase_name (`str`): The new lowercase model name. new_model_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. filenames_to_add (`list[tuple[str, bool]]`): A list of tuples of all potential filenames to add for a new model, along a boolean flag describing if we should add this file or not. For example, [(`modeling_xxx.px`, True), (`configuration_xxx.py`, True), (`tokenization_xxx.py`, False),...] rVc3@# UHoR5v M g7fNtitle.0xs r# -add_model_to_auto_mappings..`0_1WWYY0_rTrSrNautoz2(?:tokenization)|(?:image_processing)_auto_fast.pymodelszconfiguration_auto.pyz ("z", "z Config"), zOCONFIG_MAPPING_NAMES = OrderedDict[str, str]( [ # Add configs here rirjz"), zcMODEL_NAMES_MAPPING = OrderedDict[str, str]( [ # Add full (and cased) model names here _fast.py.pyz( {8,12}\(\s*"z",.*?\),\n)(?: {4,12}\(|\]))joinrYrsrPr]researchr*ryTRANSFORMERS_PATHrqrrfindallDOTALL)rzr{r|r}new_cased_nameold_lowercase_nameold_cased_namefilenameto_addcorrected_filenames_to_addfile previous_fileprevious_to_addrumatching_linesmatchs r#add_model_to_auto_mappingsrs(WW`0B0J0J3PS0T0Z0Z[^0_``N(77$33NWghihjWkWkCS8  ,f 5v>Wk"$(  99JD Q Q-G-K *M.;=VPV-W &r * & - -tn = )H$v-0GG !3 4D8H Tf H$v-0GG !3 4D9M8NeTz 7 6'' E:H'(2V;hFG1vvxH ZZ#$6#77TUW[]_]f]fN(#%069HD % .@ U ] ]&!$ (718HGs G G G! new_paper_namerc,Sn[R"SS[5RSS5U-n[R "SUSUS35n/nUH'nS US U3nS U;aUS - nUR U5 M) S RU5nX4-U-$)a Create a new doc file to fill for the new model. Args: new_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. public_classes (`list[str]`): A list of all the public classes that the model will have in the library. u ⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. --> z# ?rVz coding=utf-8 z