cCiZSSKrSSKJrJr SSKJr \R "\5r"SS\5r "SS\5r "S S \5r /S Qr g) N)PretrainedConfiglayer_type_validation)loggingc^\rSrSrSrSSSSSSSS.rSrSrSS \S \ S \S \S \S\S\S\S\S\ S\ 4U4Sjjjr Sr U=r $)Llama4VisionConfiga This is the configuration class to store the configuration of a [`Llama4VisionModel`]. It is used to instantiate a Llama4 vision model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Llama4 109B. e.g. [meta-llama/Llama-4-Scout-17B-16E](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. num_hidden_layers (`int`, *optional*, defaults to 34): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. num_channels (`int`, *optional*, defaults to 3): Number of channels in the input image. intermediate_size (`int`, *optional*, defaults to 5632): Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder. vision_output_dim (`int`, *optional*, defaults to 7680): Dimensionality of the vision model output. Includes output of transformer encoder with intermediate layers and global transformer encoder. image_size (`int`, *optional*, defaults to 448): The size (resolution) of each image *tile*. patch_size (`int`, *optional*, defaults to 14): The size (resolution) of each patch. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the layer normalization layers. vision_feature_select_strategy (`int`, *optional*, defaults to `"default"`): TODO initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. pixel_shuffle_ratio (`int`, *optional*, defaults to 0.5): TODO projector_input_dim (`int`, *optional*, defaults to 4096): TODO projector_output_dim (`int`, *optional*, defaults to 4096): TODO multi_modal_projector_bias (`int`, *optional*, defaults to `False`): TODO projector_dropout (`int`, *optional*, defaults to 0.0): TODO attention_dropout (`int`, *optional*, defaults to 0.0): TODO rope_theta (`int`, *optional*, defaults to 10000): TODO colwiserowwise colwise_rep)zmodel.layers.*.self_attn.q_projzmodel.layers.*.self_attn.k_projzmodel.layers.*.self_attn.v_projzmodel.layers.*.self_attn.o_projzvision_adapter.mlp.fc1zvision_adapter.mlp.fc2zpatch_embedding.linearllama4_vision_model vision_config hidden_size hidden_actnum_hidden_layersnum_attention_heads num_channelsintermediate_sizevision_output_dim image_size patch_sizenorm_epsinitializer_rangec |>XlX lX0lXPlX`lXlXplXlXlX@l Xl Xl Xl Xl UUlUUlUUlXlUUlUR'SS5Ul[*S5nUR,S5n[.TU]`"S0UD6 g)Nvision_feature_layercP[R"S[5 UR$)NzRThe `vision_feature_layer` attribute is deprecated and will be removed in v4.58.0.)warningswarn FutureWarning_vision_feature_layer)selfs i/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/llama4/configuration_llama4.pyr9Llama4VisionConfig.__init__..vision_feature_layers" MMd -- -cXlg)N)r!)r"values r#rr$s). &r%)rrrrrrrrrrrpixel_shuffle_ratioprojector_input_dimprojector_output_dimmulti_modal_projector_biasprojector_dropoutattention_dropoutvision_feature_select_strategy rope_thetagetr!propertysettersuper__init__)r"rrrrrrrrrrr/rr)r*r+r,r-r.r0kwargsr __class__s r#r5Llama4VisionConfig.__init__Ts.'$!2(!2$!2$ #6 !2#6 #6 $8!*D'!2!2.L+$%+ZZ0F%K"  .  .  $ $ / % / "6"r%)r!r.rrrrrr,rrrrrr)r-r*r+r0r/r)igelu"riiih㈵>default{Gz?g?r@FrAi')__name__ __module__ __qualname____firstlineno____doc__base_model_tp_plan model_typebase_config_keyintstrfloatr5__static_attributes__ __classcell__r7s@r#rrs+\,5+4+4+4"+"+"/'J%O !##%!%!%'0#' !#()9#9#9# 9# ! 9#  9#9#9#9#9#9#!9#9#r%rc^\rSrSrSrSrS/rSSSSSSSS SS SSSS S .rSSSSS S S SSSSS. rSU4Sjjr Sr U=r $)Llama4TextConfiga This is the configuration class to store the configuration of a [`Llama4TextModel`]. It is used to instantiate a Llama4 text model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Llama4 109B. e.g. [meta-llama/Llama-4-Scout-17B-16E](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 202048): Vocabulary size of the Llama4 text model. Defines the maximum number of different tokens that can be represented by the `inputs_ids` passed when calling [`Llama4TextModel`]. hidden_size (`int`, *optional*, defaults to 5120): Dimensionality of the embeddings and hidden states. intermediate_size (`int`, *optional*, defaults to 8192): Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder. intermediate_size_mlp (`int`, *optional*, defaults to 16384): TODO num_hidden_layers (`int`, *optional*, defaults to 48): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 40): Number of attention heads for each attention layer in the Transformer encoder. num_key_value_heads (`int`, *optional*, defaults to 8): This is the number of key_value heads that should be used to implement Grouped Query Attention. If not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 128): TODO hidden_act (`str` or `Callable`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the encoder and pooler. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions. pad_token_id (`int`, *optional*, defaults to 128004): The id of the padding token. bos_token_id (`int`, *optional*, defaults to 1): The id of the beginning of sentence token. eos_token_id (`int`, *optional*, defaults to 2): The id of the end of sentence token. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings rope_theta (`float`, *optional*, defaults to `500000.0`): The base period of the RoPE embeddings. attention_dropout (`int`, *optional*, defaults to 0.0): TODO num_experts_per_tok (`int`, *optional*, defaults to 1): TODO num_local_experts (`int`, *optional*, defaults to 16): TODO moe_layers (`int`, *optional*): TODO interleave_moe_layer_step (`int`, *optional*, defaults to 1): TODO use_qk_norm (`int`, *optional*, defaults to `True`): TODO output_router_logits (`int`, *optional*, defaults to `False`): TODO router_aux_loss_coef (`int`, *optional*, defaults to 0.001): TODO router_jitter_noise (`int`, *optional*, defaults to 0.0): TODO rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. `beta_fast` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. `beta_slow` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. `short_factor` (`list[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `long_factor` (`list[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `low_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE `high_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE no_rope_layers (`list[int]`, *optional*): List with at least the same length as the number of layers in the model. A `1` at an index position indicates that the corresponding layer will use RoPE, while a `0` indicates that it's a NoPE layer. no_rope_layer_interval (`int`, *optional*, defaults to 4): If `no_rope_layers` is `None`, it will be created using a NoPE layer every `no_rope_layer_interval` layers. attention_chunk_size (`int`, *optional*, defaults to 8192): layer_types (`list`, *optional*): Attention pattern for each layer. attn_temperature_tuning (`bool`, *optional*, defaults to `True`): Whether to dynamically scale the attention temperature for each query token based on sequence length. Recommended for long sequences (e.g., >32k tokens) to maintain stable output results. floor_scale (`int`, *optional*, defaults to 8192): TODO attn_scale (`int`, *optional*, defaults to 0.1): TODO Example: llama4_textpast_key_valuesr r local_colwise local_rowwiselocal_packed_rowwiselocalgather)layers.*.self_attn.q_projlayers.*.self_attn.k_projlayers.*.self_attn.v_projlayers.*.self_attn.o_projz-layers.*.feed_forward.shared_expert.gate_projz+layers.*.feed_forward.shared_expert.up_projz-layers.*.feed_forward.shared_expert.down_proj*layers.*.feed_forward.experts.gate_up_proj'layers.*.feed_forward.experts.down_projlayers.*.feed_forward.expertslayers.*.feed_forward.gate_projlayers.*.feed_forward.up_projlayers.*.feed_forward.down_projzlayers.*.feed_forward grouped_gemm ep_router) rZr[r\r]r^r_r`rarbrczlayers.*.feed_forward.routerc$ >[T(U]"SUUUUS.U$D6 U!UlU#UlU"UlXlXlX lX0lX@l XPl X`l UUl SUl UcUnXplXlXlXlXlUUlUUlUbUOURUR-UlUUlUUlUUlUUlUUlUUlU/:XaSn[9UR5V%s/sHn%[;U%S-U-S:g5PM n&n%U(aUOU&UlUUlUbUO[A[9US- UU55Ul!UUl"U Ul#U c+UR<V's/sHn'U'(aSOSPM sn'Ul#[IURFUR5 gs sn%fs sn'f)N) pad_token_id bos_token_id eos_token_idtie_word_embeddingsFrchunked_attentionfull_attentionr()%r4r5attn_temperature_tuning attn_scale floor_scale vocab_sizemax_position_embeddingsrrintermediate_size_mlprr rope_scalingattention_biasnum_key_value_headsrr rms_norm_eps use_cacher0r.head_dim use_qk_normnum_experts_per_toknum_local_expertsoutput_router_logitsrouter_aux_loss_coefrouter_jitter_noiserangerJno_rope_layersinterleave_moe_layer_steplist moe_layersattention_chunk_size layer_typesr))r"rqrrrsrrrvryrrrrrwrxrgrhrirjr0r.r{r|rrrzr}r~rrtrno_rope_layer_intervalrrrnrpror6 layer_idxdefault_no_rope_layersno_roper7s) r#r5Llama4TextConfig.__init__$sN  %%% 3    (?$$&$'>$&!2%:"!2#6 (#  &"5 #6 $!2("$!2$,$8d>N>NRVRjRj>j &#6 !2$8!$8!#6  R !NQVVZVlVlPm" Pm9CQ"88A= >Pm " 1?nDZ)B&% e59;LNghi  %9!&  TXTgTg Tgw#4DDTg D  d..0F0FG'"  s F8:F=)rurr.rornrpryrrrrrrsrrrrrrr{rrvr|r}rwrtr0r~rrxrzrq)#i@i i@0(siluir?r=TNrkFi rArkr;NrkTFgMbP?rANNrNTrg?) rBrCrDrErFrHkeys_to_ignore_at_inferencerGbase_model_ep_planr5rMrNrOs@r#rQrQsqfJ#4"5%.%.%.%.9H7F9H6L3B)0+:)8+:!)"&/%.%.%.6D3A)1+:)8+:(3  # )!"#"" ! $IfHfHr%rQc^^\rSrSrSrSrSSSS.r\\S.r S S 0r S U4S jjr S r U=r $) Llama4ConfigiaG This is the configuration class to store the configuration of a [`Llama4Model`]. It is used to instantiate an Llama4 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Llama4 109B. e.g. [meta-llama/Llama-4-Scout-17B-16E](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vision_config (`Llama4VisionConfig`, *optional*): The Llama4 Vision config. text_config (`Llama4TextConfig`, *optional*): The Llama4 Text config. boi_token_index (`int`, *optional*, defaults to 200080): The begin-of-image token index to wrap the image prompt. eoi_token_index (`int`, *optional*, defaults to 200081): The end-of-image token index to wrap the image prompt. image_token_index (`int`, *optional*, defaults to 200092): The image token index to encode the image prompt. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether the model's input and output word embeddings should be tied. ```python >>> from transformers import Llama4Model, Llama4Config >>> # Initializing a Llama4 7B style configuration >>> configuration = Llama4Config() >>> # Initializing a model from the Llama4 7B style configuration >>> model = Llama4Model(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```llama4image_token_indexboi_token_indexeoi_token_index)image_token_id boi_token_id eoi_token_id) text_configrzmulti_modal_projector.linear_1r c >Uc%[5Ul[RS5 OA[ U[ 5(a[S0UD6UlO[ U[5(aXlX0lX@lXPlUc%[5Ul [RS5 OA[ U[ 5(a[S0UD6Ul O[ U[5(aX l [TU]0"SSU0UD6 g)Nz9vision_config is None, using default llama4 vision configz5text_config is None, using default llama4 text configrjr() rrloggerinfo isinstancedictrrrrQrr4r5) r"rrrrrrjr6r7s r#r5Llama4Config.__init__s  !3!5D  KKS T  t , ,!3!Dm!DD   '9 : :!. ..!2  /1D  KKO P  T * */>+>D   %5 6 6*  K-@KFKr%)rrrrr)NNi i i F)rBrCrDrErFrH attribute_maprQr sub_configsrGr5rMrNrOs@r#rrsZ$ LJ-))M #3EWXK(-  !LLr%r)rrQr) rconfiguration_utilsrrutilsr get_loggerrBrrrQr__all__r(r%r#rs^"J   H %s#)s#lzH'zHzOL#OLd Er%