cCirSSKJrJrJrJr SSKrSSKJs Jr SSKJr SSK J r SSK J r Jr SSKJr SSKJr SS KJr SS KJrJrJr SS KJrJr SS KJrJr SS KJ r SSK!J"r"J#r#J$r$ SSK%J&r& SSK'J(r( \#"5(a SSK)J*r* SSK+J,r, \$RZ"\.5r/"SS\SS9r0"SS\Rb5r2"SS\Rb5r3"SS\Rb5r4"SS\Rb5r5"S S!\Rb5r6S"r7S@S#jr8S$\RrS%\:S&\Rr4S'jr;SAS(\RbS)\RrS*\RrS+\RrS,\\RrS-\"S2S3\5r?\""S4S5\55r@"S6S7\Rb5rA\""S8S9\@55rBSBS:\\Rr\C\RrS4S;\\:S,\\RrS&\\Rr\:44S<jjrD"S=S>\@\5rE/S?QrFg)C)CallableOptional TypedDictUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringis_torch_flex_attn_availablelogging)deprecate_kwarg)GraniteMoeSharedConfig) BlockMask)make_flex_block_causal_maskc\rSrSr%Sr\R \S'\R \S'\\S'\\S'\R\S'Sr g ) GraniteFlashAttentionKwargs3aR Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kseq_idxN) __name__ __module__ __qualname____firstlineno____doc__torch LongTensor__annotations__int IntTensor__static_attributes__r'x/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/granitemoeshared/modeling_granitemoeshared.pyr r 3s7"###### __r3r F)totalcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) GraniteMoeSharedMLPLzj MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configcX>[TU]5 URUlURUl[ UR Ul[R"URURS-SS9Ul [R"URURSS9Ul g)NFbias) super__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr9 __class__s r4r?GraniteMoeSharedMLP.__init__Us  ,,!:: !2!23IIdoot7G7G!7KRWXYYt'7'7uUr3 hidden_statesreturncURU5nURSSS9nURUS5US-nURU5nU$)Nr;dimrr)rFchunkrDrG)rIrLchunked_hidden_statess r4forwardGraniteMoeSharedMLP.forward^s^))-8 - 3 3A2 3 >(=a(@ADYZ[D\\ **=9 r3)rDr@rFrArG) r(r)r*r+r,rr?r-TensorrTr2 __classcell__rJs@r4r7r7Ls7V5VU\\ellr3r7c8^\rSrSrSU4SjjrSrSrSrU=r$)GraniteMoeSharedRMSNormfc>[TU]5 [R"[R "U55UlX lg)z6 GraniteMoeSharedRMSNorm is equivalent to T5LayerNorm N)r>r?r Parameterr-onesweightvariance_epsilon)rIr@epsrJs r4r? GraniteMoeSharedRMSNorm.__init__gs/ ll5::k#:; #r3cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nr;rOT)keepdim) dtypetor-float32powmeanrsqrtr`r_)rIrL input_dtypevariances r4rTGraniteMoeSharedRMSNorm.forwardosw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r3c^[URR5SUR3$)Nz, eps=)tupler_shaper`)rIs r4 extra_repr"GraniteMoeSharedRMSNorm.extra_reprvs* ))*+6$2G2G1HIIr3)r`r_)gư>) r(r)r*r+r?rTrqr2rWrXs@r4rZrZfs$;JJr3rZcB^\rSrSrS\S\S\SS4U4SjjrSrS rU=r$) GraniteMoeSharedParallelExpertsz num_expertsrA output_sizerMNc>[TU]5 [R"[R "XU55UlXlX lX0l g)a] Initialize the GraniteMoeSharedParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) r>r?rr]r-emptyr_rvrArw)rIrvrArwrJs r4r?(GraniteMoeSharedParallelExperts.__init__{s<" ll5;;{#TU &$&r3cURUSS9n/n[UR5H8nUR[R "X5UR U55 M: [R"USS9nU$)z Forward pass of the GraniteMoeSharedParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rrP) splitrangervappendFlinearr_r-cat)rIinputs expert_size input_list output_listiresultss r4rT'GraniteMoeSharedParallelExperts.forwardsh\\+1\5  t''(A   qxx t{{1~F G)))KQ/r3)rArvrwr_ r(r)r*r+r0r?rTr2rWrXs@r4rtrtzs.'C'S's't'.r3rtc>^\rSrSrS\S\S\4U4SjjrSrSrU=r$)GraniteMoeSharedTopKGatingrArvtop_kcz>[TU]5 X lXlX0l[ R "XSS9Ulg)z Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. Fr<N)r>r?rvrArrrElayer)rIrArvrrJs r4r?#GraniteMoeSharedTopKGating.__init__s2 &$ YYzUC r3czURU5R5nURURSS9up4[R "USS9R U5n[R"URS5UR/URURS9nURSUS5nUR5RS5nUR5nUR!5n U R#S5upU R%URSS9n UR!5nX[n XXU4$)NrrPrredevicetrunc) rounding_mode)rfloattopkrr-softmaxtype_aszerossizervrerscatterlongsumtolistflattensortdiv)rIrLlogits top_k_logits top_k_indices top_k_gatesrgatesr top_k_experts_index_sorted_experts batch_index batch_gatess r4rT"GraniteMoeSharedTopKGating.forwards"M*002&,kk$**!k&D# mmLa8@@O    a $"2"2 3;;L;LU`UgUg  a2jjl&&q) "((* &--/ "/"4"4Q"7*..tzz.Q "))+ !7 #+FRRr3)rArrvrrrXs@r4rrs-D3DSDD&SSr3rc:^\rSrSrSrS\4U4SjjrSrSrU=r $)GraniteMoeSharedMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. r9c>[TU]5 URUlURUl[ UR Ul[URURURS-5Ul [URURUR5Ul [URURURS9Ulg)Nr;)rArvr)r>r?r@rAintermediate_sizer rCrDrtnum_local_expertsrFrGrnum_experts_per_tokrouterrHs r4r?GraniteMoeSharedMoE.__init__s  ,,!33 !2!23;  $ $doot7G7G!7K =  $ $d&6&6 100,,  r3cUR5up#nURSU5nURU5upVpxn Xn URX5n U R SSS9n UR U S5U S-n UR X5n XSS2S4-n [R"X#-UR4U RU RS9nURSXm5nURX#UR5nX4$)z Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. rOr;rPrrNr)rreshaperrFrRrDrGr-rrArer index_addview)rI layer_inputbszlengthemb_sizerrrr router_logits expert_inputsrLrSexpert_outputsr layer_outputs r4rTGraniteMoeSharedMoE.forwards!, 0 0 2X!))"h7 BF++kBZ? -#0 ))-E - 3 3A2 3 >(=a(@ADYZ[D\\ ++MG'ag*>> S\4??;>CWCW`n`u`uvq+F #((dooF **r3)rDr@rFrArGr) r(r)r*r+r,rr?rTr2rWrXs@r4rrs 5 &++r3rcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..NrOr;rP)rpr-r)xx1x2s r4 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r3cURU5nURU5nX-[U5U--nX-[U5U--nXg4$)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkcossin position_ids unsqueeze_dimq_embedk_embeds r4apply_rotary_pos_embrsS( -- &C -- &Cw;q>C/0Gw;q>C/0G  r3rLn_reprMcURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rpexpandr)rLrbatchnum_key_value_headsslenhead_dims r4 repeat_kvr5s_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr3modulequerykeyvalueattention_maskscalingdropoutc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)Nr;rrO)rQre)ptrainingr)rnum_key_value_groupsr-matmul transposerpr functionalrrgrfrerr contiguous) rrrrrrrkwargs key_states value_states attn_weights causal_mask attn_outputs r4eager_attention_forwardrAs3 ; ;[TU]5 XlX lUc-[R SUR RS35 URUlURUl URUl URUR-Ul URUl URUR-UlSUlUR UlURUR-UR:wa&[%SURSURS35e[&R("URURUR-UR*S9Ul[&R("URURUR-UR*S9Ul[&R("URURUR-UR*S9Ul[&R("URURUR*S9Ulg)NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).r<)r>r?r9rlogger warning_oncerJr(attention_dropoutr@num_attention_heads num_headsrrr is_causalattention_multiplierr ValueErrorrrEattention_biasq_projk_projv_projo_projrIr9rrJs r4r?"GraniteMoeSharedAttention.__init__as  "      !8!8 9:,,  "(!9!9!--33((DNN: #)#=#= $(NNd6N6N$N!22 MMDNN *t/?/? ?QRVRbRbQc$T^^$4B8  ii 0 0$..4==2PW]WlWlm ii 0 0$2J2JT]]2Zagavavw ii 0 0$2J2JT]]2Zagavavw ii 0 0$2B2BI^I^_ r3past_key_valuepast_key_values4.58new_nameversionrLrr use_cachecache_positionposition_embeddingsrMc UR5upn URU5n URU5n URU5nU R XUR UR 5RSS5n U R XURUR 5RSS5n UR XURUR 5RSS5nUbUOSunnUb[XUU5upUb$UXS.nURXURU5up[nURRS:wa[URRnU"UU U UU4UR (dSO UR"UR$S.UD6unnUR XS5nUR'U5nUU4$) Nrr;)NN)rrr eager)rrrO)rrrrrrrrrrupdaterrr9_attn_implementationrrrrr)rIrLrrrr r rrrq_lenr query_statesrrrr cache_kwargsattention_interfacerrs r4rT!GraniteMoeSharedAttention.forwards&**, A{{=1 [[/ {{=1 #((T^^T]]S]]^_abc __S1I1I4==Yccdeghi #((T5M5Mt}}]gghiklm *=*I&|S  *';LVY[^'_ $L  &#&sUL'6'='=jX\XfXfht'u $J(? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$2H2HLL %  % ! \"&&s26 kk+. L((r3)rr9rr@rrrrrrrrrrN)NNNFNN)r(r)r*r+r,rrr0r?rr-rVr.r boolrorTr2rWrXs@r4rr^sG`5`(3-``@%0A6R2637+/59KO0)||0)!.0)u//0 0) "% 0)  0)!!1!120)&eELL%,,,F&GH0) u||Xell3XeELL>Q5RR S0)S0)r3rc^\rSrSrS\S\4U4Sjjr\"SSSS9SS \RS \ \RS \ \RS\ \ S \ \ S \ \ S\ \RS\ \ S\ \\R\R4S\\S\\R"\ \\R"\R"444Sjj5rSrU=r$)GraniteMoeSharedDecoderLayerir9rc>[TU]5 URUl[XS9UlUR S:a[ U5Ul[URURS9Ul [URURS9Ul URUl URS:XaSUlg[U5Ulg)N)r9rrra)r>r?r@r self_attnrrblock_sparse_moerZ rms_norm_epsinput_layernormpost_attention_layernormresidual_multiplierrBr7 shared_mlprs r4r?%GraniteMoeSharedDecoderLayer.__init__s !--2&V  # #a '$7$?D !6v7I7IvObObc(?@R@RX^XkXk(l%#)#=#= "("A"AQ"F$L_`fLgr3rrrr rLrroutput_attentionsr r output_router_logitsrrrMc tUn URU5nUR"SUUUUUUUU S.U D6upXUR--nUn URU5nUR U5upUR cU nOXR U5-nA XUR--nU4nU(aX4- nU(aX4- nU$)aq Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. )rLrrrr'r r rr')r"rr$r#r r%)rIrLrrrr'r r r(rrresidualself_attn_weightsmoe_hidden_statesroutputss r4rT$GraniteMoeSharedDecoderLayer.forwardsN!,,]; ,0>> , ')%+/) 3 ,  , ( !43K3K#KK !55mD +/+@+@+O( ?? "-M- 0NNM  43K3K#KK  "  + +G  ' 'Gr3)r r@r"r#r$rr%)NNNFFNFN)r(r)r*r+rr0r?rr-rVrr.r rrorr FloatTensorrTr2rWrXs@r4rrsH h5 h# h%0A6R2637+/,1$)59/4KOO||O!.Ou//0 O "% O $D> OD>O!!1!12O'tnO&eELL%,,,F&GHO45O u  (51B1BEDUDU1U+V"WW XOSOr3rcT^\rSrSr%\\S'SrSrS/rS/r Sr Sr Sr U4Sjr S rU=r$) GraniteMoeSharedPreTrainedModelir9modelTrrFc>[TU]U5 [U[5(a9URR R SURRS9 gg)Nr)ristd) r> _init_weights isinstancertr_datanormal_r9initializer_range)rIrrJs r4r5-GraniteMoeSharedPreTrainedModel._init_weights"sJ f% f= > > MM   & &CT[[5R5R & S ?r3r')r(r)r*r+rr/base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphr5r2rWrXs@r4r1r1sD ""&*#78#4"5N"TTr3r1c^\rSrSr%\R \S'SS\4U4Sjjjr\R"5\ S55r Sr U=r $)GraniteMoeSharedRotaryEmbeddingi(inv_freqr9c>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typetypedefaultrDF) persistent)r>r?hasattrr6rFdictgetrGmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr9r rope_init_fnattention_scalingregister_bufferrDoriginal_inv_freq)rIr9rrDrJs r4r?(GraniteMoeSharedRotaryEmbedding.__init__+s  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs $BF  F.)rRr9rOrTrPrQrGr)r(r)r*r+r-rVr/rr?no_gradrrTr2rWrXs@r4rCrC(sAll/5//" ]]_ < [TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9UlSUlUR$UlURUlUR&UlURUR(-UlUR,UlUR.UlUR0UlUR0S:Xa [3U5OSUlUR75 gs snf)NrFrope)r>r? pad_token_id padding_idx vocab_sizer Embeddingr@ embed_tokens ModuleListr}num_hidden_layersrlayersrZr!normgradient_checkpointingembedding_multiplierrrrrN rope_thetaposition_embedding_typerC rotary_emb post_initrs r4r?GraniteMoeSharedModel.__init__NsE  !.. ++LL):):F,>FDWDWX &+#$*$?$?!!--33((DNN: '-'E'E$ ++'-'E'E$EIEaEaekEk9&Aqu ! osF input_idsrrr inputs_embedsr r'output_hidden_statesr( return_dictr rMc UbUOURRnUbUOURRnUbUOURRnU bU OURRn USLUSL- (a [ S5eUR (a/UR(aU(a[RS5 SnUcURU5nXPR-nU(aUc[URS9nU cDUbUR5OSn [R"XUR S-UR"S9n UcU R%S5nUR'X%XU5nUnSnUR(bUR)X5nU(aSOSnU(aSOSnU (aSOSnUR*HEnU(aUU4- nU"UUUUUUU U US 9 nUSnU(a UUS4- nU (dM<UUS 4- nMG UR-U5nU(aUU4- nU (d[/S XUU455$[1UUUUUS 9$) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.F)r9rrrr')rrrr'r r r(rrOc3.# UH ocMUv M g7frr').0vs r4 0GraniteMoeSharedModel.forward..s^a^s )last_hidden_staterrL attentionsr)r9r'ryr use_return_dictrrprrrrkrqr get_seq_lengthr-arangerprr_update_causal_maskrtrnroror)rIrwrrrrxr r'ryr(rzr rpast_seen_tokensrrLrall_hidden_statesall_self_attnsall_router_logits decoder_layer layer_outputss r4rTGraniteMoeSharedModel.forwardgs 2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKO  !CRC^==?de "\\ ]5H5H5K"KTaThThN  )33A6L.. >L] & " ?? &"&//-"N #7BD0d"6BD![[M#!m%55!)*) /"3#-%9$7 M*!,M =#3"55##!mB&7%99!-)0 -0  -!1 1 )T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCK[Kr3rrrerc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuererr)diagonalr|rOr)rQr-rrfullrtriurrrclonerprf masked_fill) rrrrer rrrr mask_length padding_masks r4rKGraniteMoeSharedModel._prepare_4d_causal_attention_mask_with_cache_positions}<  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r3)rkrqrprr@rnrNrorrhrsrrrtri) NNNNNNNNNNN)F)r(r)r*r+rr?rrr-r.rVrr listr/rrorrTr staticmethodr0rerr2rWrXs@r4rdrdLs52151537KO59$(,0/3/3&*59h E,,-h !.h u//0 h "%tE4E4E/F(F"GH h   1 12 h D>h $D>h 'tnh 'tnh d^h !!1!12h  u-- .h h `#( BellK78BllB B  B  BH4 444{{ 4  4  44r3rd gate_logitsrvcUb[U[5(dg[U[5(aCUSRn[R"UVs/sHoUR U5PM snSS9n[R RRWSS9n[R"XrSS9up[R RRX5n Uc:[R"U R5SS9n [R"USS9n GO"URupURSX--nUSSS2SS2SS4RXXU45RSX!5R W5n[R "U R5U-SS9[R "USS9- n USSS2SS2S4RXXRS45RSURS5R U5n[R "UU-SS9[R "USS9- n URR"bURR"OSnURS[%U5-n[R "U SS2UUURS-24U R'S5-5nUU-$s snf)ax Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. NrrPrOr)r6rorr-rrfrrrrone_hotrirrprrrindexr0r)rrvrrcompute_device layer_gateconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expertrrrmexpert_attention_mask router_per_expert_attention_mask device_indexrank overall_losss r4load_balancing_loss_funcrNs:*[%"@"@+u%%$Q..#(99^i-j^iPZmmN.K^i-jpq#r hh))112JPR1SO**_DA((%%--.>LK!JJ{'8'8':B"'O!C&4&:&:# 4::1=*B^_ 4AtT1 2 V&OKX Y WR , R  "IIk&7&7&9[TU]U5 [U5UlURUl[ R "URURSS9UlURUl URUl URUl UR5 g)NFr<)r>r?rdr2rirrEr@lm_headrouter_aux_loss_coefrrvrrurHs r4r?$GraniteMoeSharedForCausalLM.__init__s  *62  ++yy!3!3V5F5FUS $*$?$?!!33#)#=#=  r3rwrrrrxlabelsr r'ryr(rzr logits_to_keeprMc UbUOURRnU bU OURRn U bU OURRn U bU OURRn UR "SUUUUUUUU U U U S. UD6nUSn[ U [5(a [U *S5OU nURUSS2USS245nUURR- nSnUb:UR5nUR"UU4SURR0UD6nSnU (af[U (a UROUSUR UR"U5nUb+UUR$UR'UR(5-- nU (d!U4USS-nU (aU4U-nUbU4U-$U$[+UUUUR,UR.UR0URS9$) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, GraniteMoeSharedForCausalLM >>> model = GraniteMoeSharedForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rwrrrrxr r'ryr(rzr rrirOr)lossaux_lossrrrLrrr')r9r'r(ryrr2r6r0slicerlogits_scalingr loss_functionrirrrvrrrfrrrrLr)rIrwrrrrxrr r'ryr(rzr rrr-rL slice_indicesrrroutputs r4rT#GraniteMoeSharedForCausalLM.forwards)P2C1N-TXT_T_TqTq$8$D $++JjJj %9$D $++JjJj &1%SV8W8W~ot4]k mA}a,?@A$++444  \\^F%% ;;11 D /)4%%'"+  (( H !11HKK 4LLLY,F#"v-'+'7D7V# CV C(#33!//))!//  r3)rr2rvrrri) NNNNNNNNNNNNr)r(r)r*r+_tied_weights_keysrr?rrr-r.rVrr rr/rr0rorrTr2rWrXs@r4rrsw*+ 5 151537KO59-1$(,0/3/3&*5934k E,,-k !.k u//0 k "%tE4E4E/F(F"GH k   1 12 k ))*k D>k $D>k 'tnk 'tnk d^k !!1!12k c5<</0k u// 0!k k r3r)rrdr1)Nr)r)Nr;N)Gtypingrrrrr-torch.nn.functionalrrr activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrutils.deprecationrconfiguration_granitemoesharedr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerr(rr Moduler7rZrtrrrrrVr0rrrrrr1rCrdrorr__all__r'r3r4rsZ,87 !.)>9jjKF&JJ0B !!;J   H %)52"))4JbiiJ(*bii*Z-S-S`9+"))9+x(6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%:T) T)n^#=^BToTT"!?S&#S&U\\* S&  5<<  S&l| "A?| ~ fr3