bCi\SSKJrJrJr SSKrSSKJr SSKJr SSK J r J r SSK J r SSKJrJr SSKJr SS KJr SS KJrJr SS KJrJr SS KJrJr SS KJr SSK J!r!J"r"J#r# SSK$J%r% SSK&J'r' SSK(J)r) "SS\RT5r+"SS\RT5r,S\RZS\.S\RZ4Sjr/S3S\RTS\RZS\RZS\RZS\\RZS \0S!\0S"\\!4S#jjr1S$r2S4S%jr3"S&S'\RT5r4"S(S)\RT5r5"S*S+\5r6\""S,S-\55r7\""S.S/\755r8\""S0S1\7\ 55r9/S2Qr:g)5)CallableOptionalUnionN)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)deprecate_kwarg)check_model_inputs) Cohere2Configc^\rSrSr%\R \S'SS\4U4Sjjjr\R"5\ S55r Sr U=r $)Cohere2RotaryEmbedding+inv_freqconfigc>[TU]5 [US5(aZ[UR[ 5(a;URR SURR S55UlOSUlURUl URUl Xl [URUl URURU5uo0lURSUSS9 UR Ulg)N rope_scaling rope_typetypedefaultr F) persistent)super__init__hasattr isinstancer#dictgetr$max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr!r rope_init_fnattention_scalingregister_bufferr original_inv_freq)selfr!devicer __class__s f/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/cohere2/modeling_cohere2.pyr)Cohere2RotaryEmbedding.__init__.s  6> * *z&:M:Mt/T/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acde ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk ^^ U C&,,.1F1L1L1NNYYZ[]^_E))%;C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs BF F)r2r!r/r4r0r1r$N)__name__ __module__ __qualname____firstlineno__rJTensor__annotations__rr)no_gradrrW__static_attributes__ __classcell__r7s@r8rr+s@ll/}//" ]]_ < [TU]5 [R"[R "U55UlX lg)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)r(r)nn ParameterrJonesweightvariance_epsilon)r5 hidden_sizeepsbiasr7s r8r)Cohere2LayerNorm.__init__Ps- ll5::k#:; #r:cURnUR[R5nUR SSS9nX- R S5R SSS9nX- [R "X@R-5-nURR[R5U-nURU5$)Nr<T)keepdimrA) rErPrJfloat32meanpowrsqrtrlrk)r5 hidden_states input_dtypertvariances r8rWCohere2LayerNorm.forwardVs#)) %((7 !!"d!3!(--a055b$5G&-XH]H]=]1^^  u}}5 E  ,,r:)rlrk)Ngh㈵>FrZr[r\r]r)rWrarbrcs@r8rereOs$ --r:rerwn_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rHrGreshape)rwr|batchnum_key_value_headsslenhead_dims r8 repeat_kvr`s_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr:modulequerykeyvalueattention_maskscalingdropoutkwargsc @[X R5n[X0R5n [R"XR SS55U-n Ub"USS2SS2SS2SUR S24n X-n [ RRU S[RS9RUR5n [ RRXURS9n [R"X5n U R SS5R5n X4$)NrArr<)rCrE)ptrainingr)rnum_key_value_groupsrJmatmulrLrHrh functionalsoftmaxrsrPrErr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r8eager_attention_forwardrls3 ; ;C/0Gw;q>C/0G ::E: "GJJUJ$; ;;r:cf^\rSrSrSrSS\S\\4U4Sjjjr\ "SSSS 9SS \ RS \ \ R\ R4S \\ RS\\ S \\ RS\\S\ \ R\\ R\\ \ R44Sjj5rSrU=r$)Cohere2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperr! layer_idxc>[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URS-Ul URUl SUl URUS:Xa UROSUl[R "UR UR UR-UR"S9Ul[R "UR URUR-UR"S9Ul[R "UR URUR-UR"S9Ul[R "UR UR-UR UR"S9Ulg)NrgTsliding_attentionro)r(r)r!rgetattrrmnum_attention_headsrrrrattention_dropout is_causal layer_typessliding_windowrhLinearattention_biasq_projk_projv_projo_projr5r!rr7s r8r)Cohere2Attention.__init__sm  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!97=7I7I)7TXk7kf33quii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r:past_key_valuepast_key_values4.58new_nameversionrwposition_embeddingsrcache_positionrr}c dURSSn/UQSPURP7nURU5RU5R SS5n UR U5RU5R SS5n UR U5RU5R SS5n UupURb[XX5upUb$XUS.nURXURU5up[nURRS:wa[URRnU"UU U U U4UR(dSO UR UR"URS.UD6unnUR$"/UQSP76R'5nUR)U5nUU4$)Nr<rrA)rOrNreager)rrr)rHrrviewrLrrrrupdaterrr!_attn_implementationrrrrrrr)r5rwrrrrr input_shape hidden_shape query_statesrrrNrO cache_kwargsattention_interfacerrs r8rWCohere2Attention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &    *';LVY'_ $L  &#&nUL'6'='=jX\XfXfht'u $J(? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$2H2HLL.. %  % ! \"));;;;FFH kk+. L((r:) rr!rrrrrrrrrrrY)NN)rZr[r\r]__doc__rrintr)rrJr^tupler LongTensorrr rWrarbrcs@r8rrsG } #  0%0A6R ,059 *)||*)#5<<#=>*)!. *) "% *) !!1!12 *)-.*) u||Xell3XeELL>Q5RR S*)S*)r:rc.^\rSrSrU4SjrSrSrU=r$) Cohere2MLPc>[TU]5 XlURUlURUl[ R "URURSS9Ul[ R "URURSS9Ul[ R "URURSS9Ul [URUl gNFr) r(r)r!rmintermediate_sizerhr gate_projup_proj down_projr hidden_actact_fnr5r!r7s r8r)Cohere2MLP.__init__s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r:cURURURU55URU5-5nU$rY)rrrr)r5rQrs r8rWCohere2MLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r:)rr!rrrmrrr{rcs@r8rrs0r:rcb^\rSrSrS\S\4U4Sjjr\"SSSS9SS \RS \ \R\R4S \ \RS\ \ S \ \ S \ \RS\\S\ \R"\ \ \R"\R"444Sjj5rSrU=r$)Cohere2DecoderLayerir!rc>[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul URUUl g)N)r!rrmrn) r(r)rmr self_attnrmlprelayer_norm_epsinput_layernormrattention_typers r8r)Cohere2DecoderLayer.__init__sc !--)Mf%/V=O=OV\VkVkl$00;r:rrrrrwrr use_cacherrr}c UnURU5nUR"SUUUUUUS.UD6upURU5n X-U -nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )rwrrrrr)rrr) r5rwrrrrrrresidualhidden_states_attention_hidden_states_mlps r8rWCohere2DecoderLayer.forwardsq<!,,]; %)^^& ' 3)+) & & "!HH]3 :=NN r:)rrmrrr)NNFN)rZr[r\r]rrr)rrJr^rrrboolrrr FloatTensorrWrarbrcs@r8rrs<}<<%0A6R 26+/$)59+||+#5<<#=>+!. + "% + D> +!!1!12+-.+ u  (51B1BEDUDU1U+V"WW X+S+r:rcR\rSrSr%\\S'SrSrS/rS/r Sr Sr Sr Sr Sr\\S.rSrg ) Cohere2PreTrainedModeli=r!modelTrr)rw attentionsrN)rZr[r\r]rr_base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsrarr:r8rr=sQ &*#./#4"5N!"&,&r:rc"^\rSrSrS\4U4Sjjr\"5\SS\\ RS\\ RS\\ RS\\ S\\ RS \\S \\ RS \\S \4S jj55rSrU=r$) Cohere2ModeliPr!c >[TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR)5 gs snf)Nrr!F)r(r) pad_token_id padding_idx vocab_sizerh Embeddingrm embed_tokens ModuleListrangenum_hidden_layersrlayersrernormr rotary_embgradient_checkpointing post_initrs r8r)Cohere2Model.__init__Rs  !.. ++LL):):FO  !CRC^==?de "\\ ]5H5H5K"KTaThThN  )33A6L?-FF++ -"0"0#2 , K#5"C{"C%F%U%U#  & "oomJ![[M)$723O3OP /#- M) -0 &++  r:)rrrrrrr)NNNNNNN)rZr[r\r]rr)rrrrJrr^rrrrrrrWrarbrcs@r8r r Ps} 151537+/59$(59< E,,-< !.< u//0 < "% <   1 12 < D>< !!1!12< +,<  !< < r:r c^\rSrSrS/rSS0rSS/S/40rU4Sjr\\ SS\ \ RS \ \ RS \ \ RS \ \\\\ R"4S \ \ R"S \ \ RS\ \S\ \S\ \S\ \ RS\\\ R4S\\S\4Sjj55rSrU=r$)Cohere2ForCausalLMizlm_head.weightlm_head colwise_reprwlogitsc(>[TU]U5 [U5UlURUl[ R "URURSS9UlURUl URUl UR5 gr) r(r)r rrrhrrmr+ logit_scaletie_word_embeddingsrrs r8r)Cohere2ForCausalLM.__init__sq  !&)  ++yy!3!3V5F5FUS !--#)#=#=  r:rrrRrrlabelsroutput_attentionsoutput_hidden_statesrlogits_to_keeprr}c UbUOURRnU bU OURRn UR"SUUUUUUUU U S. U D6n U Rn[ U [ 5(a [U *S5OU nURUSS2USS245nUUR-nSnUb)UR"SUX`RRS.U D6n[UUU RU RU RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, Cohere2ForCausalLM >> model = Cohere2ForCausalLM.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrRrrrr3r4r)r-r2r)lossr-rrwrr)r!r3r4rr r+rslicer+r/ loss_functionrrrrwr)r5rrrRrrr2rr3r4rr5routputsrw slice_indicesr-r7s r8rWCohere2ForCausalLM.forwards(N2C1N-TXT_T_TqTq$8$D $++JjJj  ,0:: , )%+'/!5) ,  ,  11 8B>SV8W8W~ot4]k mA}a,?@A$***  %%pVF{{OeOepiopD%#33!//))   r:)r+r/rr0r) NNNNNNNNNNr)rZr[r\r]_tied_weights_keys_tp_plan_pp_planr)rrrrJrr^rrlistrrrrrrrWrarbrcs@r8r*r*sw*+=)H_-z:;H 151537KO59-1$(,0/35934H E,,-H !.H u//0 H "%tE4E4E/F(F"GH H   1 12 H ))*H D>H $D>H 'tnH !!1!12H c5<</0H +,H  H H r:r*)r*r r)r)Nr);typingrrrrJtorch.nnrh activationsr cache_utilsrr generationr masking_utilsr r modeling_flash_attention_utilsr modeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrutils.deprecationr utils.genericrconfiguration_cohere2rModulerrer^rrrFrrrrrrrr r*__all__rr:r8rSs,-, !.)RB9OKF&II0/0!