hISSKrSSKJr SSKJrJrJrJr SSKJ r "SS\R5r "SS\ 5r "S S \ 5r g) N)AutoModelForCausalLMAutoModelForSeq2SeqLMis_torch_npu_availableis_torch_xpu_available)PreTrainedModelWrapperc2^\rSrSrSrU4SjrSrSrU=r$) ValueHeadz] The ValueHead class implements a head for GPT2 that returns a scalar for each output token. c >[TU]5 [US5(dURSS5nO URnU(a[ R "U5O[ R"5Ul[US5(a URn[US5(a URnOd[US5(aSUR(aB[US5(a1[URS5(aURRn[ R"WS5Ul[ R"5Ulg)Nsummary_dropout_probg? hidden_sizeword_embed_proj_dimis_encoder_decoderdecoderr)super__init__hasattrpopr nnDropoutIdentitydropoutrrrrLinearsummaryFlattenflatten)selfconfigkwargsr r __class__s X/home/james-whalen/.local/lib/python3.13/site-packages/trl/models/modeling_value_head.pyrValueHead.__init__s v566#)::.Dc#J #)#>#> ;Orzz"67UWU`U`Ub  6= ) ) ,,K 60 1 1 44K V1 2 2((WVY-G-G6>>=99"(.."<"a  An autoregressive model with a value head in addition to the language model head. This class inherits from `~trl.PreTrainedModelWrapper` and wraps a `transformers.PreTrainedModel` class. The wrapper class supports classic functions such as `from_pretrained`, `push_to_hub` and `generate`. To call a method of the wrapped model, simply manipulate the `pretrained_model` attribute of this class. Class attributes: - **transformers_parent_class** (`transformers.PreTrainedModel`) -- The parent class of the wrapped model. This should be set to `transformers.AutoModelForCausalLM` for this class. - **supported_args** (`tuple`) -- A tuple of strings that are used to identify the arguments that are supported by the `ValueHead` class. Currently, the supported args are: - **summary_dropout_prob** (`float`, `optional`, defaults to `None`) -- The dropout probability for the `ValueHead` class. - **v_head_initializer_range** (`float`, `optional`, defaults to `0.2`) -- The initializer range for the `ValueHead` if a specific initialization strategy is selected. - **v_head_init_strategy** (`str`, `optional`, defaults to `None`) -- The initialization strategy for the `ValueHead`. Currently, the supported strategies are: - **`None`** -- Initializes the weights of the `ValueHead` with a random distribution. This is the default strategy. - **"normal"** -- Initializes the weights of the `ValueHead` with a normal distribution. r v_head_initializer_rangev_head_init_strategyc >[TU]"U40UD6 URU5un n[URR 40UD6UlUR"S0UD6 g)aZ Initializes the model. Args: pretrained_model (`transformers.PreTrainedModel`): The model to wrap. It should be a causal language model such as GPT2. or any model mapped inside the `AutoModelForCausalLM` class. kwargs (`dict`, `optional`): Additional keyword arguments, that are passed to the `ValueHead` class. N)rr _split_kwargsr pretrained_modelrv_head _init_weightsrr?r v_head_kwargs_r!s r"r*AutoModelForCausalLMWithValueHead.__init__\s] )4V4"008 q! 5 5 < <N N  +]+r$c BURSS5nURSS5nUcgUS:XaqURRRRR SUS9 URRR RR5 gg)a. Initializes the weights of the value head. The default initialization strategy is random. Users can pass a different initialization strategy by passing the `v_head_init_strategy` argument when calling `.from_pretrained`. Supported strategies are: - `normal`: initializes the weights with a normal distribution. Args: **kwargs (`dict`, `optional`): Additional keyword arguments, that are passed to the `ValueHead` class. These arguments can contain the `v_head_init_strategy` argument as well as the `v_head_initializer_range` argument. r:皙?r;Nnormalmeanstdrr@rr(datanormal_biaszero_rr initializer_range init_strategys r"rA/AutoModelForCausalLMWithValueHead._init_weightsls#JJ'A3G #94@   h & KK   & & + + 3 3BS 3 T KK   $ $ ) ) / / 1'r$c SUS'X%S'UR(a5URRRS:XaUR S5 UR"SUUS.UD6nUR SnUR nURn URURRRR:wa9URURRRR5nURU5RS5n UR[R :waUR#5nU(aXXR$4$XU 4$)a Applies a forward pass to the wrapped model and returns the logits of the value head. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. past_key_values (`tuple(tuple(torch.FloatTensor))`, `optional`): Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model (see `past_key_values` input) to speed up sequential decoding. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. return_past_key_values (bool): A flag indicating if the computed hidden-states should be returned. kwargs (`dict`, `optional`): Additional keyword arguments, that are passed to the wrapped model. Toutput_hidden_statespast_key_values PREFIX_TUNING) input_idsattention_maskr=) is_peft_modelr?active_peft_config peft_typerr*logitslossdevicer@rr(r)squeezer'torchfloat32floatrX rrZrXr[return_past_key_valuesr base_model_outputlast_hidden_state lm_logitsravalues r"r,)AutoModelForCausalLMWithValueHead.forwards92*.%&$3 !   $"7"7"J"J"T"TXg"g JJ( ) 11 )   .;;B?%,,  %%  # #t{{':':'A'A'H'H H 1 4 4T[[5H5H5O5O5V5V W  -.66r: ??emm +!)I !U,M,MN NU+ +r$c:URR"U0UD6$)a A simple wrapper around the `generate` method of the wrapped model. Please refer to the [`generate`](https://huggingface.co/docs/transformers/internal/generation_utils) method of the wrapped model for more information about the supported arguments. Args: *args (`list`, *optional*): Positional arguments passed to the `generate` method of the wrapped model. **kwargs (`dict`, *optional*): Keyword arguments passed to the `generate` method of the wrapped model. r?generaterargsr s r"rp*AutoModelForCausalLMWithValueHead.generates $$--t>v>>r$cUR(dURR"U0UD6nO0nURR"U0UD6nUR 5H upVXcSU3'M U$z Returns the state dictionary of the model. We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the key with `v_head.`. v_head.r]r? state_dictr@itemsrrrr pretrained_model_state_dictv_head_state_dictkvs r"rx,AutoModelForCausalLMWithValueHead.state_dictu !!*.*?*?*J*JD*[TZ*[ '+- ' KK22DCFC%++-DA9:'! 6.**r$cpURURlURR"U0UD6$r&r@r? push_to_hubrqs r"r-AutoModelForCausalLMWithValueHead.push_to_hub0'+{{$$$00$A&AAr$c8^[UR55H.nSU;dM URU5XRSS5'M0 URR USS9 A[ URS5(GaSURRR5;d(SURRR5;a [S5e[[URRR555S m[T[5(a/[5(aS T3mO[5(aS T3mOS T3mURR!T5UlU4S jnUR#U5 SUlgg) We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the key with `v_head.`. This function removes the `v_head.` prefix from the keys of the value head state dictionary. rvFstrict hf_device_mapcpudiskdThe model is offloaded on CPU or disk - CPU & disk offloading is not supported for ValueHead models.rznpu:zxpu:zcuda:c>SnUH=n[U[R5(aX4RT54- nM8X44- nM? U$)Nr= isinstancerdTensorr))moduleinputoutputs new_outputr+ first_devices r"set_device_hookDAutoModelForCausalLMWithValueHead.post_init..set_device_hooksI %F!&%,,77"yy'>&@@ "i/ & "!r$TN)listkeysrreplacer@load_state_dictrr?rvalues ValueErrorsetrintrrr)register_forward_hookis_sequential_parallel)rrxr}rrs @r" post_init+AutoModelForCausalLMWithValueHead.post_inits\ joo'(AA~7A~~a7H 99Y34) ##Ju#=  4((/ : :..<<CCEET22@@GGII z D$9$9$G$G$N$N$P QRSTUL,,,)++%),#8L+--%),#8L%*<.#9L++..6DK "  & & 7*.D '= ;r$)rr@NNNF)r.r/r0r1r2rtransformers_parent_classsupported_argsrrAr,rprxrrr3r4r5s@r"r7r7>sO,!5N , 20$ 5,n ?+ B */*/r$r7cr^\rSrSrSr\r/SQrSrU4Sjr Sr Sr Sr S r S rSS jrS rS rU=r$)"AutoModelForSeq2SeqLMWithValueHeadi a} A seq2seq model with a value head in addition to the language model head. This class inherits from `~trl.PreTrainedModelWrapper` and wraps a `transformers.PreTrainedModel` class. The wrapper class supports classic functions such as `from_pretrained` and `push_to_hub` and also provides some additional functionalities such as `generate`. Args: pretrained_model (`transformers.PreTrainedModel`): The model to wrap. It should be a causal language model such as GPT2. or any model mapped inside the `AutoModelForSeq2SeqLM` class. kwargs: Additional keyword arguments passed along to the `ValueHead` class. )lm_head embed_outoutput_projectionr9c >[TU]"U40UD6 URU5un nSUlUR 5(d [ S5e[ URR40UD6Ul UR"S0UD6 g)NTzOThe model does not have a language model head, please use a model that has one.r=) rrr>r _has_lm_headrr r?rr@rArBs r"r+AutoModelForSeq2SeqLMWithValueHead.__init__ s~ )4V4"008 q!"&  ""no o 5 5 < <N N  +]+r$c^URR5H,umn[U4SjUR55(dM, g g)Nc3,># UH oT;v M g7fr&r=.0 attributenames r" BAutoModelForSeq2SeqLMWithValueHead._has_lm_head../sK6J$6JTF)r? named_modulesanylm_head_namings)r_modulers @r"r/AutoModelForSeq2SeqLMWithValueHead._has_lm_head,s@!22@@BMD'Kd6J6JKKKCr$c^^[UR55H.nSU;dM URU5XRSS5'M0 URR USS9 A[ URS5(aSURRR5;d(SURRR5;a [S5eURR5HBumn[U4S jUR55(dM,URRm O URR!T5UlU4S jnUR#U5 S Ulg g ) rrvrFrrrrrc3,># UH oT;v M g7fr&r=rs r"r?AutoModelForSeq2SeqLMWithValueHead.post_init..JsO:NYD(:Nrc>SnUH=n[U[R5(aX4RT54- nM8X44- nM? U$)a- A hook that sets the device of the output of the model to the device of the first parameter of the model. Args: module (`nn.Module`): The module to which the hook is attached. input (`tuple`): The input to the module. outputs (`tuple`): The output of the module. r=r)rrrrr+lm_head_devices r"rEAutoModelForSeq2SeqLMWithValueHead.post_init..set_device_hookQsK %F!&%,,77"yy'@&BB "i/ & "!r$TN)rrrrr@rrr?rrrrrrr(rbr)rr)rrxr}rrrrs @@r"r,AutoModelForSeq2SeqLMWithValueHead.post_init3s9 joo'(AA~7A~~a7H 99Y34) ##Ju#=  4((/ : :..<<CCEET22@@GGII z !% 5 5 C C E fO$:N:NOOO%+]]%9%9N!F ++..8DK "*  & & 7*.D 'Q ;r$cUR(dURR"U0UD6nO0nURR"U0UD6nUR 5H upVXcSU3'M U$rurwrzs r"rx-AutoModelForSeq2SeqLMWithValueHead.state_dictirr$cpURURlURR"U0UD6$r&rrqs r"r.AutoModelForSeq2SeqLMWithValueHead.push_to_hubyrr$c BURSS5nURSS5nUcgUS:XaqURRRRR SUS9 URRR RR5 gg)z. We initialize the weights of the value head. r:rGr;NrHrIrJrMrRs r"rA0AutoModelForSeq2SeqLMWithValueHead._init_weights~s#JJ'A3G #94@   h & KK   & & + + 3 3BS 3 T KK   $ $ ) ) / / 1'r$c X%S'UR(a5URRRS:XaUR S5 UR"SUUSS.UD6nUR SnUR nURn URU5RS5n UR[R:waUR5nU(aXXR4$XU 4$)NrXrYT)rZr[rWr\r=)r]r?r^r_rdecoder_hidden_statesr`rar@rcr'rdrerfrXrgs r"r,*AutoModelForSeq2SeqLMWithValueHead.forwards%4 !   $"7"7"J"J"T"TXg"g JJ( ) 11 )!%   .CCBG%,,  %% -.66r: ??emm +!)I !U,M,MN NU+ +r$c:URR"U0UD6$)z* We call `generate` on the wrapped model. rorqs r"rp+AutoModelForSeq2SeqLMWithValueHead.generates $$--t>v>>r$)rrr@r)r.r/r0r1r2rrrrrrrrxrrAr,rpr3r4r5s@r"rr sZ !6CON ,4/l+ B 2 $ ,D??r$r)rdtorch.nnr transformersrrrr modeling_baserModuler r7rr=r$r"rsF tt1% %PH/(>H/Vi?)?i?r$