h SSKrSSKJrJrJr SSKrSSKJrJr SSKJ r SSK J r \ R"\R5\ R"S5:a,SSKJr \"\"SSS 9\"SSS 9S .5\"SSS 9\"SSS 9S .S .rO \"SSS 9\"SSS 9S ./\"SSS 9\"SSS 9S .S .rSS \ S\SS\\4SjjrS \ 4SjrSS\S \ S\\S\\4Sjjrg)N)CallableLiteralOptional)DatasetValue)version) AutoTokenizerz4.0.0)Liststring)dtypeid)contentrole) completionprompt)chatml instruction tokenizermessages_field)messages conversationstoolsc^^^UUU4SjnU$)z return a callable function that takes in a "messages" dataset and returns a formatted dataset, based on the tokenizer apply chat template to the dataset along with the schema of the list of functions in the tools list. c >[UTS[5(aH/n[[UT55H)nUR TR UTUSTS95 M+ U$TR UTSTS9$)NrF)tokenizer isinstancelistrangelenappendapply_chat_template)examples output_textsirrrs W/home/james-whalen/.local/lib/python3.13/site-packages/trl/extras/dataset_formatting.pyformat_dataset9conversations_formatting_function..format_dataset.s h~.q14 8 8L3x789##11(>2J12MX]ej1k: 00.1ITYaf0g g)rrrr's``` r&!conversations_formatting_functionr+&s h r)c^U4SjnU$)z return a callable function that takes in an "instructions" dataset and returns a formatted dataset, based on the tokenizer apply chat template to the dataset c">[US[5(aW/n[[US55H8nSUSUS.SUSUS./nUR TR USS95 M: U$SUSS.SUSS./nTR USS9$)Nruser)rr assistantrF)rr)r#r$r%converted_samplers r&r'8instructions_formatting_function..format_datasetBs hx($ / /L3x123#0B10EF(Xl5KA5NO$ ##I$A$ABR]b$A$cd 4   HX,>?$,1GH  001AE0R Rr)r*)rr's` r& instructions_formatting_functionr2<s S" r)datasetreturnc[U[5(aSUR;a=URS[S:Xa#[R "S5 [ USU5$SUR;a?URS[S:Xa#[R "S5 [ USU5$gUR[S:Xa![R "S5 [U5$g)aY Finds the correct formatting function based on the dataset structure. Currently supported datasets are: - `ChatML` with [{"role": str, "content": str}] - `instruction` with [{"prompt": str, "completion": str}] Args: dataset (Dataset): User dataset tokenizer (AutoTokenizer): Tokenizer used for formatting tools (list, *optional*): List of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. Returns: Callable: Formatting function if the dataset format is supported else None rrz%Formatting dataset with chatml formatrrz*Formatting dataset with instruction formatN)rrfeaturesFORMAT_MAPPINGlogginginfor+r2)r3rrs r& get_formatting_func_from_datasetr:Vs"'7## )) ) +~h/GG DE8JPUVV g.. .0N84LL DE8OUZ[[M     !> > LLE F3I> > r))N)r8typingrrrdatasetsrr packagingr transformersr parse __version__r r7rr+r2r:r*r)r&rAs...#& ==%%&'--*@@5xD#A5W_dhKijk&+(t&DPU\dimPnoN %8=uS[`dGefg&+(t&DPU\dimPnoNmq.56Q.R[cdh[i, 6IM !.7?~ hr)