h2 S r SSKrSSKrSSKJr SSKJr SSKJrJ r J
r
SSK J r SSK
JrJr SSKJr \R$ R' S S
5 S r\S :X a \" 5 gg)
z
pip install math_verify num2words peft trackio vllm
export TRACKIO_PROJECT="RLOO-NuminaMath-TIR"
accelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml examples/scripts/rloo.py
N) load_dataset)NormalizationConfig)LatexExtractionConfigparseverify)
LoraConfig)
RLOOConfig RLOOTrainer)think_format_rewardTRACKIO_SPACE_IDz trl-trackioc ^ [ SSS/S9u pSmU4S jnU R USS/S 9n UR USS/S 9nS
[ [ 4S jn[ S S
[
R 0S[ SS9SSSSSSSSSSS9n[ SU[ U/U U[ 5 S9nUR 5 UR UR 5 UR SS 9 g ) NzAI-MO/NuminaMath-TIRz
train[:5%]z test[:5%])splita[ A conversation between user and assistant. The user asks a question, and the assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within tags, i.e.,
This is my reasoning.
This is my answer.c " > SSTS.SU S S./0$ )Npromptsystem)rolecontentuserproblem )example
SYSTEM_PROMPTs O/home/james-whalen/.local/lib/python3.13/site-packages/examples/scripts/rloo.pymake_conversation main..make_conversation< s* !m< GI,>?
messagesr )remove_columnssolutionc 6 / nU Vs/ s H
oDS S PM nn[ XQ5 H u pg [ USS9n[ U5 S:w a6 [ U[ [ SSSSSS9SSS 9/SS
9n [
[ X5 5 n
OF[
UR 5 R 5 UR 5 R 5 :H 5 n
UR U
5 M U$ s snf ! [ a / n Nf = f! [ a n [ S U S U S
U 35 Sn
Sn A NSSn A ff = f)u Reward function that checks if the completion matches the ground truth.
- If both gold and prediction are parseable → use math verification.
- If not parseable → compare as normalized text.
r r first_match)extraction_modeFTall)nitsmalformed_operators basic_latexboxedunits)normalization_configboxed_match_prioritytry_extract_without_anchor)extraction_configr" zverify failed: z
, answer: z, gold: N) zipr Exceptionlenr r floatr printstriplowerappend) completionsr kwargsrewards
completioncontentsr sol gold_parsed
answer_parsedrewardes r accuracy_reward main..accuracy_rewardH s;
?JK{qM),{K 3LG
! #C G ; 1$"$) 15H).8=04*/*. 6" 67;@
+ )6 %M" #6+#E FF w}} 4 4 6#))+:K:K:M MN NN6 "E 4H K L
!
!0 ! "OA3j #OP !F"s. C
C 4C. C+*C+.
D8DDzQwen3-0.6B-RLOOdtypegh㈵>F)
use_reentrantT i i colocateg ?z Qwen3-0.6B-RLOO-NuminaMath-TIR)
output_dirmodel_init_kwargs
learning_rate gradient_checkpointing_kwargslog_completionsnum_completions_to_printmax_prompt_lengthmax_completion_lengthgradient_accumulation_stepssteps_per_generationuse_vllm vllm_modevllm_gpu_memory_utilizationrun_namezQwen/Qwen3-0.6B)modelargs reward_funcs
train_dataset eval_dataset peft_config) dataset_name)r mapliststrr torchbfloat16dictr
r r train
save_modelrF push_to_hub)rW rX r r?
training_argstrainerr s @r mainrf 1 s "./El\gMh"i M 3
"%%&7U^H_%`M ##$5zS\F]#^L+tCy +\ $"ENN3&*&? !" "$% $'1 M"
)?;#! L
G MMO }//0 %;rs sY 6
! 5 < < ' +
(- 8c=L zF r