h
S r SSKrSSKrSSKJr SSKJr SSKJrJ r J
r
SSK J r J
r
JrJrJrJrJrJr SSKJr \R, R/ SS 5 \S
:X Ga \" \\ \45 r\R5 5 u rr r \ R< S ; a \ R< O\ " \\ R< 5 r \ " \ RB \ RD \ S 9\ l# \" \ 5 r$\$b# \" 5 \ RF S
' \$\ RF S' \" SSS9r%\%RM SSS9r%Sr'S r(\%RS \(5 r%S r*\%RW \*5 r%S r,\%RS \,5 r%\%S r-\ R\ S:w a \%S OSr/S\0\1 4S jr2\
" \ Rf \ \\2/\-\/\" \ 5 S 9r4\4Rk 5 \4Rm \ Rn 5 \ Rp ( a \4Rq \Rr S 9 ggg) aZ
pip install math_verify
# For Qwen/Qwen2.5-VL-3B-Instruct
accelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml examples/scripts/grpo_vlm.py --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct --output_dir grpo-Qwen2.5-VL-3B-Instruct --learning_rate 1e-5 --gradient_checkpointing --dtype bfloat16 --max_prompt_length 2048 --max_completion_length 1024 --use_vllm --vllm_mode colocate --use_peft --lora_target_modules "q_proj", "v_proj" --log_completions
# For HuggingFaceTB/SmolVLM2-2.2B-Instruct
pip install num2words
accelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml examples/scripts/grpo_vlm.py --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct --output_dir grpo-SmolVLM2-2.2B-Instruct --learning_rate 1e-5 --dtype bfloat16 --max_prompt_length 2048 --max_completion_length 1024 --use_peft --lora_target_modules "q_proj", "v_proj" --log_completions --per_device_train_batch_size 1 --gradient_accumulation_steps 2 --num_generations 2
N) load_dataset)NormalizationConfig)LatexExtractionConfigparseverify)
GRPOConfig GRPOTrainer ModelConfigScriptArguments TrlParserget_kbit_device_mapget_peft_configget_quantization_config)think_format_rewardTRACKIO_SPACE_IDz trl-trackio__main__)autoN)revisionattn_implementationdtype
device_mapquantization_configz'lmms-lab/multimodal-open-r1-8k-verifiedtrain)splitd * ) test_sizeseeda[ A conversation between user and assistant. The user asks a question, and the assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within tags, i.e.,
This is my reasoning.
This is my answer.c , S[ S.SU S S./nSU0$ )Nsystem)rolecontentuserproblemprompt)
SYSTEM_PROMPT)exampler% s S/home/james-whalen/.local/lib/python3.13/site-packages/examples/scripts/grpo_vlm.pymake_conversationr) | s+ - 8 (: ;
&!! c b U S nUR S S: =( a UR S S: $ )Nimager i )sizer' r, s r( filter_big_imagesr0 s1 zz!}s":uzz!}s'::r* c Z U S nUR S:w a UR S5 nXS' U $ )Nr, RGB)modeconvertr/ s r( convert_to_rgbr5 s3 :: MM%(E r* notestsolutionc 6 / nU Vs/ s H
oDS S PM nn[ XQ5 H u pg [ USS9n[ U5 S:w a6 [ U[ [ SSSSSS9SSS 9/SS
9n [
[ X5 5 n
OF[
UR 5 R 5 UR 5 R 5 :H 5 n
UR U
5 M U$ s snf ! [ a / n Nf = f! [ a n [ S U S U S
U 35 Sn
Sn A NSSn A ff = f)u Reward function that checks if the completion matches the ground truth.
- If both gold and prediction are parseable → use math verification.
- If not parseable → compare as normalized text.
r r" first_match)extraction_modeFTall)nitsmalformed_operators basic_latexboxedunits)normalization_configboxed_match_prioritytry_extract_without_anchor)extraction_configr; zverify failed: z
, answer: z, gold: N) zipr Exceptionlenr r floatr printstriplowerappend) completionsr8 kwargsrewards
completioncontentsr" sol gold_parsed
answer_parsedrewardes r( accuracy_rewardrX s;
?JK{qM),{K 3LG
! #C G ; 1$"$) 15H).8=04*/*. 6" 67;@
+ )6 %M" #6+#E FF w}} 4 4 6#))+:K:K:M MN NN6 "E 4H K L
!
!0 ! "OA3j #OP !F"s. C
C 4C. C+*C+.
D8DD)modelargs reward_funcs
train_dataset eval_dataset peft_config) dataset_name):__doc__ostorchdatasetsr latex2sympy2_extendedr math_verifyr r r trlr r r
r r r
r r trl.rewardsr environ
setdefault__name__parserparse_args_and_config script_args
training_args
model_argsr getattrdictmodel_revisionr model_init_kwargsr datasettrain_test_splitr& r) mapr0 filterr5 r\
eval_strategyr] liststrrX model_name_or_pathtrainerr
save_model
output_dir push_to_hubr_ r* r( r s 8'R
! 5 < < ,
(- 8 z
[A
BF-3-I-I-K*K
!+ 0 0N BJ PUWaWgWgHhE&*** & : :'M#
2*=&8K8M
'' 5AT
''(=>
DGTG&&2&>G 3 " kk+,G; nn ./G kk.)GG$M&3&A&AT&I76?tL
+tCy +` ++
)?;#!#J/
G MMO }//0 )A)AB !w r*