hv} JSrSSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSK J r SSK r SSK rSSKJrJr SSKJrJr SSKJr SSKJrJrJrJrJrJrJrJrJ r J!r! SS K"J#r#J$r$J%r%J&r&J'r' SS K(J)r) SS K*J+r+ SS K,J-r-J.r. \ S - SSSSSSSS4 Sjr/"SS5r0"SS5r1g)a Benchmark a YOLO model formats for speed and accuracy. Usage: from ultralytics.utils.benchmarks import ProfileModels, benchmark ProfileModels(['yolo11n.yaml', 'yolov8s.yaml']).run() benchmark(model='yolo11n.pt', imgsz=160) Format | `format=argument` | Model --- | --- | --- PyTorch | - | yolo11n.pt TorchScript | `torchscript` | yolo11n.torchscript ONNX | `onnx` | yolo11n.onnx OpenVINO | `openvino` | yolo11n_openvino_model/ TensorRT | `engine` | yolo11n.engine CoreML | `coreml` | yolo11n.mlpackage TensorFlow SavedModel | `saved_model` | yolo11n_saved_model/ TensorFlow GraphDef | `pb` | yolo11n.pb TensorFlow Lite | `tflite` | yolo11n.tflite TensorFlow Edge TPU | `edgetpu` | yolo11n_edgetpu.tflite TensorFlow.js | `tfjs` | yolo11n_web_model/ PaddlePaddle | `paddle` | yolo11n_paddle_model/ MNN | `mnn` | yolo11n.mnn NCNN | `ncnn` | yolo11n_ncnn_model/ IMX | `imx` | yolo11n_imx_model/ RKNN | `rknn` | yolo11n_rknn_model/ ) annotationsN)Path)YOLO YOLOWorld) TASK2DATA TASK2METRIC)export_formats) ARM64ASSETS ASSETS_URL IS_JETSONLINUXLOGGERMACOSTQDM WEIGHTS_DIRYAML)IS_PYTHON_3_13 check_imgszcheck_requirements check_yolo is_rockchip) safe_download) file_size) get_cpu_info select_devicez yolo11n.ptFcpuMbP?c F^%[U5n[U[5(aUSUS:XdS5eSSKn U RR S5 U RR S5 U RRS5 U RRS5 U RRS5 U RRS5 [USS 9n[U[[45(a [U5n[UR R SS S5n U=(d ["UR$n[&UR$n /n [(R("5nUR+5nU(a*[-[/5S 5nUU;dS US US35e[1[/5R356GHunnnnnnSunnU(aX:waMUS:XaUR$S:wdS5eORUS:Xa[4(a [6(aS5eO.US;a([8(d[4(a [6(aS5eUS:Xa[:(aS5eUS;a[U[<5(aS5eUS:Xai[U[<5(aS5eUR$S:wdS5eU (aS5e[4(a [>(a[8(dS5eUS :Xa[U[<5(aS!5eUS":Xa[U[<5(aS#5eUS$:XaWU (ae[U[<5(aS%5eUR$S&:XdS'5eS(URA5;dS)5eUS*:XaR[U[<5(aS+5eU (aS,5e[4(dS-5e[C5(aS.5eS/URD;aU(dS05eS1URD;aU(dS25eUS3:Xa5URF=(d URH=(d URJnUnOAURL"ScX(X4XSS4.U D6n[UUR$S59nU[U5;dS65eS7nUR$S8:wd US:wdS95eUS:;dS;5eUS:wd [NRP"5S<:XdS=5eUS":XaU (aS>5eURS[TS?- X%USS@9 URWUSUSUUUSSASB9 nURXU URZSCnn[]SDUU-- SE5nU R_USF[][aU5S5[]USG5[]USE5U/5 GM [kUSJ9 U RmU SKSLSMU SNSO/SPSQ9nURoSRSSS9nURqU Rs5RuU Rv5RyS355nURJn[(R("5U- n STn!SUUSVUSWUSXU SYSZU!S[US[3 n"[fRz"U"5 [}S\S]S^S_S`9n#U#RU"5 SSS5 U(aN[U[5(a9UU R5n$Um%[sU%4SajU$55(d SbT%35eU$![baxnU(a[EU5[dLd SHUSIU35e[fRh"SHUSIU35 U R_UU[][aU5S5SSS/5 SnAGMSnAff=f!,(df  N=f)daH Benchmark a YOLO model across different formats for speed and accuracy. Args: model (str | Path): Path to the model file or directory. data (str | None): Dataset to evaluate on, inherited from TASK2DATA if not passed. imgsz (int): Image size for the benchmark. half (bool): Use half-precision for the model if True. int8 (bool): Use int8-precision for the model if True. device (str): Device to run the benchmark on, either 'cpu' or 'cuda'. verbose (bool | float): If True or a float, assert benchmarks pass with given metric. eps (float): Epsilon value for divide by zero prevention. format (str): Export format for benchmarking. If not supplied all formats are benchmarked. **kwargs (Any): Additional keyword arguments for exporter. Returns: (polars.DataFrame): A polars DataFrame with benchmark results for each format, including file size, metric, and inference time. Examples: Benchmark a YOLO model with default settings: >>> from ultralytics.utils.benchmarks import benchmark >>> benchmark(model="yolo11n.pt", imgsz=640) rTz'benchmark() only supports square imgsz.NASCII_BORDERS_ONLY_CONDENSEDF)verboseend2endArgumentzExpected format to be one of z , but got 'z'.)❌Npbobbz.TensorFlow GraphDef not supported for OBB taskedgetpuz3Edge TPU export only supported on non-aarch64 Linux>tfjscoremlzECoreML and TF.js export only supported on macOS and non-aarch64 Linuxr-z#CoreML not supported on Python 3.13>r)r,tfliter+ saved_modelz;YOLOWorldv2 TensorFlow exports not supported by onnx2tf yetpaddlez,YOLOWorldv2 Paddle exports not supported yetzBPaddle OBB bug https://github.com/PaddlePaddle/Paddle/issues/72024z3End-to-end models not supported by PaddlePaddle yetz3Windows and Jetson Paddle exports not supported yetmnnz)YOLOWorldv2 MNN exports not supported yetncnnz*YOLOWorldv2 NCNN exports not supported yetimxz%YOLOWorldv2 IMX exports not supporteddetectz%IMX only supported for detection taskC2fz*IMX only supported for YOLOv8n and YOLO11nrknnz*YOLOWorldv2 RKNN exports not supported yetz+End-to-end models not supported by RKNN yetzRKNN only supported on Linuxz1RKNN Inference only supported on Rockchip devicesrzinference not supported on CPUcudazinference not supported on GPU-)imgszformathalfint8datadevicer%)taskz export failedu❎posez(GraphDef Pose inference is not supported>r,r+zinference not supportedDarwinz(inference only supported on macOS>=10.13zHEnd-to-end torch.topk operation is not supported for NCNN prediction yetzbus.jpg)r9r>r;r%r) r=batchr9plotsr>r;r<r%conf inferenceu✅zBenchmark failure for : )r>Formatu Status❔z Size (MB)zInference time (ms/im)FPSrow)schemaorient )offsetudBenchmarks legend: - ✅ Success - ❎ Export passed but validation failed - ❌️ Export failedz Benchmarks complete for z on z at imgsz=z (z.2fzs)  zbenchmarks.logaignoreutf-8)errorsencodingc3h># UH'n[R"U5(aM UT:v M) g7fN)npisnan).0xfloors V/home/james-whalen/.local/lib/python3.13/site-packages/ultralytics/utils/benchmarks.py benchmark..s!AgRXXa[91u9gs2 2z%Benchmark failure: metric(s) < floor )Br isinstancelistpolarsConfig set_tbl_cols set_tbl_rowsset_tbl_width_charsset_tbl_hide_column_data_typesset_tbl_hide_dataframe_shapeset_tbl_formattingrstrrrgetattrmodelrr?rtimelower frozensetr zipvaluesrr rrrr __str__rtypept_path ckpt_path model_nameexportplatformsystempredictr val results_dictspeedroundappendr ExceptionAssertionErrorrerrorr DataFramewith_row_index with_columnsallcastString fill_nullinfoopenwritefloatto_numpy)&rnr=r9r;r<r>r%epsr:kwargspl is_end2endkeyyt0 format_argformatsnamesuffixrgpu_emojifilenameexported_modelresultsmetricrfpsedf df_displaydtlegendsfmetricsr]s& @r^ benchmarkr5sDH  E#-eT#:#:58uQx oFoo DII2II2II!!"%II,,T2II**40II  !?@ 65 1F%#t%%U **2. 5AJ  (9UZZ(D ejj !C A BJN,Z89 d$A'+V`Uaac"dd -0.2B2I2I2K-L)ffc3%xQ Uj2~zzU*\,\\*9$uUUa,aa*U--uuu[5!)>P+PP)KK%eY77v9vv7!%eY77g9gg7zzU*p,pp*%\'\\~iiEEp;pp9%eY77d9dd7%eY77e9ee7%%~%eY77`9``7zzX-V/VV- /]1]]/%eY77e9ee7%T'TT~u<< e;e e9!44 O6O O4X%):h)F rHr rF%q'qq~  " "6I#5UX\fk " l%(() G$005w}}[7QEF ,a0C HHdE58) HHdE58)>> benchmark = RF100Benchmark() >>> benchmark.set_key("your_roboflow_api_key") roboflowr)Roboflow)api_keyN)rrrr)rrrs r^set_keyRF100Benchmark.set_keys :&%7+rc[RRS5(a,[R"S5[R "S54O[R "S5 [R "S5 [R "S5 [[S35 [USS9nUGHn[R"SUR55upEpgnURRU5 USU3n [U 5R5(dIUR R#U5R%U5R'U5R)S5 O[*R,"S 5 UR.R[R0"5U - S - 5 GM S S S 5 URUR.4$![2a GM4f=f!,(df  N8=f) a Parse dataset links and download datasets. Args: ds_link_txt (str): Path to the file containing dataset links. Returns: ds_names (list[str]): List of dataset names. ds_cfg_list (list[Path]): List of paths to dataset configuration files. Examples: >>> benchmark = RF100Benchmark() >>> benchmark.set_key("api_key") >>> benchmark.parse_dataset("datasets_links.txt") zrf-100zultralytics-benchmarksz/datasets_links.txtrTrVz/+r8yolov8zDataset already downloaded.z data.yamlN)ospathexistsshutilrmtreemkdirchdirrr rresplitstriprrrr workspaceprojectversiondownloadrrrcwdr) r ds_link_txtfilelinerurlrrr proj_versions r^ parse_datasetRF100Benchmark.parse_dataset su :<9Q9Qx "((8"45WYW_W_`hWi  )* $789 + 0D :<((4:V7AIMM((1&-Yay#9L -4466)))4<>> benchmark = RF100Benchmark() >>> benchmark.evaluate("path/to/data.yaml", "path/to/val_log.txt", "path/to/eval_log.txt", 0) )u🚀u⚠️u💡r(namesrTrc3,># UH oT;v M g7frXra)r[symbolrs r^r_*RF100Benchmark.evaluate..NsAL&~LsrOc US:g$)Nr ra)r}s r^)RF100Benchmark.evaluate..Qs#)rrQc 3># UHBnUT;dUS:XdMST;dMST;dM!TSTSTSTSTSTSTS S .v MD g 7f) rz(AP)z(AR)rr"rGrHrNra)r[r class_namesentriess r^r_rSs "%K'AJ8W~7;2 1<; 7 7s0AF F$#F*F %FF F F*)rrrrN)rrl)zdatasets_links.txt)rrl)rr)rrlrrlrrlrint) __name__ __module__ __qualname____firstlineno____doc__rrr staticmethodrr__static_attributes__rarr^rrs1$c,"#/J##7rrc\rSrSrSrSSSjjrSrSr\SSj5r \SSSjj5r SSS jjr \S 5r SSS jjr SS jr\SS j5r\SSj5rSrg) ProfileModelsipa ProfileModels class for profiling different models on ONNX and TensorRT. This class profiles the performance of different models, returning results such as model speed and FLOPs. Attributes: paths (list[str]): Paths of the models to profile. num_timed_runs (int): Number of timed runs for the profiling. num_warmup_runs (int): Number of warmup runs before profiling. min_time (float): Minimum number of seconds to profile for. imgsz (int): Image size used in the models. half (bool): Flag to indicate whether to use FP16 half-precision for TensorRT profiling. trt (bool): Flag to indicate whether to profile using TensorRT. device (torch.device): Device used for profiling. Methods: run: Profile YOLO models for speed and accuracy across various formats. get_files: Get all relevant model files. get_onnx_model_info: Extract metadata from an ONNX model. iterative_sigma_clipping: Apply sigma clipping to remove outliers. profile_tensorrt_model: Profile a TensorRT model. profile_onnx_model: Profile an ONNX model. generate_table_row: Generate a table row with model metrics. generate_results_dict: Generate a dictionary of profiling results. print_table: Print a formatted table of results. Examples: Profile models and print results >>> from ultralytics.utils.benchmarks import ProfileModels >>> profiler = ProfileModels(["yolo11n.yaml", "yolov8s.yaml"], imgsz=640) >>> profiler.run() Nc XlX lX0lX@lXPlX`lXpl[U[R5(aXl g[U5Ul g)a Initialize the ProfileModels class for profiling models. Args: paths (list[str]): List of paths of the models to be profiled. num_timed_runs (int): Number of timed runs for the profiling. num_warmup_runs (int): Number of warmup runs before the actual profiling starts. min_time (float): Minimum time in seconds for profiling a model. imgsz (int): Size of the image used during profiling. half (bool): Flag to indicate whether to use FP16 half-precision for TensorRT profiling. trt (bool): Flag to indicate whether to profile using TensorRT. device (torch.device | str | None): Device used for profiling. If None, it is determined automatically. Notes: FP16 'half' argument option removed for ONNX as slower on CPU than FP32. Examples: Initialize and profile models >>> from ultralytics.utils.benchmarks import ProfileModels >>> profiler = ProfileModels(["yolo11n.yaml", "yolov8s.yaml"], imgsz=640) >>> profiler.run() N) pathsnum_timed_runsnum_warmup_runsmin_timer9r;trtrbtorchr>r) rrrrrr9r;rr>s r^rProfileModels.__init__sKB ,.    *65<< @ @f mTZF[ rc UR5nU(d[R"S5 /$/n/nUGHnURS5nURS;a[ [ U55nUR5 UR5nUR(a`URRS:waFUR5(d1URSURURURSS9nURSURURSS 9nO'URS :XaUR!U5nUnOGMUR#[ U55n UR%[ U55n UR'UR)UR*XU55 UR'UR-UR*XU55 GM UR/U5 U$) a Profile YOLO models for speed and accuracy across various formats including ONNX and TensorRT. Returns: (list[dict]): List of dictionaries containing profiling results for each model. Examples: Profile models and print results >>> from ultralytics.utils.benchmarks import ProfileModels >>> profiler = ProfileModels(["yolo11n.yaml", "yolov8s.yaml"]) >>> results = profiler.run() z'No matching *.pt or *.onnx files found.z.engine.pt.yml.yamlrengineF)r:r;r9r>r%onnx)r:r9r>r%z.onnx) get_filesrwarning with_suffixrrrlfuserrr>ruis_fileryr;r9get_onnx_model_infoprofile_tensorrt_modelprofile_onnx_modelrgenerate_table_rowstemgenerate_results_dict print_table) rfiles table_rowsoutputr engine_filern model_info onnx_filet_enginet_onnxs r^runProfileModels.runs  NND EI D**95K{{66SY "ZZ\ 88 0 0E 9+BUBUBWBW"',,'!YY"jj#{{ % #/#K"LL!**;;! ) '!55d;  223{3CDH,,S^ $ rc h/nURHn[U5nUR5(aR/SQnURUVVs/sH-n[R"[ X$- 55HoUPM M/ snn5 MuUR S;aUR[ U55 MUR[R"[ U555 M [R"S[U535 [U5Vs/sHn[U5PM sn$s snnfs snf)z Return a list of paths for all relevant model files given by the user. Returns: (list[Path]): List of Path objects for the model files. )z*.ptz*.onnxz*.yamlrz Profiling: ) rris_dirrglobrlrrrrsorted)rr,r extensionsextrs r^r ProfileModels.get_filessJJD:D{{}}9  j`jsTYYsSWS]E_TdE_dj`a 88 SY' TYYs4y12  k&-12'-e}5}tT }55a6s 4D) D/cg)zWExtract metadata from an ONNX model file including parameters, GFLOPs, and input shape.)rrrrra)r1s r^r%!ProfileModels.get_onnx_model_infos"rc[R"U5n[U5Han[R"U5[R"U5pTXXAU-- :XX--:-n[ U5[ U5:Xa U$UnMc U$)aD Apply iterative sigma clipping to data to remove outliers. Args: data (np.ndarray): Input data array. sigma (float): Number of standard deviations to use for clipping. max_iters (int): Maximum number of iterations for the clipping process. Returns: (np.ndarray): Clipped data array with outliers removed. )rYarrayrangemeanstdr)r=sigma max_itersrrBrC clipped_datas r^iterative_sigma_clipping&ProfileModels.iterative_sigma_clipping sxx~y!A rvvd|#s{(:!:tU[FX?X YZL< CI-  D "  rcUR(a[U5R5(dg[U5n[R "UR UR S4[RS9nSn[S5H]n[R"5n[UR5HnU"X@R SS9 M [R"5U- nM_ [[URXR-- UR-5URS-5n/n [[U5US9H5nU"X@R SS9n U R!U S R"S 5 M7 UR%[R&"U 5S SS 9n [R("U 5[R*"U 54$) av Profile YOLO model performance with TensorRT, measuring average run time and standard deviation. Args: engine_file (str): Path to the TensorRT engine file. eps (float): Small epsilon value to prevent division by zero. Returns: mean_time (float): Mean inference time in milliseconds. std_time (float): Standard deviation of inference time in milliseconds. )rrr)dtyperF)r9r%2descrrErGrDrE)rrr$rrYzerosr9uint8rArormaxrrrrrrrGr@rBrC) rr/rrn input_dataelapsedr start_timenum_runs run_timesrs r^r&$ProfileModels.profile_tensorrt_model#sbxxtK088::[!XXtzz4::q9J qAJ4//0j EB1iikJ.G uT]]gm.Ms#QLSz#s+8q8Ls)+)r) tensor_shapes r^ check_dynamicProfileModels.check_dynamicJsQLQQQQrcR[S/5 SSKnUR5nURRUlSUlURXS/S9n[5nUR5GHnURnURUR5(a[UR5S:waHURURSS5(a%[S URS UR35e[UR5S:XaSS UR UR 4OS/URSSQ7n O URn S U;a["R$n OjS U;a["R&n OSSU;a["R(n O?@!))+ 030J0J0Y0Y -,- )##IH^G_#`& OO-L%**J!!,"4"455|))*a/D4F4F|GYGYZ[Z\G]4^4^$'A,BTBTAUUYZfZkZkYl%mnn69,:L:L6MQR6RQ4::tzz2YZXt]i]o]opqpr]sXt+00 J& jj J& jj Z' jj J& hh J& hh  #=j\!JKK5<<[IJ%**J  " "J#; <7.:&&(+00 qAJ4//0+81iikJ.G uT]]gm ?7 11"((92EQZ[1\ wwy!266)#444rcUupVpxSUSSURSUSSSUSSS USSSUSSS US - SSUSS 3$) a Generate a table row string with model performance metrics. Args: model_name (str): Name of the model. t_onnx (tuple): ONNX model inference time statistics (mean, std). t_engine (tuple): TensorRT engine inference time statistics (mean, std). model_info (tuple): Model information (layers, params, gradients, flops). Returns: (str): Formatted table row string with model metrics. z| 18sz | z | - | rz.1f±r"z ms | g.Az |)r9) rrxr3r2r0layersparams gradientsflopss r^r( ProfileModels.generate_table_rows~&,6( C DJJ(pixels)zmAPval
50-95zSpeed
CPU (z) ONNX
(ms)zSpeed
z TensorRT
(ms)zparams
(M)zFLOPs
(B)|c3.# UH nSUS3v M g7f)rONrar[hs r^r_,ProfileModels.print_table..s:'Q!A3a'sc3D# UHnS[U5S--v M g7f)r8rGN)rrs r^r_rs"Gw!3#a&1*#5ws z N)rr7 is_availableget_device_namerjoinrr)r-rheadersheader separatorrLs r^r+ProfileModels.print_tables05zz/F/F/H/Hejj((+e  # "!,.!1 @SE!2 3  sxx:':::S@#(("Gw"GGG#M  d6(O$ IC KK r)r>r;r9rrrrr)d <iTTN)r list[str]rrrrrrr9rr;boolrrr>ztorch.device | str | None)r1rl)rGr)r=z np.ndarrayrDrrEr)r)r/rlrr)r1rlrr)rxrlr3tuple[float, float]r2rr0z!tuple[float, float, float, float])r-r)rr r r r rr4r r r%rGr&r]r'r(r*r+rrarr^rrpsIH"!,0(\(\(\ (\  (\  (\(\(\*(\T5n6*""*%5NRRH5T  $ &  6  2  # & 6   6rr)2r  __future__rr8rrzrrropathlibrnumpyrY torch.cudar ultralyticsrrultralytics.cfgrrultralytics.engine.exporterr ultralytics.utilsr r r r rrrrrrultralytics.utils.checksrrrrrultralytics.utils.downloadsrultralytics.utils.filesrultralytics.utils.torch_utilsrrrrrrarr^rs8#  '26qqqmm5-E  $     fROOdvvr