hL S r SSKrSSKrSSKrSSKrSSKrSSKJr SSKJr SSK J
r
J r J r SSK
r
SSKJr SSKJrJr SS KJr SS
KJr SS KJr SS KJr SS
KJ r SSK J r J r \ " \ 5 r!\ " S S5 5 r"\ " S S5 5 r# " S S\$5 r% " S S\$5 r&\ " S S5 5 r'\ " S S5 5 r( " S S \)\*\(4 5 r+g) aw DatasetInfo record information we know about a dataset.
This includes things that we know about the dataset statically, i.e.:
- description
- canonical location
- does it have validation and tests splits
- size
- etc.
This also includes the things that can and should be computed once we've
processed the dataset as well:
- number of examples (in each split)
- etc.
N) dataclass)Path)ClassVarOptionalUnion) url_to_fs) DatasetCardDatasetCardData )config)Features) SplitDict)Version)
get_logger)asdict
unique_valuesc 2 \ rS rSr% Sr\\S' Sr\\S' Srg)SupervisedKeysData7 inputoutput N) __name__
__module__ __qualname____firstlineno__r str__annotations__r __static_attributes__r G/home/james-whalen/.local/lib/python3.13/site-packages/datasets/info.pyr r 7 s E3OFCr! r c 2 \ rS rSr% Sr\\S' Sr\\S' Srg)DownloadChecksumsEntryData= r keyvaluer N) r r r r r& r r r' r r r! r" r$ r$ = s CME3Or! r$ c \ rS rSrSrSrg) MissingCachedSizesConfigErrorC z;The expected cached sizes of the download file are missing.r Nr r r r __doc__r r r! r" r) r) C s Er! r) c \ rS rSrSrSrg)NonMatchingCachedSizesErrorG z/The prepared split doesn't have expected sizes.r Nr+ r r! r" r. r. G s 9r! r. c ` \ rS rSr% Sr\\ \S' Sr\\ \S' S r
\ S\ SS 4S j5 r S r
g)
PostProcessedInfoK Nfeaturesresources_checksumsc U R bF [ U R [ 5 ( d& [ R " U R 5 U l g g g N)r3
isinstancer
from_dictselfs r"
__post_init__ PostProcessedInfo.__post_init__P s; == $Z
x-P-P $ . .t}} =DM .Q $r! post_processed_info_dictreturnc
[ R " U 5 Vs1 s H o"R iM nnU " S0 UR 5 VVs0 s H u pEXC; d M XE_M snnD6$ s snf s snnf Nr dataclassesfieldsnameitems)clsr= f field_nameskvs r" r8 PostProcessedInfo.from_dictU sb '2'9'9#'>?'>!vv'> ?]'?'E'E'G\'Gtq1K[dad'G\]] @\ A" A'A')r3 )r r r r r3 r r
r r4 dictr; classmethodr8 r r r! r" r1 r1 K sL #'Hhx '*.$ .>
^ ^:M ^ ^r! r1 c \ rS rSr% Sr\R " \S9r\\ S' \R " \S9r
\\ S' \R " \S9r \\ S' \R " \S9r \\ S' Sr
\\ \ S ' Sr\\ \ S
' Sr\\ \ S ' Sr\\ \ S ' Sr\\ \ S
' Sr\\ \ S' Sr\\\\4 \ S' Sr\\ \ S' Sr \\ \ S' Sr \\ \ S' Sr \\ \ S' Sr \\ \ S' Sr!\\ \ S' / SQr"\#\$\ \ S' S r%S*S\\ 4S jjr&S+S jr'S r(\)S \$S 4S j5 r*\)S,S \S\\ S S 4S! jj5 r+\)S"\S S 4S# j5 r,S-S.S$ jjr-S/S% jr.S \4S& jr/\)S'\S S 4S( j5 r0S)r1g)0 DatasetInfo[ a Information about a dataset.
`DatasetInfo` documents datasets, including its name, version, and features.
See the constructor arguments and properties for a full list.
Not all fields are known on construction and may be updated later.
Attributes:
description (`str`):
A description of the dataset.
citation (`str`):
A BibTeX citation of the dataset.
homepage (`str`):
A URL to the official homepage for the dataset.
license (`str`):
The dataset's license. It can be the name of the license or a paragraph containing the terms of the license.
features ([`Features`], *optional*):
The features used to specify the dataset's column types.
post_processed (`PostProcessedInfo`, *optional*):
Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index.
supervised_keys (`SupervisedKeysData`, *optional*):
Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS).
builder_name (`str`, *optional*):
The name of the `GeneratorBasedBuilder` subclass used to create the dataset. It is also the snake_case version of the dataset builder class name.
config_name (`str`, *optional*):
The name of the configuration derived from [`BuilderConfig`].
version (`str` or [`Version`], *optional*):
The version of the dataset.
splits (`dict`, *optional*):
The mapping between split name and metadata.
download_checksums (`dict`, *optional*):
The mapping between the URL to download the dataset's checksums and corresponding metadata.
download_size (`int`, *optional*):
The size of the files to download to generate the dataset, in bytes.
post_processing_size (`int`, *optional*):
Size of the dataset in bytes after post-processing, if any.
dataset_size (`int`, *optional*):
The combined size in bytes of the Arrow tables for all splits.
size_in_bytes (`int`, *optional*):
The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files).
**config_kwargs (additional keyword arguments):
Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`].
)default_factory descriptioncitationhomepagelicenseNr3 post_processedsupervised_keys builder_name dataset_name config_nameversionsplitsdownload_checksums
download_sizepost_processing_size dataset_size
size_in_bytes)r[ r_ ra r3 r] _INCLUDED_INFO_IN_YAMLc U R bD [ U R [ 5 ( d% [ R " U R 5 U l U R bC [ U R [
5 ( d$ [
R U R 5 U l U R b~ [ U R [ 5 ( d_ [ U R [ 5 ( a [ U R 5 U l O%[ R " U R 5 U l U R bD [ U R [ 5 ( d% [ R " U R 5 U l U R bx [ U R [ 5 ( dX [ U R [ [ 45 ( a [ U R 6 U l g [ S0 U R D6U l g g g r@ )r3 r7 r
r8 rW r1 r\ r r r] r from_split_dictrX r tuplelistr9 s r" r; DatasetInfo.__post_init__ sS == $Z
x-P-P $ . .t}} =DM *:d>Q>QSd3e3e"3"="=d>Q>Q"RD << #Jt||W,M,M$,,,, &t|| 4 & 0 0 > ;; ":dkk9+M+M#33DKK@DK +Jt?S?SUg4h4h$..
>>'94;O;O'P$'9'QD
Example:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
>>> ds.info.write_to_directory("/path/to/directory/")
```
wb) pretty_printN)
r open posixpathjoinr DATASET_INFO_FILENAME
_dump_inforV LICENSE_FILENAME
_dump_license)r: dataset_info_dirrl ri fs_rG s r" write_to_directory DatasetInfo.write_to_directory s , +G0E2G
WWY^^$4f6R6RSUY
Z^_ OOAO 9 [ << (8&:Q:QRTXY]^""1% ZY [
Z ZYs C*C
C
C%c UR [ R " [ U 5 U( a SOSS9R S5 5 g)zQDump info in `file` file-like object open in bytes mode (to support remote files) Nindentutf-8)writejsondumpsr encode)r: filerl s r" rq DatasetInfo._dump_info s-
4::fTl 1$OVVW^_`r! c X UR U R R S5 5 g)zTDump license in `file` file-like object open in bytes mode (to support remote files)r} N)r~ rV r )r: r s r" rs DatasetInfo._dump_license s
4<<&&w/0r!
dataset_infosc J ^ T Vs/ s H o"c M UR 5 PM snm[ T5 S: a [ U4S jT 5 5 ( a TS $ SR [ S T 5 5 5 R 5 nSR [ S T 5 5 5 R 5 nSR [ S T 5 5 5 R 5 nSR [ S T 5 5 5 R 5 nS nS nU " UUUUUUS9$ s snf ) Nr c 3 4 ># U H
nTS U:H v M g7f)r Nr ).0 dset_infor s r" )DatasetInfo.from_merge.. s )gYfI-*:i*GYfs z
c 3 8 # U H oR v M g 7fr6 )rS r infos r" r r s /[]T0@0@] c 3 8 # U H oR v M g 7fr6 )rT r s r" r r ,U}t]]}r c 3 8 # U H oR v M g 7fr6 )rU r s r" r r r r c 3 8 # U H oR v M g 7fr6 )rV r s r" r r s +S]TLL]r )rS rT rU rV r3 rX )copylenallro r strip) rF r r rS rT rU rV r3 rX s ` r"
from_mergeDatasetInfo.from_merge s ;Hb=i) )=b
} !c)gYf)g&g&g # # kk-/[]/["[\bbd ;;},U},U UV\\^;;},U},U UV\\^++m+S]+S STZZ\ # +
cs
D D rt r> c l [ U40 U=( d 0 D6tp4[ R SU 35 U( d [ S5 eUR [
R " U[ R 5 SSS9 n[ R " U5 nSSS5 U R W5 $ ! , ( d f N = f)a Create [`DatasetInfo`] from the JSON file in `dataset_info_dir`.
This function updates all the dynamically generated fields (num_examples,
hash, time of creation,...) of the [`DatasetInfo`].
This will overwrite all previous metadata.
Args:
dataset_info_dir (`str`):
The directory containing the metadata file. This
should be the root directory of a specific dataset version.
storage_options (`dict`, *optional*):
Key/value pairs to be passed on to the file-system backend, if any.
Example:
```py
>>> from datasets import DatasetInfo
>>> ds_info = DatasetInfo.from_directory("/path/to/directory/")
```
zLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rr} encodingN) r loggerdebug
ValueErrorrm rn ro r rp r loadr8 )rF rt ri ru rv rG dataset_info_dicts r" from_directoryDatasetInfo.from_directory s 4 +G0E2G 12B1CDE de e
WWY^^$4f6R6RSUXcjW
kop $ ! l}} .// l
ks 5B%%
B3r c
[ R " U 5 Vs1 s H o"R iM nnU " S0 UR 5 VVs0 s H u pEXC; d M XE_M snnD6$ s snf s snnf r@ rA )rF r rG rH rI rJ s r" r8 DatasetInfo.from_dict sb '2'9'9#'>?'>!vv'> ?V'8'>'>'@U'@tqADTdad'@UVV @UrL c U R nUR " S0 UR R 5 VVs0 s H( u pEUc U( a M U[ R " U5 _M* snnD6 g s snnf r@ )__dict__updaterE r deepcopy)r: other_dataset_info ignore_none self_dictrI rJ s r" r DatasetInfo.update sh MM
/ 7 7 = = ? ?DAM $4==## ?
s A. A.c U R " S0 U R R 5 VVs0 s H u pU[ R " U5 _M snnD6$ s snnf r@ ) __class__r rE r r )r: rI rJ s r" r DatasetInfo.copy( sE ~~X ATATAV WAVDMM!$4!4AV WXX Ws #Ac 0 n[ U 5 nU Hm nX0R ; d M [ X5 n[ US5 ( a UR 5 X' MD [ US5 ( a UR 5 X' Mi XAU' Mo U$ )N
_to_yaml_list_to_yaml_string)r rc getattrhasattrr r )r: yaml_dictr r& r' s r"
_to_yaml_dictDatasetInfo._to_yaml_dict+ s{ "4L$C111 *5/22%*%8%8%:IN U$566%*%:%:%?'>!vv'> ?Ny'8M'8tqA rP )2r r r r r, rB fieldr rS r rT rU rV r3 r r
rW r1 rX r rY rZ r[ r\ r r r] rM r^ r_ intr` ra rb rc r rg r; rw rq rs rN r r r8 r r r r r r r! r" rP rP [ s *Z #((=K= %%c:Hc: %%c:Hc: $$S9GS9#'Hhx '26NH ./648OX 018 #'L(3-&"&L(3-&!%K# %-1GXeCL)
*1 !FHTN!)--#'M8C='*.(3-."&L(3-&#'M8C='3HT#Y / R&&X`aeXf &:a1
tM':
. 0c 0HTN 0^k 0 0B W$ W= W W
Y t O O O Or! rP c d \ rS rSrS
S S jjr\S S j5 r\S\SS 4S j5 rS\SS4S jr S r
g)
DatasetInfosDictiD r> Nc 0 n[ R R U[ R 5 n[ R R U[ R
5 nU( d U R
U5 nUR U 5 [ R R U5 ( a` [ USSS9 nUR 5 VV s0 s H u pU[ U 5 _M n
nn [ R " XU( a SOS S9 S S S 5 [ R R U5 ( a# [ R " U5 n U R n O S n [# 5 n U( aK UR% U 5 U c [ S[' U 5 - S- 5 OU n U R) [+ U5 5 g g s sn nf ! , ( d f N= f)Nwr} r rz r{ z---
z
---
)ospathro r DATASETDICT_INFOS_FILENAMEREPOCARD_FILENAMEr r existsrm rE r r dumpr r datar
to_dataset_card_datar saver )
r: dataset_infos_dir overwriterl total_dataset_infosdataset_infos_pathdataset_readme_pathrG r[ r dataset_infos_dict dataset_carddataset_card_datas
r" rw #DatasetInfosDict.write_to_directoryE sv WW\\*;V=^=^ _ ggll+V>V W"&"5"56G"H ""4(
77>>, - -(#@AQdQjQjQl&Ql7M{K !22Ql # & ,\tT A 77>>- . .&++,?@L , 1 1 L / 1 4 45F GMYMa Gc*;&< >"'',,'8&:R:RS T T + 0 06G1H6KcKc1c d i i !22112CDD
77>>"'',,'8&:[:[\ ] ]bggll#4f6W6WXcjkop ?Ciil>P>P>R>R :K $[%:%:;L%MM>R lk 5L
lks *F**"F$ F*$F**
F8r c [ UR S5 [ [ 45 ( a [ US [ 5 ( a@ U " US Vs0 s H) nUR SS5 [ R U5 _M+ sn5 $ [ R US 5 nUS R SS5 Ul U " UR U05 $ U " 5 $ s snf )Nr r[ default)r7 r rg rM rP r r[ )rF r dataset_info_yaml_dictr s r" r 'DatasetInfosDict.from_dataset_card_datau s '++N;dD\ J J+N;TBB
7H6W 7X 2 /22=)LkNiNi 2O 7X + : :;L^;\ ] +<^+L+P+PQ^`i+j (L 4 4lCDD5Ls 0Cc U ( Gax SU; a3 [ US [ 5 ( a US R SS5 US 0nO:SU; a2 [ US [ 5 ( a US Vs0 s H nUS U_M nnO0 n0 UEU R 5 VVs0 s H u pEXER 5 _M snnEnUR 5 H u pGXGS' M [
U5 S:X aN [ [ UR 5 5 5 US' US R SS 5 nUS:w a
SU0US EUS' g g / US' [ UR 5 5 H1 u pHUR SS 5 SU0UEnUS R U5 M3 g g s snf s snnf )Nr r[ r r )
r7 rM r rg rE r r nextitervaluespopsortedappend) r: r dataset_metadata_infosconfig_metadatar[ r r dset_info_yaml_dictr s r" r %DatasetInfosDict.to_dataset_card_data s !22zBSTbBcei7j7j%n599-SUfguUv*& #44DUVdDegk9l9l ,=^+L*+L $M2OC+L ' *&
*,&#(#X\XbXbXdeXd>Tk; 7 7 99Xde#
5H4M4M4O0 5@M2 5P&'1,48>Q>X>X>Z9[4\!.1 / ? C CMSW X )+ &{9+N;9%n5 , 57!.1;ABUB[B[B];^7K*..}dC.;[-cLb-c*%n5<<=ST <_C * fs 'F Fr )FF)r> N)r> r ) r r r r rw rN r r
r r r r r! r" r r D sT 98 & K] $&Uo &U$ &Ur! r ),r, r rB r r rn r pathlibr typingr r r fsspec fsspec.corer huggingface_hubr r
r r r3 r
r] r utilsr
utils.loggingr utils.py_utilsr r r r r r$ Exceptionr) r. r1 rP rM r r r r! r" r s
! , ,
! 8 % 1
H
FI F:) : ^ ^ ^ eO eO eOPjUtC ,- jUr!