h7# S SK r S SKJr S SKJrJrJr S SKrS SK J
r S SK J
r S SKrS SKJr \R$ R& R) \5 r\ " S S\R. 5 5 r " S S\R2 5 rg) N) dataclass)LiteralOptionalUnion)
table_castc ^ \ rS rSr% SrSr\\ \S' Sr \\
\ \S' Sr \\
R \S' Sr\\\R$ \
\ \
\
\ 4 \S' Sr\\R* \S' S r\S
\S ' U 4S jrS
rU =r$ )
ParquetConfig a
BuilderConfig for Parquet.
Args:
batch_size (`int`, *optional*):
Size of the RecordBatches to iterate on.
The default is the row group size (defined by the first row group).
columns (`list[str]`, *optional*)
List of columns to load, the other ones are ignored.
All columns are loaded by default.
features: (`Features`, *optional*):
Cast the data to `features`.
filters (`Union[pyarrow.dataset.Expression, list[tuple], list[list[tuple]]]`, *optional*):
Return only the rows matching the filter.
If possible the predicate will be pushed down to exploit the partition information
or internal metadata found in the data source, e.g. Parquet statistics.
Otherwise filters the loaded RecordBatches before yielding them.
fragment_scan_options (`pyarrow.dataset.ParquetFragmentScanOptions`, *optional*)
Scan-specific options for Parquet fragments.
This is especially useful to configure buffering and caching.
on_bad_files (`Literal["error", "warn", "skip"]`, *optional*, defaults to "error")
Specify what to do upon encountering a bad file (a file that can't be read). Allowed values are :
* 'error', raise an Exception when a bad file is encountered.
* 'warn', raise a warning when a bad file is encountered and skip that file.
* 'skip', skip bad files without raising or warning when they are encountered.
Example:
Load a subset of columns:
```python
>>> ds = load_dataset(parquet_dataset_id, columns=["col_0", "col_1"])
```
Stream data and efficiently filter data, possibly skipping entire files or row groups:
```python
>>> filters = [("col_0", "==", 0)]
>>> ds = load_dataset(parquet_dataset_id, streaming=True, filters=filters)
```
Increase the minimum request size when streaming from 32MiB (default) to 128MiB and enable prefetching:
```python
>>> import pyarrow
>>> import pyarrow.dataset
>>> fragment_scan_options = pyarrow.dataset.ParquetFragmentScanOptions(
... cache_options=pyarrow.CacheOptions(
... prefetch_limit=1,
... range_size_limit=128 << 20
... ),
... )
>>> ds = load_dataset(parquet_dataset_id, streaming=True, fragment_scan_options=fragment_scan_options)
```
N
batch_sizecolumnsfeaturesfiltersfragment_scan_optionserror)r warnskip on_bad_filesc " > [ TU ] 5 g N)super
__post_init__)self __class__s c/home/james-whalen/.local/lib/python3.13/site-packages/datasets/packaged_modules/parquet/parquet.pyr ParquetConfig.__post_init__V s
)__name__
__module__ __qualname____firstlineno____doc__r r int__annotations__r liststrr
datasetsFeaturesr r ds
Expressiontupler ParquetFragmentScanOptionsr r r __static_attributes__
__classcell__)r s @r r r s ;z !%J
$#'GXd3i
',0Hhx(()0NRGXeBMM4;T%[8I IJ
KREI8B$A$ABI5 " USU0S95 GM6 U R R@ b [C U R R@ 5 [C U R R 5 :w aw [ R" " U R R R
5 V V
s0 s H" u pXR R@ ; d M X_M$ sn
n 5 U R l U$ s snf ! , ( d f GM = f! [* R, a nU R R. S:X a2 [0 R3 SU S[5 U5 R6 SU 35 e U R R. S :X a9 [0 R9 S
U S [5 U5 R6 SU S 35 SnAGM_ [0 R; S
U S [5 U5 R6 SU S 35 SnAGM SnAff = fs sn
n f )z-We handle string, list and dicts in datafilesz=At least one data file must be specified, but got data_files=TNrbr z Failed to read schema from '
' with error : r zSkipping bad schema from ''. `zPAt least one valid data file must be specified, all the data_files are invalid: files)name
gen_kwargs)"r3
data_filesr5 download_configextract_on_the_flydownload_and_extractitems
isinstancer&
iter_filesinfor
itertoolschain
from_iterableopenr' r( from_arrow_schemapq read_schemapa ArrowInvalidr loggerr typer warningdebugappendSplitGeneratorr r4 ) r
dl_managerrB splits
split_namer? filefecolfeats r _split_generatorsParquet._split_generatorsi s {{%% \]a]h]h]s]s\t uv v8<
""5 44T[[5K5KL
!+!1!1!3 J%%% =BCUT**40UECyy!!) %OO 9 9% @D i !$ -191B1B1T1TUWUcUcdeUf1gDII . ! . - A yy!!) fgkgrgrg}g}f~
MM(11zwX]N^_ `1 "42 ;; *s4;;3F3F/G3tyyOaOaKb/b!)!2!2,0II,>,>,D,D,Fe,Fys#Q\Q\QdQdJd,Fe"DII
3 D
. - ?? i ;;33w> "LL+Gv][_`a[b[k[kZllnopnq)r s ! ![[ 5 5 ? "NN-GvSQUVWQXQaQaPbbdefdggh+i j j "LL+EdV3tTUwO_O_N``bcdbeef)g h hi fsP (J> J&
AJJ& N
/N
J# J&#J&&N:BN2NNpa_tablereturnc U R R b) [ XR R R 5 nU$ r )rI r
r arrow_schema)r rc s r _cast_tableParquet._cast_table s3 99 ) "(II,>,>,K,KLHr c # # U R R b U R R b [ S U R R R
5 5 [ U R R 5 :w a: [
SU R R SU R R S35 e[ U R R [ 5 ( a* [ R " U R R 5 OU R R n[ R " U R R S9n[ [ R" R% U5 5 H u pE [' US5 nUR) U5 nUR* ( a U R R, =( d UR* S R. n[ UR1 UU R R USSS95 H? u p[2 R4 R7 U
/5 n U S U 3U R9 U 5 4v MA S S S 5 M g ! , ( d f M = f! [2 R: [ 4 a n U R R< S
:X a2 [> RA S U S [C U 5 RD S
U 35 e U R R< S:X a9 [> RG SU S[C U 5 RD S
U S35 S n A GM [> RI SU S[C U 5 RD S
U S35 S n A GM S n A ff = f7f)Nc 3 8 # U H oR v M g 7fr )r@ ).0fields r +Parquet._generate_tables.. s N.MUjj.Ms z)Tried to load parquet data with columns 'z ' with mismatching features '') default_fragment_scan_optionsr: r )r r filterbatch_readaheadfragment_readahead_r zFailed to read file 'r; r< r zSkipping bad file 'r= r> )%r3 r
r sortedrI rf r5 rG r r% rO filters_to_expressionr) ParquetFileFormatr enumeraterJ rK rL rM
make_fragment
row_groupsr num_rows
to_batchesrQ Table from_batchesrg rR r rS r rT r rU rV )
r r? filter_exprparquet_file_formatfile_idxr\ r] parquet_fragmentr batch_idx record_batchrc r^ s
r _generate_tablesParquet._generate_tables s ;; + 0C0C0ONdii.@.@.M.MNNRXY]YdYdYlYlRmm ? @S@S?TTqrvr{r{ sE sE rF FG H
$++--t44
$ $T[[%8%8 9$$
! 2 2QUQ\Q\QrQr s ' (E(Ee(L MNH
Z$%':'H'H'K$'22%)[[%;%;%f?O?Z?Z[\?]?f?f
7@ , 7 7+5(, (;(;'20134 8 8 3I (*xx'<'r sw ! + + %
* *8 4 F H** F F RcZh(( cZr