hlSSKJr SSKrSSKrSSKJr SSKJr SSKJ r J r J r SSK J r SSKJr SSKJr SSKJrJr SS KJr SS KJrJrJr SS KJr SS KJ r SS K!J"r" SSK#J$r$ SSK%J&r& \RN"\(5 SSK)J*r* SSK)J+r, SSS5 \ (a,SSKJ-r- SSKJ.r.J/r/J0r0 SSK1J2r2J3r3J4r4J5r5J6r6J7r7 SSK8J9r9 SSK:J;r; \"SSSS9\"SSSS9SSSSSSSSSSSSSSSS SSSSS!SS".S*S#jj55rS-S.S'jjr+\"SSSS9\"SSSS9SSSSSSSSSSSSSSSSS SS!SS!SSSSSS(.S/S)jj55r?g!,(df  GNG=f)0) annotationsN)Sequence)Path)IO TYPE_CHECKINGAny)concat)import_optional)deprecate_renamed_parameterissue_deprecation_warning)issue_unstable_warning)is_int_sequenceis_path_or_str_sequencenormalize_filepath)wrap_ldf) from_arrow)prepare_file_arg)!_init_credential_provider_builder) ScanOptions) PyLazyFrame)read_parquet_metadata)Literal) DataFrameDataType LazyFrame) ColumnMappingDefaultFieldValues DeletionFiles FileSourceParallelStrategy SchemaDict)CredentialProviderFunction)ScanCastOptionsrow_count_namerow_index_namez0.20.4versionrow_count_offsetrow_index_offsetautoTFraise)columnsn_rowsr%r)paralleluse_statisticshive_partitioningglobschema hive_schematry_parse_hive_datesrechunk low_memorystorage_optionscredential_providerretries use_pyarrowpyarrow_options memory_mapinclude_file_pathsmissing_columnsallow_missing_columnsc nU b Sn[U5 U b Sn[U5 U(aNUb Sn[U5eUb Sn[U5eU b Sn[U5eU b Sn[U5e[UUUUUU S9$Ub[ SS S 9 U(aS OS n[ U40S U_SU_SU_SU_SU_SU_SU _SU _SU _SU _SU _SS_SU_SU_SU_SU_SU_SU_6nUbV[ U5(a5UR[R"U55nUR5$URU5nUR5$) a Read into a DataFrame from a parquet file. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path(s) to a file or directory When needing to authenticate for scanning cloud locations, see the `storage_options` parameter. File-like objects are supported (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). For file-like objects, the stream position may not be updated accordingly after reading. columns Columns to select. Accepts a list of column indices (starting at zero) or a list of column names. n_rows Stop reading from parquet file after reading `n_rows`. Only valid when `use_pyarrow=False`. row_index_name Insert a row index column with the given name into the DataFrame as the first column. If set to `None` (default), no row index column is created. row_index_offset Start the row index at this offset. Cannot be negative. Only used if `row_index_name` is set. parallel : {'auto', 'columns', 'row_groups', 'none'} This determines the direction of parallelism. 'auto' will try to determine the optimal direction. use_statistics Use statistics in the parquet to determine if pages can be skipped from reading. hive_partitioning Infer statistics and schema from Hive partitioned URL and use them to prune reads. This is unset by default (i.e. `None`), meaning it is automatically enabled when a single directory is passed, and otherwise disabled. glob Expand path given via globbing rules. schema Specify the datatypes of the columns. The datatypes must match the datatypes in the file(s). If there are extra columns that are not in the file(s), consider also passing `missing_columns='insert'`. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. hive_schema The column names and data types of the columns by which the data is partitioned. If set to `None` (default), the schema of the Hive partitions is inferred. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. try_parse_hive_dates Whether to try parsing hive values as date/datetime types. rechunk Make sure that all columns are contiguous in memory by aggregating the chunks into a single array. low_memory Reduce memory pressure at the expense of performance. storage_options Options that indicate how to connect to a cloud provider. The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here: * `aws `_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. use_pyarrow Use PyArrow instead of the Rust-native Parquet reader. The PyArrow reader is more stable. pyarrow_options Keyword arguments for `pyarrow.parquet.read_table `_. memory_map Memory map underlying file. This will likely increase performance. Only used when `use_pyarrow=True`. include_file_paths Include the path of the source file(s) as a column with this name. Only valid when `use_pyarrow=False`. missing_columns Configuration for behavior when columns defined in the schema are missing from the data: * `insert`: Inserts the missing columns using NULLs as the row values. * `raise`: Raises an error. allow_missing_columns When reading a list of parquet files, if a column existing in the first file cannot be found in subsequent files, the default behavior is to raise an error. However, if `allow_missing_columns` is set to `True`, a full-NULL column is returned instead of erroring for the files that do not contain the column. .. deprecated:: 1.30.0 Use the parameter `missing_columns` instead and pass one of `('insert', 'raise')`. Returns ------- DataFrame See Also -------- scan_parquet: Lazily read from a parquet file or multiple files via glob patterns. scan_pyarrow_dataset Warnings -------- Calling `read_parquet().lazy()` is an antipattern as this forces Polars to materialize a full parquet file and therefore cannot push any optimizations into the reader. Therefore always prefer `scan_parquet` if you want to work with `LazyFrame` s. z@the `schema` parameter of `read_parquet` is considered unstable.zEthe `hive_schema` parameter of `read_parquet` is considered unstable.z/`n_rows` cannot be used with `use_pyarrow=True`z;`include_file_paths` cannot be used with `use_pyarrow=True`z/`schema` cannot be used with `use_pyarrow=True`zwcannot use `hive_partitions` with `use_pyarrow=True` Hint: Pass `pyarrow_options` instead with a 'partitioning' entry.r-r8r<r=r6zthe parameter `allow_missing_columns` for `read_parquet` is deprecated. Use the parameter `missing_columns` instead and pass one of `('insert', 'raise')`.1.30.0r&insertr,r.r%r)r/r0r1r3r4r5r6r7cacheFr8r9r:r2r>r?) r ValueError TypeError_read_parquet_with_pyarrowr scan_parquetrselectFnthcollect)sourcer-r.r%r)r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@msglfs U/home/james-whalen/.local/lib/python3.13/site-packages/polars/io/parquet/functions.py read_parquetrR4sHPs#Us#  CCS/ !  )OCS/ !  CCS/ !  "X C. ) ++!   (! %  '<(   & *      &  ,    2    ( 0  ! "# $.% &(' B, 7 # #155>*B ::<7#B ::<rBc[SSSS9nU=(d 0n/n[U[5(a>[U5S:a+[US[[ R 45(aUnOU/nOU/n/nUHHn[USUS9n UR"U 4UUS.UD6n SSS5 [W US 9n URU 5 MJ [U5S :XaUS$[U5$!,(df  NL=f) Nzpyarrow.parquetz`_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. Returns ------- dict Dictionary with the metadata. Empty if no custom metadata is available. Fcheck_not_directoryrIN)r8r9r:)rYstrrrr_read_parquet_metadatarZitems)rNr8r9r:credential_provider_builders rQrrvsgd&3+&&#FF"C_n#  !-<-HD&&( )d7  rS)r.r%r)r/r0r1r2hidden_file_prefixr3r4r5r6r7rEr8r9r:r>r?r@ extra_columns cast_options_column_mapping_default_values_deletion_files_table_statisticsc4U b Sn[U5 U b Sn[U5 Ub Sn[U5 Ub Sn[U5 Ub[SSS9 U(aS OS n[U[[45(a [ US S 9nO+[ U5(aUVs/sH n[ US S 9PM snn[UXS 5nA[U[5(a[U[[45(aU/OUn[R"UU UU U[S$0SUbX#4OS_SUbSU4OS_SU_SU_SU_SU_SU_S[U[5(aU/OU_SU_SU _SU _SU _SU_SUb[UR55OS_SU_SU_SU_S U_S!U_S"U_6S#9n[U5$s snf)%a Lazily read from a local or cloud-hosted parquet file (or files). This function allows the query optimizer to push down predicates and projections to the scan level, typically increasing performance and reducing memory overhead. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. .. versionchanged:: 1.30.0 * The `allow_missing_columns` is deprecated in favor of `missing_columns`. Parameters ---------- source Path(s) to a file or directory When needing to authenticate for scanning cloud locations, see the `storage_options` parameter. n_rows Stop reading from parquet file after reading `n_rows`. row_index_name If not None, this will insert a row index column with the given name into the DataFrame row_index_offset Offset to start the row index column (only used if the name is set) parallel : {'auto', 'columns', 'row_groups', 'prefiltered', 'none'} This determines the direction and strategy of parallelism. 'auto' will try to determine the optimal direction. The `prefiltered` strategy first evaluates the pushed-down predicates in parallel and determines a mask of which rows to read. Then, it parallelizes over both the columns and the row groups while filtering out rows that do not need to be read. This can provide significant speedups for large files (i.e. many row-groups) with a predicate that filters clustered rows or filters heavily. In other cases, `prefiltered` may slow down the scan compared other strategies. The `prefiltered` settings falls back to `auto` if no predicate is given. .. warning:: The `prefiltered` strategy is considered **unstable**. It may be changed at any point without it being considered a breaking change. use_statistics Use statistics in the parquet to determine if pages can be skipped from reading. hive_partitioning Infer statistics and schema from hive partitioned URL and use them to prune reads. glob Expand path given via globbing rules. hidden_file_prefix Skip reading files whose names begin with the specified prefixes. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. schema Specify the datatypes of the columns. The datatypes must match the datatypes in the file(s). If there are extra columns that are not in the file(s), consider also passing `missing_columns='insert'`. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. hive_schema The column names and data types of the columns by which the data is partitioned. If set to `None` (default), the schema of the Hive partitions is inferred. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. try_parse_hive_dates Whether to try parsing hive values as date/datetime types. rechunk In case of reading multiple files via a glob pattern rechunk the final DataFrame into contiguous memory chunks. low_memory Reduce memory pressure at the expense of performance. cache Cache the result after reading. storage_options Options that indicate how to connect to a cloud provider. The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here: * `aws `_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. include_file_paths Include the path of the source file(s) as a column with this name. missing_columns Configuration for behavior when columns defined in the schema are missing from the data: * `insert`: Inserts the missing columns using NULLs as the row values. * `raise`: Raises an error. allow_missing_columns When reading a list of parquet files, if a column existing in the first file cannot be found in subsequent files, the default behavior is to raise an error. However, if `allow_missing_columns` is set to `True`, a full-NULL column is returned instead of erroring for the files that do not contain the column. .. deprecated:: 1.30.0 Use the parameter `missing_columns` instead and pass one of `('insert', 'raise')`. extra_columns Configuration for behavior when extra columns outside of the defined schema are encountered in the data: * `ignore`: Silently ignores. * `raise`: Raises an error. cast_options Configuration for column type-casting during scans. Useful for datasets containing files that have differing schemas. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. See Also -------- read_parquet scan_pyarrow_dataset Examples -------- Scan a local Parquet file. >>> pl.scan_parquet("path/to/file.parquet") # doctest: +SKIP Scan a file on AWS S3. >>> source = "s3://bucket/*.parquet" >>> pl.scan_parquet(source) # doctest: +SKIP >>> storage_options = { ... "aws_access_key_id": "", ... "aws_secret_access_key": "", ... "aws_region": "us-east-1", ... } >>> pl.scan_parquet(source, storage_options=storage_options) # doctest: +SKIP Nz@the `schema` parameter of `scan_parquet` is considered unstable.zEthe `hive_schema` parameter of `scan_parquet` is considered unstable.zFThe `cast_options` parameter of `scan_parquet` is considered unstable.zLThe `hidden_file_prefix` parameter of `scan_parquet` is considered unstable.zthe parameter `allow_missing_columns` for `scan_parquet` is deprecated. Use the parameter `missing_columns` instead and pass one of `('insert', 'raise')`.rCr&rDr,FrlrI row_index pre_slicerrtrsr?r>r2rrr1r4r5r6rEr8r9r:column_mappingdefault_valuesdeletion_filestable_statistics)rcr3r/r7r0 scan_options)r r rYrnrrrrrr\rnew_from_parquetrrZrpr)rNr.r%r)r/r0r1r2rrr3r4r5r6r7rEr8r9r:r>r?r@rsrtrurvrwrxrOrqrcpylfs rQrIrIsTFPs#Us#Vs#%\s#(! %  '<(&3+&&#FF  ( (PV PVf v5 APV #DVn# &(++z&3,/O/O    ' '%  "- 2  &,%7q&kT  & ( ,  2  0#66$$' "0# $$% &"6' () *+ .2A1L_**,-RV/ 2!<3 45 6+7 8+9 :+; </= & DP D>q sF)0rNrr-list[int] | list[str] | Noner. int | Noner% str | Noner)intr/r r0boolr1 bool | Noner2rr3SchemaDict | Noner4rr5rr6rr7rr8dict[str, Any] | Noner93CredentialProviderFunction | Literal['auto'] | Noner:rr;rr<rr=rr>rr?Literal['insert', 'raise']r@rreturnr)rNzWstr | Path | IO[bytes] | bytes | list[str] | list[Path] | list[IO[bytes]] | list[bytes]r-rr8rr<rr=rr6rrr)rNstr | Path | IO[bytes] | bytesrzdict[str, DataType])Nr*r+) rNrr8rr9rr:rrzdict[str, str])8rNrr.rr%rr)rr/r r0rr1rr2rrrzstr | Sequence[str] | Noner3rr4rr5rr6rr7rrErr8rr9rr:rr>rr?rr@rrszLiteral['ignore', 'raise']rtzScanCastOptions | NoneruzColumnMapping | NonervzDefaultFieldValues | NonerwzDeletionFiles | NonerxzDataFrame | Nonerr)@ __future__r contextlibr]collections.abcrpathlibrtypingrrrpolars.functions functionsrKpolarsr rapolars._dependenciesr polars._utils.deprecationr r polars._utils.unstabler polars._utils.variousrrrpolars._utils.wraprpolars.convertrpolars.io._utilsr,polars.io.cloud.credential_provider._builderrpolars.io.scan_options._optionsrsuppress ImportError polars._plrrrrorrrrpolars._typingrrrrr r!polars.io.cloudr"polars.io.scan_optionsr#rRrHrjrIrrSrQrst" $))%0: (%8%'K&55;6-/?R/1CXV-1!%!'%) $%)!%-1OU-1%)29)-1k k*k  k  k  kkk#k k k#kkkk +!k"M#k$%k&'k(+)k*+k,#-k.0/k0'1k23kWSkp-1-1-13! 3!*3!+3!+3!3!3!3!l1<.2OU A *A*AMA A  AH-/?R/1CXV!%!'%)59 $%)!%-1OU%)29)-07+/,015,0*.9V V V V  V  VV#V V3V V#VVVV !V"+#V$M%V&'V(#)V*0+V,'-V../V0)1V2*3V4/5V6*7V8(9V:;VWSVy &%s ? G  G