h&SSKJr SSKrSSKrSSKJr SSKJrJr SSK J r SSK J r J r JrJrJr SSKJr SSKJr SSKJr SSKJrJrJrJrJr SS KJ r J!r! SS K"J#r#J$r$J%r% SS K&J'r'J(r(J)r)J*r* SS K+J,r, SS K-J.r.J/r/ SSK0J1r1 \Rd"\35 SSK4J5r5J6r6 SSS5 \ (a$SSKJ7r7 SSK8J9r9J:r: SSK;Jr> SSK?J@r@ SSK+JArA \"SSSS9\"SSSS9\"SSSS9SSSSSS SSSSSS!S!S!SS\#S"SS#S!S!S!SSSSS$S%SS!S!SS&.!S3S'jj555rBSSSSS SSSSSS!S!S!S\#S"SS#S!S!SSSS$S%SS!S!SS(.S4S)jjrC\"SSSS9\"SSSS9\"SSSS9SSSSSS SSSSS!S!S!S\#S*SS#S!S!SSSS$S%SS!S!S+.S5S,jj555rD\"SSSS9\"SSSS9\"SSSS9SSSS SSSSSS!S!SSS\#SS#S!S!SSSS!S%SSS!S!SSS-S.SSS/."S6S0jj555rESSSS SSSSSS!S!SS\#SS#S!S!SSSS!S%SSS!SSSS.SSS1. S7S2jjrFg!,(df  GN=f)8) annotationsN)Sequence)BytesIOStringIO)Path)IO TYPE_CHECKINGAnyCallableLiteral)deprecate_renamed_parameter)_process_null_valuesis_path_or_str_sequenceis_str_sequencenormalize_filepathqualified_type_name)wrap_dfwrap_ldf)N_INFER_DEFAULTStringparse_into_dtype)is_glob_patternparse_columns_argparse_row_index_argsprepare_file_arg)!_init_credential_provider_builder)_check_arg_is_1byte_update_columns)BatchedCsvReader) PyDataFrame PyLazyFrame)Mapping) DataFrame LazyFrame) CsvEncodingPolarsDataType SchemaDict)CredentialProviderFunction)CredentialProviderBuilderdtypesschema_overridesz0.20.31)versionrow_count_namerow_index_namez0.20.4row_count_offsetrow_index_offsetT,"Fi utf8i )! has_headercolumns new_columns separatorcomment_prefix quote_char skip_rows skip_linesschemar+ null_valuesmissing_utf8_is_empty_string ignore_errorstry_parse_dates n_threads infer_schemainfer_schema_length batch_sizen_rowsencoding low_memoryrechunk use_pyarrowstorage_optionsskip_rows_after_headerr.r0 sample_sizeeol_charraise_if_emptytruncate_ragged_lines decimal_commaglobc! [SUSS9 [SUSS9 [SUSS9 [U5un"nU=(d 0nU(a3U(d,UH&n#U#RS5(aMSn$[U$5e U b([ U [ [ 45(d S n$[U$5eU(GaU GcUGcUGcU(Gd|U GcxS n%U(a.U(d%UV#s/sHn#S [U#S S 5S - 3PM n%n#OUn%U(dU"(aU"V&s/sHn&S U&3PM n%n&[US SUUS9n'SS K n(SS K n)U(RRU'U(RRUUU(+US9U(RRUU(aUOSUS L=(a US:HS9U(RR!S U%U S95n*S S S 5 U(d>W*R+U*R,V#s/sHn#S[U#S S 5S -3PM sn#5n*[&R(R/W*US9n,U(a [1U,U5$U,$U"(aU (a[ U [25(ak[5U"5[5U 5:a Sn$[U$5e[6/[9U"5S --n-[;U"5Hun.n&U.[5U 5:dMU U.U-U&'M! U-n U(aUU (aN[ U [25(a9[5U5[5U 5:a Sn$[U$5e[ [=X*55n U(GaU (Ga[ U [ 5(Ga~S n/U(a4[5U5[5U5:a Sn$[U$5eUS[5U5n/OU(dxU"(aEU(a%[5U5[5U5:a Sn$[U$5eU"V&s/sH n&SU&S -3PM n/n&O[?S [5U5S -5V&s/sHn&SU&3PM n/n&O[[5U 5[5U5::aCUS[5U 5V0s/sHn0U0U ;dM U U0PM n1n0[5U15[5U 5:XaU1n U/(a`[ U [ 5(aK[ [=UU/55n2U RA5V3V4s0sHun3n4U2RCU3U35U4_M n n3n4U(dSn[ U [ 5n5US;n6[DRF"S5S:H=(d [DRF"S5S:Hn7U7(dh[ U=n8[$[H45(Gax[%U85RS5(d+[DRF"S5S:XGa>U5(Gd6U6(Ga.[ U[$[H45(a [KUSS9nO*[MUSS 9(aUVs/sH n[KUSS9PM snnU7(d,U5(a S!n$[U$5eU6(dS"US#3n$[U$5e[OU40S$U_SU_S%U_SU_S&U_S'U_S(U _S)U _S*U _S+U _S,U _S-U_S.U_S/U_S0U_S1U_S2U_S3U_S4U_S5U_SU_S6U_S7U_S8U _S9U!_6n9U(aU9RQU5n9O,U"(a%U9RQ[RRT"U"55n9U9RW5n,O[UUSUUS9n'[YU'40S$U_S:U(aUOU"_SU_S%U_SU_S&U_S'U_S(U _S)U _S*U _S+U _S,U _S-U_S;U_S.U_S_S1U_S2U_S3U_S4U_S5U_SU_S6U_S7U_S8U _S9U!_6n,S S S 5 U(a [1W,U5$W,$s sn#fs sn&f!U(R"a?n+U(dS[%U+5;ae[&R("5sS n+A+sS S S 5 $S n+A+ff=f!,(df  GN=fs sn#fs sn&fs sn&fs sn0fs sn4n3fs snf!,(df  N=f)?uv Read a CSV file into a DataFrame. Polars expects CSV data to strictly conform to RFC 4180, unless documented otherwise. Malformed data, though common, may lead to undefined behavior. .. versionchanged:: 0.20.31 The `dtypes` parameter was renamed `schema_overrides`. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path to a file or a file-like object (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). If `fsspec` is installed, it will be used to open remote files. For file-like objects, the stream position may not be updated accordingly after reading. has_header Indicate if the first row of the dataset is a header or not. If set to False, column names will be autogenerated in the following format: `column_x`, with `x` being an enumeration over every column in the dataset, starting at 1. columns Columns to select. Accepts a list of column indices (starting at zero) or a list of column names. new_columns Rename columns right after parsing the CSV file. If the given list is shorter than the width of the DataFrame the remaining columns will have their original name. separator Single byte character to use as separator in the file. comment_prefix A string used to indicate the start of a comment line. Comment lines are skipped during parsing. Common examples of comment prefixes are `#` and `//`. quote_char Single byte character used for csv quoting, default = `"`. Set to None to turn off special handling and escaping of quotes. skip_rows Start reading after ``skip_rows`` rows. The header will be parsed at this offset. Note that we respect CSV escaping/comments when skipping rows. If you want to skip by newline char only, use `skip_lines`. skip_lines Start reading after `skip_lines` lines. The header will be parsed at this offset. Note that CSV escaping will not be respected when skipping lines. If you want to skip valid CSV rows, use ``skip_rows``. schema Provide the schema. This means that polars doesn't do schema inference. This argument expects the complete schema, whereas `schema_overrides` can be used to partially overwrite a schema. Note that the order of the columns in the provided `schema` must match the order of the columns in the CSV being read. schema_overrides Overwrite dtypes for specific or all columns during schema inference. null_values Values to interpret as null values. You can provide a: - `str`: All values equal to this string will be null. - `List[str]`: All values equal to any string in this list will be null. - `Dict[str, str]`: A dictionary that maps column name to a null value string. missing_utf8_is_empty_string By default a missing value is considered to be null; if you would prefer missing utf8 values to be treated as the empty string you can set this param True. ignore_errors Try to keep reading lines if some lines yield errors. Before using this option, try to increase the number of lines used for schema inference with e.g `infer_schema_length=10000` or override automatic dtype inference for specific columns with the `schema_overrides` option or use `infer_schema=False` to read all columns as `pl.String` to check which values might cause an issue. try_parse_dates Try to automatically parse dates. Most ISO8601-like formats can be inferred, as well as a handful of others. If this does not succeed, the column remains of data type `pl.String`. If `use_pyarrow=True`, dates will always be parsed. n_threads Number of threads to use in csv parsing. Defaults to the number of physical cpu's of your system. infer_schema When `True`, the schema is inferred from the data using the first `infer_schema_length` rows. When `False`, the schema is not inferred and will be `pl.String` if not specified in `schema` or `schema_overrides`. infer_schema_length The maximum number of rows to scan for schema inference. If set to `None`, the full data may be scanned *(this is slow)*. Set `infer_schema=False` to read all columns as `pl.String`. batch_size Number of lines to read into the buffer at once. Modify this to change performance. n_rows Stop reading from CSV file after reading `n_rows`. During multi-threaded parsing, an upper bound of `n_rows` rows cannot be guaranteed. encoding : {'utf8', 'utf8-lossy', 'windows-1252', 'windows-1252-lossy', ...} Lossy means that invalid utf8 values are replaced with `�` characters. When using other encodings than `utf8` or `utf8-lossy`, the input is first decoded in memory with python. Defaults to `utf8`. low_memory Reduce memory pressure at the expense of performance. rechunk Make sure that all columns are contiguous in memory by aggregating the chunks into a single array. use_pyarrow Try to use pyarrow's native CSV parser. This will always parse dates, even if `try_parse_dates=False`. This is not always possible. The set of arguments given to this function determines if it is possible to use pyarrow's native parser. Note that pyarrow and polars may have a different strategy regarding type inference. storage_options Extra options that make sense for `fsspec.open()` or a particular storage connection. e.g. host, port, username, password, etc. skip_rows_after_header Skip this number of rows when the header is parsed. row_index_name Insert a row index column with the given name into the DataFrame as the first column. If set to `None` (default), no row index column is created. row_index_offset Start the row index at this offset. Cannot be negative. Only used if `row_index_name` is set. sample_size Set the sample size. This is used to sample statistics to estimate the allocation needed. .. deprecated:: 1.10.0 This parameter is now a no-op. eol_char Single byte end of line character (default: `\n`). When encountering a file with windows line endings (`\r\n`), one can go with the default `\n`. The extra `\r` will be removed when processed. raise_if_empty When there is no data in the source, `NoDataError` is raised. If this parameter is set to False, an empty DataFrame (with no columns) is returned instead. truncate_ragged_lines Truncate lines that are longer than the schema. decimal_comma Parse floats using a comma as the decimal separator instead of a period. glob Expand path given via globbing rules. Returns ------- DataFrame See Also -------- scan_csv : Lazily read from a CSV file or multiple files via glob patterns. Warnings -------- Calling `read_csv().lazy()` is an antipattern as this forces Polars to materialize a full csv file and therefore cannot push any optimizations into the reader. Therefore always prefer `scan_csv` if you want to work with `LazyFrame` s. Notes ----- If the schema is inferred incorrectly (e.g. as `pl.Int64` instead of `pl.Float64`), try to increase the number of lines used to infer the schema with `infer_schema_length` or override the inferred dtype for those columns with `schema_overrides`. Examples -------- >>> pl.read_csv("data.csv", separator="|") # doctest: +SKIP Demonstrate use against a BytesIO object, parsing string dates. >>> from io import BytesIO >>> data = BytesIO( ... b"ID,Name,Birthday\n" ... b"1,Alice,1995-07-12\n" ... b"2,Bob,1990-09-20\n" ... b"3,Charlie,2002-03-08\n" ... ) >>> pl.read_csv(data, try_parse_dates=True) shape: (3, 3) ┌─────┬─────────┬────────────┐ │ ID ┆ Name ┆ Birthday │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ date │ ╞═════╪═════════╪════════════╡ │ 1 ┆ Alice ┆ 1995-07-12 │ │ 2 ┆ Bob ┆ 1990-09-20 │ │ 3 ┆ Charlie ┆ 2002-03-08 │ └─────┴─────────┴────────────┘ r8F can_be_emptyr:TrNcolumn_aspecified column names do not start with 'column_', but autogenerated header names were requestedN1`schema_overrides` should be of type list or dictf)rGrJrOrKr)r;skip_rows_after_namesautogenerate_column_namesrGr2) delimiterr: double_quote) column_typesinclude_columnsinclude_missing_columnsz Empty CSV)rICmore schema overrides are specified than there are selected columnszCmore dtypes overrides are specified than there are selected columnsCmore new column names are specified than there are selected columns>r3 utf8-lossyPOLARS_FORCE_NEW_STREAMING1POLARS_AUTO_NEW_STREAMINGzhf://POLARS_FORCE_ASYNCcheck_not_directory allow_strzCpassing a list to `schema_overrides` is unsupported for hf:// pathszunsupported encoding z for hf:// pathsr5r9r;r<r+r=r>r?r@rArDrFrGrHrIrLr.r0rOrPrQrRr6rBrErer3)-rr startswith ValueError isinstancedictr TypeErrorintrpyarrow pyarrow.csvcsvread_csv ReadOptions ParseOptionsConvertOptions ArrowInvalidstrplr#rename_columns column_names _from_arrowrlistlenrmax enumerateziprangeitemsgetosgetenvrrr_scan_csv_implselectFnthcollect_read_csv_impl):sourcer5r6r7r8r9r:r;r<r=r+r>r?r@rArBrCrDrErFrGrHrIrJrKrLr.r0rMrNrOrPrQrR projectioncolumnmsgra column_idxdataparttblerrdf dtypes_listidxcurrent_columnsnew_column_name dtype_listnew_to_current column_name column_dtypeschema_overrides_is_listencoding_supported_in_lazy new_streamingvlfs: Q/home/james-whalen/.local/lib/python3.13/site-packages/polars/io/csv/functions.pyrwrw.sE R YUC jtD H5A+G4J%+OzF$$Y//E!o% #J4*--Bn   $ N    04 LS"S7Qs6!":':&;#<7"S"):CMM*J:,/*OM  )+    &ffooFF&&"+.D6@.!) ' FF''"+1;:%/t%;%Q c@Q( FF))%)(70=* F$$?B?O?OP?OV73vabz?Q./0?OPC\\ % %c7 % ; "2{3 3 &:6F+M+M z?S!12 2WCS/ !.4HJ!8K,L (4OCS)***:3*? J' 5'# 3CT(J(J w<#./ /WCS/ !  G >?'J7G,N,N 7|c+..!o%&a#k*:;Os7|c+.>>#%S/)BL#AK:gj1n-. #',As;/?!/C&D#&D j\*&D ##$K(88,7q3?O;P+Q+Q&*::6$_5+Q z?c*:&;;'1$ z*:DAA!#k?"CDN2B1G1G1I 1I-K""; -. ./0"212"34 .56'<78,9:;B Nr;// I[#TN@?? &![C%@||~%C  < &=  NQz# # Nd  sz] ]' ^01B] __2_ , _: _# __A0_!^-.+^(^-^0(^--^00 ^?! _/)r5r6r8r9r:r;r<r=r+r>r?r@rArBrDrErFrGrHrIrLr.r0rMrNrOrPrQrRc[U[[45(a [USS9nOZSn[U[5(aUR 5n[U[ 5(aUR 5R5nSnSn U b[U [5(a9/nU R5H"un!n"URU![U"545 M$ O2[U [5(aU n OS[U 5<3n#[U#5e[U 5n$[U[5(aU/n[U[5(a[!U5(aSn%Ub [U5n%U b Sn#[#U#5eSSKJn& U&"U40SU_SU_S U_S U_S U_S U_S U_SU%_SU _SU _SU _SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_6n'UcU'R)5$[+USS9(aU'R-U5R)5$Sn#[#U#5e[/U5un(n[0R2"UUUUU UUUU(UUUUUUUU UUUU$U U U[5UU54UUUUUS .6n)[7U)5$)!NFrjz7`schema_overrides` should be of type list or dict, got zcannot use glob patterns and unnamed dtypes as `schema_overrides` argument Use `schema_overrides`: Mapping[str, Type[DataType]]r)scan_csvr5r8r9r:r;r<r=r+r>r?r@rDrFrHrIrLr.r0rNrOrPrQrRrlzccannot use glob patterns and integer based projection as `columns` argument Use columns: List[str])rNrOrPrQr=)rpr|rrrgetvaluerencoderqrappendrrrrrrrropolarsrrrrrr rwrr)*rr5r6r8r9r:r;r<r=r+r>r?r@rArBrDrErFrGrHrIrLr.r0rMrNrOrPrQrRpathr dtype_slicekrrprocessed_null_values dtypes_dictrscanrpydfs* rrrJs#D&3+&&!&eD fg & &__&F fh ' '__&--/F>BJ37K# & - -J(..01!!1&6q&9":;1 (( 3 3*KKL_`pLqKtuCC. 0='3)&#?6#:#:  !z*K  "K S/ !#  !   *  "    "  ) $ *F ( !4  " ! "$:# $*% &.' () **+ ,#8- .(/ 01 4 ?<<> ! W 6;;w'//1 1- S/ !+G4J    $^-=>3 4%3#= D@ 4=iP)r5r6r7r8r9r:r;r<r+r>r?r@rArBrDrErFrGrHrIrLr.r0rMrNrOrPrQc[U5unnU(a3U(d,UH&nURS5(aMSn[U5e U(aU (a[U [5(ak[ U5[ U 5:a Sn[U5e[ /[U5S--n [U5Hun!n"U![ U 5:dMU U!U U"'M! U n U(aUU (aN[U [5(a9[ U5[ U 5:a Sn[U5e[[X)55n U(GaU (Ga[U [5(Ga~Sn#U(a4[ U5[ U5:a Sn[U5eUS[ U5n#OU(dxU(aEU(a%[ U5[ U5:a Sn[U5eUV"s/sH n"SU"S-3PM n#n"O[S[ U5S-5V"s/sHn"SU"3PM n#n"O[[ U 5[ U5::aCUS[ U 5V$s/sHn$U$U ;dM U U$PM n%n$[ U%5[ U 5:XaU%n U#(a`[U [5(aK[[UU#55n&U R5V'V(s0sHun'n(U&RU'U'5U(_M n n'n([U40SU_S U(aUOU_S U_S U_S U_S U_SU_SU _SU _SU _SU _SU _SU_SU_SU_SU_SUS:XaUOS_SU_SU_SU_SU_SU_S U_S!U_S"U_S#U_S$U_6$s sn"fs sn"fs sn$fs sn(n'f)%u Read a CSV file in batches. Upon creation of the `BatchedCsvReader`, Polars will gather statistics and determine the file chunks. After that, work will only be done if `next_batches` is called, which will return a list of `n` frames of the given batch size. .. versionchanged:: 0.20.31 The `dtypes` parameter was renamed `schema_overrides`. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path to a file or a file-like object (by "file-like object" we refer to objects that have a `read()` method, such as a file handler like the builtin `open` function, or a `BytesIO` instance). If `fsspec` is installed, it will be used to open remote files. For file-like objects, the stream position may not be updated accordingly after reading. has_header Indicate if the first row of the dataset is a header or not. If set to False, column names will be autogenerated in the following format: `column_x`, with `x` being an enumeration over every column in the dataset, starting at 1. columns Columns to select. Accepts a list of column indices (starting at zero) or a list of column names. new_columns Rename columns right after parsing the CSV file. If the given list is shorter than the width of the DataFrame the remaining columns will have their original name. separator Single byte character to use as separator in the file. comment_prefix A string used to indicate the start of a comment line. Comment lines are skipped during parsing. Common examples of comment prefixes are `#` and `//`. quote_char Single byte character used for csv quoting, default = `"`. Set to None to turn off special handling and escaping of quotes. skip_rows Start reading after ``skip_rows`` rows. The header will be parsed at this offset. Note that we respect CSV escaping/comments when skipping rows. If you want to skip by newline char only, use `skip_lines`. skip_lines Start reading after `skip_lines` lines. The header will be parsed at this offset. Note that CSV escaping will not be respected when skipping lines. If you want to skip valid CSV rows, use ``skip_rows``. schema_overrides Overwrite dtypes during inference. null_values Values to interpret as null values. You can provide a: - `str`: All values equal to this string will be null. - `List[str]`: All values equal to any string in this list will be null. - `Dict[str, str]`: A dictionary that maps column name to a null value string. missing_utf8_is_empty_string By default a missing value is considered to be null; if you would prefer missing utf8 values to be treated as the empty string you can set this param True. ignore_errors Try to keep reading lines if some lines yield errors. First try `infer_schema_length=0` to read all columns as `pl.String` to check which values might cause an issue. try_parse_dates Try to automatically parse dates. Most ISO8601-like formats can be inferred, as well as a handful of others. If this does not succeed, the column remains of data type `pl.String`. n_threads Number of threads to use in csv parsing. Defaults to the number of physical cpu's of your system. infer_schema_length The maximum number of rows to scan for schema inference. If set to `0`, all columns will be read as `pl.String`. If set to `None`, the full data may be scanned *(this is slow)*. batch_size Number of lines to read into the buffer at once. Modify this to change performance. n_rows Stop reading from CSV file after reading `n_rows`. During multi-threaded parsing, an upper bound of `n_rows` rows cannot be guaranteed. encoding : {'utf8', 'utf8-lossy', ...} Lossy means that invalid utf8 values are replaced with `�` characters. When using other encodings than `utf8` or `utf8-lossy`, the input is first decoded in memory with python. Defaults to `utf8`. low_memory Reduce memory pressure at the expense of performance. rechunk Make sure that all columns are contiguous in memory by aggregating the chunks into a single array. skip_rows_after_header Skip this number of rows when the header is parsed. row_index_name Insert a row index column with the given name into the DataFrame as the first column. If set to `None` (default), no row index column is created. row_index_offset Start the row index at this offset. Cannot be negative. Only used if `row_index_name` is set. sample_size Set the sample size. This is used to sample statistics to estimate the allocation needed. .. deprecated:: 1.10.0 Is a no-op. eol_char Single byte end of line character (default: `\n`). When encountering a file with windows line endings (`\r\n`), one can go with the default `\n`. The extra `\r` will be removed when processed. raise_if_empty When there is no data in the source,`NoDataError` is raised. If this parameter is set to False, `None` will be returned from `next_batches(n)` instead. truncate_ragged_lines Truncate lines that are longer than the schema. decimal_comma Parse floats using a comma as the decimal separator instead of a period. Returns ------- BatchedCsvReader See Also -------- scan_csv : Lazily read from a CSV file or multiple files via glob patterns. Examples -------- >>> reader = pl.read_csv_batched( ... "./pdsh/tables_scale_100/lineitem.tbl", ... separator="|", ... try_parse_dates=True, ... ) # doctest: +SKIP >>> batches = reader.next_batches(5) # doctest: +SKIP >>> for df in batches: # doctest: +SKIP ... print(df) Read big CSV file in batches and write a CSV file for each "group" of interest. >>> seen_groups = set() >>> reader = pl.read_csv_batched("big_file.csv") # doctest: +SKIP >>> batches = reader.next_batches(100) # doctest: +SKIP >>> while batches: # doctest: +SKIP ... df_current_batches = pl.concat(batches) ... partition_dfs = df_current_batches.partition_by("group", as_dict=True) ... ... for group, df in partition_dfs.items(): ... if group in seen_groups: ... with open(f"./data/{group}.csv", "a") as fh: ... fh.write(df.write_csv(file=None, include_header=False)) ... else: ... df.write_csv(file=f"./data/{group}.csv", include_header=True) ... seen_groups.add(group) ... ... batches = reader.next_batches(100) rVrWrcr[Nrdrr5r6r8r9r:r;r<r+r>r?r@rArBrDrErFrGrer3rHrIrLr.r0rNr7rOrPrQ)rrnrorprrrrrrqrrrrr))rr5r6r7r8r9r:r;r<r+r>r?r@rArBrDrErFrGrHrIrLr.r0rMrNrOrPrQrrrrrrrrrrrrs) rread_csv_batchedrsH,G4JzF$$Y//E!o% &:6F+M+M z?S!12 2WCS/ !.4HJ!8K,L (4OCS)***:3*? J' 5'# 3CT(J(J w<#./ /WCS/ !  G >?'J7G,N,N 7|c+..[ o%&a#k*:;Os7|c+.>>_C$S/)BL#AK:gj1n-. #',As;/?!/C&D#&D j\*&D ##$K(88,7q3?O;P+Q+Q&*::6$_5+Q z?c*:&;;'1$ z*:DAA!#k?"CDN2B1G1G1I 1I-K""; r?r@cachewith_column_namesrCrDrFrGrHrIrLr.r0rArNr7rOrPrQrRrKcredential_providerretriesfile_cache_ttlinclude_file_pathsc"H^Ub([U[[45(d Sn#[U#5eT(d/[U[5(aS[ U5<3n#[U#5eT(aMU (a Sn#[ U#5eU(a*[U[5(a[[ TU55nS,U4Sjjn [SUSS9 [SUS S9 [U[[45(a [USS 9nO*[USS 9(aUVs/sH n[USS 9PM snnU(dS n[UUUS 5n$A[U40SU_SU_SU_SU_SU_SU_SU_SU_SU _SU _SU _SU _SU _SU_SU_SU_SU_SU_SU_SU_S U_S!U_S"U_S#U_S$U_S%U_S&U_S'U _S(U_S)U$_S*U!_S+U"_6$s snf)-u! Lazily read from a CSV file or multiple files via glob patterns. This allows the query optimizer to push down predicates and projections to the scan level, thereby potentially reducing memory overhead. .. versionchanged:: 0.20.31 The `dtypes` parameter was renamed `schema_overrides`. .. versionchanged:: 0.20.4 * The `row_count_name` parameter was renamed `row_index_name`. * The `row_count_offset` parameter was renamed `row_index_offset`. Parameters ---------- source Path(s) to a file or directory When needing to authenticate for scanning cloud locations, see the `storage_options` parameter. has_header Indicate if the first row of the dataset is a header or not. If set to False, column names will be autogenerated in the following format: `column_x`, with `x` being an enumeration over every column in the dataset, starting at 1. separator Single byte character to use as separator in the file. comment_prefix A string used to indicate the start of a comment line. Comment lines are skipped during parsing. Common examples of comment prefixes are `#` and `//`. quote_char Single byte character used for csv quoting, default = `"`. Set to None to turn off special handling and escaping of quotes. skip_rows Start reading after ``skip_rows`` rows. The header will be parsed at this offset. Note that we respect CSV escaping/comments when skipping rows. If you want to skip by newline char only, use `skip_lines`. skip_lines Start reading after `skip_lines` lines. The header will be parsed at this offset. Note that CSV escaping will not be respected when skipping lines. If you want to skip valid CSV rows, use ``skip_rows``. schema Provide the schema. This means that polars doesn't do schema inference. This argument expects the complete schema, whereas `schema_overrides` can be used to partially overwrite a schema. Note that the order of the columns in the provided `schema` must match the order of the columns in the CSV being read. schema_overrides Overwrite dtypes during inference; should be a {colname:dtype,} dict or, if providing a list of strings to `new_columns`, a list of dtypes of the same length. null_values Values to interpret as null values. You can provide a: - `str`: All values equal to this string will be null. - `List[str]`: All values equal to any string in this list will be null. - `Dict[str, str]`: A dictionary that maps column name to a null value string. missing_utf8_is_empty_string By default a missing value is considered to be null; if you would prefer missing utf8 values to be treated as the empty string you can set this param True. ignore_errors Try to keep reading lines if some lines yield errors. First try `infer_schema=False` to read all columns as `pl.String` to check which values might cause an issue. cache Cache the result after reading. with_column_names Apply a function over the column names just in time (when they are determined); this function will receive (and should return) a list of column names. infer_schema When `True`, the schema is inferred from the data using the first `infer_schema_length` rows. When `False`, the schema is not inferred and will be `pl.String` if not specified in `schema` or `schema_overrides`. infer_schema_length The maximum number of rows to scan for schema inference. If set to `None`, the full data may be scanned *(this is slow)*. Set `infer_schema=False` to read all columns as `pl.String`. n_rows Stop reading from CSV file after reading `n_rows`. encoding : {'utf8', 'utf8-lossy'} Lossy means that invalid utf8 values are replaced with `�` characters. Defaults to "utf8". low_memory Reduce memory pressure at the expense of performance. rechunk Reallocate to contiguous memory when all chunks/ files are parsed. skip_rows_after_header Skip this number of rows when the header is parsed. row_index_name If not None, this will insert a row index column with the given name into the DataFrame. row_index_offset Offset to start the row index column (only used if the name is set). try_parse_dates Try to automatically parse dates. Most ISO8601-like formats can be inferred, as well as a handful of others. If this does not succeed, the column remains of data type `pl.String`. eol_char Single byte end of line character (default: `\n`). When encountering a file with windows line endings (`\r\n`), one can go with the default `\n`. The extra `\r` will be removed when processed. new_columns Provide an explicit list of string column names to use (for example, when scanning a headerless CSV file). If the given list is shorter than the width of the DataFrame the remaining columns will have their original name. raise_if_empty When there is no data in the source, `NoDataError` is raised. If this parameter is set to False, an empty LazyFrame (with no columns) is returned instead. truncate_ragged_lines Truncate lines that are longer than the schema. decimal_comma Parse floats using a comma as the decimal separator instead of a period. glob Expand path given via globbing rules. storage_options Options that indicate how to connect to a cloud provider. The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here: * `aws `_ * `gcp `_ * `azure `_ * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \ `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable. If `storage_options` is not provided, Polars will try to infer the information from environment variables. credential_provider Provide a function that can be called to provide cloud storage credentials. The function is expected to return a dictionary of credential keys along with an optional credential expiry time. .. warning:: This functionality is considered **unstable**. It may be changed at any point without it being considered a breaking change. retries Number of retries if accessing a cloud instance fails. file_cache_ttl Amount of time to keep downloaded cloud files since their last access time, in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable (which defaults to 1 hour) if not given. include_file_paths Include the path of the source file(s) as a column with this name. Returns ------- LazyFrame See Also -------- read_csv : Read a CSV file into a DataFrame. Examples -------- >>> import pathlib >>> >>> ( ... pl.scan_csv("my_long_file.csv") # lazy, doesn't do a thing ... .select( ... ["a", "c"] ... ) # select only 2 columns (other columns will not be read) ... .filter( ... pl.col("a") > 10 ... ) # the filter is pushed down the scan, so less data is read into memory ... .head(100) # constrain number of returned results to 100 ... ) # doctest: +SKIP We can use `with_column_names` to modify the header before scanning: >>> df = pl.DataFrame( ... {"BrEeZaH": [1, 2, 3, 4], "LaNgUaGe": ["is", "hard", "to", "read"]} ... ) >>> path: pathlib.Path = dirpath / "mydf.csv" >>> df.write_csv(path) >>> pl.scan_csv( ... path, with_column_names=lambda cols: [col.lower() for col in cols] ... ).collect() shape: (4, 2) ┌─────────┬──────────┐ │ breezah ┆ language │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═════════╪══════════╡ │ 1 ┆ is │ │ 2 ┆ hard │ │ 3 ┆ to │ │ 4 ┆ read │ └─────────┴──────────┘ You can also simply replace column names (or provide them if the file has none) by passing a list of new column names to the `new_columns` parameter: >>> df.write_csv(path) >>> pl.scan_csv( ... path, ... new_columns=["idx", "txt"], ... schema_overrides=[pl.UInt16, pl.String], ... ).collect() shape: (4, 2) ┌─────┬──────┐ │ idx ┆ txt │ │ --- ┆ --- │ │ u16 ┆ str │ ╞═════╪══════╡ │ 1 ┆ is │ │ 2 ┆ hard │ │ 3 ┆ to │ │ 4 ┆ read │ └─────┴──────┘ rXz(expected 'schema_overrides' dict, found zIcannot set both `with_column_names` and `new_columns`; mutually exclusivecZ>[U5[T5:aTU[T5S-$T$)N)r)colsr7s rr#scan_csv..with_column_names5s24y3{++"T#k*:*<%===""rr8FrTr:Trjrlrrr5r9r;r<r+r=r>r?r@rrrDrFrHrIrLrGr.r0rArNrOrPrQrRrrKrrr)r list[str]returnr)rprqrrrrrorrr|rrrrr)%rr5r8r9r:r;r<r=r+r>r?r@rrrCrDrFrGrHrIrLr.r0rArNr7rOrPrQrRrKrrrrrcredential_provider_builders% ` rrrs~N#J4*--Bn :&6AA89LM]9^8abn  ]CS/ !  +;X F F#C 5E$FG  #  YUC jtD&3+&&#FF 5 9PV PVf v5 APV  "CV_j#  " " " & "   "   " " *" "  " &B" $" " ," 0"  !" "#" $%" & 6'" ()" *&+" ,*-" .(/" 01" 2&3" 445" 6$7" 89" :;" <(=" >8?" @&A" B.C" " sF) r5r8r9r:r;r<r=r+r>r?r@rrrDrFrGrHrIrLr.r0rArNrOrPrQrRrKrrrrc dSn!UbZ[U[5(d Sn"[U"5e/n!UR5H"un#n$U!R U#[ U$545 M$ [ U 5n%[U[5(aUn&SnO/n&U(a[UR55nOSn[R"UU&40SU_SU_SU _SU_SU_SU_SU _S U!_S U_S U_S U_S U%_SU _SU_SU _SU_SU_SU_S[UU5_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_SU_S U _6n'[U'5$)!Nz.expected 'schema_overrides' dict, found 'list'r8r5r@r;r<rFroverwrite_dtyperHr9r:r>r?rDwith_schema_modifyrIrLrG row_indexrArNrOrPrQrRr= cloud_optionsrrrr) rprqrrrrrrrr! new_from_csvrr)(rr5r8r9r:r;r<r=r+r>r?r@rrrDrFrGrHrIrLr.r0rArNrOrPrQrRrKrrrrrrrrrsourcespylfs( rrrrs X;?J#*D11BCC.  $**,DAq   q"21"56 7-0=&$4467  # #" "  "  $ "   " " " " #" " &" " *" &B"  0!" "-#" $%" & 6'" ()" *'~7GH+" ,(-" ./" 0&1" 243" 4$5" 67" 89" :&;" <0=" >?" @&A" B.C" DF D>r)Frz(str | Path | IO[str] | IO[bytes] | bytesr5boolr6$Sequence[int] | Sequence[str] | Noner7Sequence[str] | Noner8r|r9 str | Noner:rr;rsr<rsr=SchemaDict | Noner+>Mapping[str, PolarsDataType] | Sequence[PolarsDataType] | Noner>+str | Sequence[str] | dict[str, str] | Noner?rr@rrArrB int | NonerCrrDrrErsrFrrGCsvEncoding | strrHrrIrrJrrKdict[str, Any] | NonerLrsr.rr0rsrMrsrNr|rOrrPrrQrrRrrr#)>rzstr | Path | IO[bytes] | bytesr5rr6rr8r|r9rr:rr;rsr<rsr=zNone | SchemaDictr+z.None | (SchemaDict | Sequence[PolarsDataType])r>rr?rr@rrArrBrrDrrErsrFrrGr%rHrrIrrLrsr.rr0rsrMrsrNr|rOrrPrrQrrRrrr#)rr?rr@rrArrBrrDrrErsrFrrGrrHrrIrrLrsr.rr0rsrMrsrNr|rOrrPrrQrrr)Hrzqstr | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]] | list[bytes]r5rr8r|r9rr:rr;rsr<rsr=rr+z,SchemaDict | Sequence[PolarsDataType] | Noner>rr?rr@rrrr'Callable[[list[str]], list[str]] | NonerCrrDrrFrrGr%rHrrIrrLrsr.rr0rsrArrNr|r7rrOrrPrrQrrRrrKrrz3CredentialProviderFunction | Literal['auto'] | Nonerrsrrrrrr$)Drzjstr | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]] | list[bytes]r5rr8r|r9rr:rr;rsr<rsr=rr+rr>rr?rr@rrrrrrDrrFrrGr%rHrrIrrLrsr.rr0rsrArrNr|rOrrPrrQrrRrrKrrz CredentialProviderBuilder | Nonerrsrrrrrr$)G __future__r contextlibrcollections.abcriorrpathlibrtypingrr r r r polars._reexport _reexportr}polars.functions functionsrpolars._utils.deprecationr polars._utils.variousrrrrrpolars._utils.wraprrpolars.datatypesrrrpolars.io._utilsrrrr,polars.io.cloud.credential_provider._builderrpolars.io.csv._utilsrrpolars.io.csv.batched_readerrsuppress ImportError polars._plrr r!r"rr#r$polars._typingr%r&r'polars.io.cloudr(r)rwrrrrrrrs " $ <<A1FF F9%4&'+FF:VX'99M-/?R/1CXV48(,!%  $ ?C).! &5"(-1"#!%"'KV 4VV2 V & V  VVVVV V GV=V #'!V"#V$%V&'V()V*$+V,-V. /V0 1V23V45V67V8+9V: ;V<=V>?V@AVBCVDEVF GVHIVJ KVLMVWSNVx48!%  $GK?C).! &5""#!%"'?O *OO2 O  O  OOOO OEO=O#'OOO !O"$#O$%O& 'O()O*+O,-O. /O01O23O45O67O89O: ;O<=O> ?O@AOdX'99M-/?R/1CXV48(,!%  ?C).! &5"("#!%"'A{ {{2 { & {  {{{{{ G{={#'{ !{"#{$%{&$'{(){* +{, -{./{01{2 3{45{67{89{:;{<={> ?{@A{BC{WSN{| X'99M-/?R/1CXV!%  $EI?C).AE&5""#!%!(,"'-1OU!%%)_O OOO !O"#O$%O&'O( )O*C+O,=-O.#'/O01O2 3O4?5O67O8$9O: ;O<=O>?O@AOB CODEOFGOHIOJKOL&MONOOP QORSOT UOV+WOXMYOZ[O\]O^#_O`aOWSNOz !%  $*.?C).AE&5""#!%!"&-1<@!%%)Ue eeeeee !e" #e$(%e&='e(#')e*+e, -e.?/e0$1e2 3e45e67e89e: ;e<=e>?e@AeBCeDEeF GeHIeJ KeL+MeN:OePQeRSeT#UeVWea*&%s  M M