hoSSKrSSKrSSKJr SSKJr SSKJr SSKJ r J r J r J r J r Jr SSKrSSKrSSKJr SSKJr \ (aSSKJrJr \"\5rS rS \S \R>4S jr S \RBS \R>4Sjr"S \S \RF4Sjr$S \S \RJ4Sjr&S \S \R>4Sjr'S\(4Sjr)S\*\+S\+S \+4Sjr,"SS5r-"SS\-5r"SS\5r."SS\.5r/\0\\0\(4r1"SS \.5r2\ "S!\.\*\.\*\*\.5r3"S"S#\5r4SHS$\*\S%\+S \4S&jjr5S'\S \*\4S(jr6S)r7S*\RpS+\+S \94S,jr:S*\RpS \Rv4S-jrS0\ S \4S1jr?\7SIS*\RvS2\RzS3\9S4\9S \\Rv\R\Rp\R\R44 S5jj5rC\7SIS*\RvS6S7S3\9S4\9S \Rv4 S8jj5rD\7SJS*\RvS6S74S9jj5rE"S:S;\F5rGS'\R>SjrHS'\R>S?\RJ4S@jrISJS'\R>4SAjjrJS'\R>S?\RJ4SBjrKS'\R>4SCjrLS'\R>SD\ \Rv/S44SEjrMSKS'\SF\+S \\R>4SGjjrNg)LN)Iterator)partial)groupby) TYPE_CHECKINGAnyCallableOptionalTypeVarUnion) get_logger)Features FeatureTypec^U4SjnU$)Nc>TRURb UROS-UlURRSS5Ul[TS5(aTRUlU$)N pyarrow.TableTable__annotations__)__doc__replacehasattrr)fnarrow_table_methods H/home/james-whalen/.local/lib/python3.13/site-packages/datasets/table.pywrapper1inject_arrow_table_documentation..wrappers_'//AW2::]_` ZZ''A %'8 9 9!3!C!CB  )rrs` r inject_arrow_table_documentationr s Nrfilenamereturnc[R"U5n[RRU5nUR 5nU$N)pa input_streamipc open_streamread_all)r!in_memory_stream opened_streampa_tables r _in_memory_arrow_table_from_filer-!s9x0FF&&'78M%%'H Orbufferc[R"U5n[RRU5nUR 5nU$r$)r% BufferReaderr'r(r))r.streamr+tables r"_in_memory_arrow_table_from_bufferr3(s7 __V $FFF&&v.M  " " $E Lrcl[R"U5n[RRU5$r$)r% memory_mapr'r()r!memory_mapped_streams r,_memory_mapped_record_batch_reader_from_filer7/s'==2 66  2 33rc[R"U5n[RRU5RnSSS5 U$!,(df  W$=f)z{ Infer arrow table schema from file without loading whole file into memory. Useful especially while having very big files. N)r%r5r'r(schema)r!r6r9s rread_schema_from_filer:4sF x $8##$89@@ ! M ! Ms *A  Ac<[U5nUR5nU$r$)r7r))r!r+r,s r$_memory_mapped_arrow_table_from_filer<>s@JM%%'H Ormemoc URnURU5nX1[U5'URR 5H%upE[ X4[ R"XQ55 M' U$)z!deepcopy a regular class instance) __class____new__id__dict__itemssetattrcopydeepcopy)xr=clsresultkvs r _deepcopyrLDsU ++C [[ FAK   "4==12# MrarrrGcXS[U5S- p2X#:apXUs=::aX:a_O O\X#U- XU- -XX- --nXUs=::a XS-:aU$ XU:aUS-Up2OX$p2X#:aXUs=::a X:aMZO [SUS[U5(aUSOSS35e)a= Return the position i of a sorted array so that arr[i] <= x < arr[i+1] Args: arr (`List[int]`): non-empty sorted list of integers x (`int`): query Returns: `int`: the position i so that arr[i] <= x < arr[i+1] Raises: `IndexError`: if the array is empty or if the query is outside the array values rr zInvalid query 'z ' for size none.)len IndexError)rMrGijrJs r_interpolation_searchrVNs c#hlq %CFa(#&( a%AAJ'CFSVO< = 6Q #U #H $ VaZq5!qq %CFa(#&( qcCSWf4UUVW XXrc\rSrSrS\R 4SjrS\\\ \ R4S\R 4Sjr S S\R 4Sjjr S rg) IndexedTableMixinhr2c HURUlUR5Vs/sHn[U5S:dMUPM snUl[ R "S/URVs/sHn[U5PM sn-[ RS9Ulgs snfs snf)Nr)dtype) r9_schema to_batchesrR_batchesnpcumsumint64_offsets)selfr2 recordbatchbs r__init__IndexedTableMixin.__init__is"',, +0+;+;+=/ +=K[AQTUAUK+=/  %'IIqcT]]4S]SV]4S.S[][c[c$d / 5TsBB)B indicesr"c n[U5(d [S5e[R"URUSS9S- n[ R R[X!5VVs/sH3up4URURX@RU- S5PM5 snnURS9$s snnf)a Create a pa.Table by gathering the records at the records at the specified indices. Should be faster than pa.concat_tables(table.fast_slice(int(i) % table.num_rows, 1) for i in indices) since NumPy can compute the binary searches in parallel, highly optimized C zIndices must be non-emptyright)sider r9) rR ValueErrorr_ searchsortedrbr%r from_batcheszipr^slicer\)rcrh batch_indices batch_idxrTs r fast_gatherIndexedTableMixin.fast_gatherps 7||89 9 wWMPQQ xx$$%( $? $?LI i(..q==3K/KQO$? << %   s&:B1 NcUS:a [S5eXRS:d Ub.US::a([RR /UR S9$[ URU5nUbX!-URS:a6URUSnUSRXRU- 5US'O|[ URX-S- 5nURX5S-nUSRSX-URU- 5US'USRXRU- 5US'[RR X@R S9$)a" Slice the Table using interpolation search. The behavior is the same as `pyarrow.Table.slice` but it's significantly faster. Interpolation search is used to find the start and end indexes of the batches we want to keep. The batches to keep are then concatenated to form the sliced Table. rzOffset must be non-negativerONrlr ) rSrbr%rror\rVr^rq)rcoffsetlengthrTbatchesrUs r fast_sliceIndexedTableMixin.fast_slices= A::; ; }}R( (V-?FaK88((DLL(A A !$-- 8 >V_ b0AAmmAB'G ))&==3C*CDGAJ%dmmV_q5HIAmmAA.G!"+++AvqAQ/QRGBK ))&==3C*CDGAJxx$$W\\$BBr)r^rbr\rN)__name__ __module__ __qualname____firstlineno__r%rrfr listintr_ndarrayrtrz__static_attributes__rrrrXrXhsUebhhe 5cBJJ)>#? BHH "C288CCrrXc^\rSrSrSrS\R 4U4SjjrS\4Sjr Sr Sr S r S r S rS rS rS-S\\4SjjrSrSrSr\S5r\S5r\S5r\S5r\S5r\S5r\S5rSrSr Sr!Sr"Sr#Sr$S r%S!r&S"r'S#r(S$r)S%r*S&r+S'r,S(r-S)r.S*r/S+r0S,r1U=r2$).ra Wraps a pyarrow Table by using composition. This is the base class for `InMemoryTable`, `MemoryMappedTable` and `ConcatenationTable`. It implements all the basic attributes/methods of the pyarrow Table class except the Table transforms: `slice, filter, flatten, combine_chunks, cast, add_column, append_column, remove_column, set_column, rename_columns` and `drop`. The implementation of these methods differs for the subclasses. r2c0>[TU]U5 Xlgr$)superrfr2)rcr2r?s rrfTable.__init__s  rr=cURU[UR5'[UR5U[UR5'[ X5$r$)r2rArr^rL)rcr=s r __deepcopy__Table.__deepcopy__sA $zzR ^"&t}}"5R $$rc:URR"U0UD6$)ao Perform validation checks. An exception is raised if validation fails. By default only cheap validation checks are run. Pass `full=True` for thorough validation checks (potentially `O(n)`). Args: full (`bool`, defaults to `False`): If `True`, run expensive checks, otherwise cheap checks only. Raises: `pa.lib.ArrowInvalid`: if validation fails )r2validatercargskwargss rrTable.validateszz""D3F33rc [SU55nUVVs0sH)up4U[U[5(a UROU_M+ nnnURR"U0UD6$s snnf)a Check if contents of two tables are equal. Args: other ([`~datasets.table.Table`]): Table to compare against. check_metadata `bool`, defaults to `False`): Whether schema metadata equality should be checked as well. Returns: `bool` c3h# UH(n[U[5(a UROUv M* g7fr$) isinstancerr2).0args r Table.equals..s%RTc*S%"8"8SYYcATs02)tuplerrr2equals)rcrrrJrKs rr Table.equalssaRTRRHNO! 1e 4 4QWW!;Ozz  $1&11Ps0A'c:URR"U0UD6$)a+ Convert Table to list of (contiguous) `RecordBatch` objects. Args: max_chunksize (`int`, defaults to `None`): Maximum size for `RecordBatch` chunks. Individual chunks may be smaller depending on the chunk layout of individual columns. Returns: `List[pyarrow.RecordBatch]` )r2r]rs rr]Table.to_batchesszz$$d5f55rc:URR"U0UD6$)zF Convert the Table to a `dict` or `OrderedDict`. Returns: `dict` )r2 to_pydictrs rrTable.to_pydictzz##T4V44rc:URR"U0UD6$)z2 Convert the Table to a list Returns: `list` )r2 to_pylistrs rrTable.to_pylistrrc:URR"U0UD6$)a Convert to a pandas-compatible NumPy array or DataFrame, as appropriate. Args: memory_pool (`MemoryPool`, defaults to `None`): Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed. strings_to_categorical (`bool`, defaults to `False`): Encode string (UTF8) and binary types to `pandas.Categorical`. categories (`list`, defaults to `empty`): List of fields that should be returned as `pandas.Categorical`. Only applies to table-like data structures. zero_copy_only (`bool`, defaults to `False`): Raise an `ArrowException` if this function call would require copying the underlying data. integer_object_nulls (`bool`, defaults to `False`): Cast integers with nulls to objects. date_as_object (`bool`, defaults to `True`): Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype. timestamp_as_object (`bool`, defaults to `False`): Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is useful if you have timestamps that don't fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If `False`, all timestamps are converted to `datetime64[ns]` dtype. use_threads (`bool`, defaults to `True`): Whether to parallelize the conversion using multiple threads. deduplicate_objects (`bool`, defaults to `False`): Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower. ignore_metadata (`bool`, defaults to `False`): If `True`, do not use the 'pandas' metadata to reconstruct the DataFrame index, if present. safe (`bool`, defaults to `True`): For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not. split_blocks (`bool`, defaults to `False`): If `True`, generate one internal "block" for each column when creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this can temporarily reduce memory note that various pandas operations can trigger "consolidation" which may balloon memory use. self_destruct (`bool`, defaults to `False`): EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling `to_pandas` with this option it will crash your program. types_mapper (`function`, defaults to `None`): A function mapping a pyarrow DataType to a pandas `ExtensionDtype`. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of `pandas_metadata` in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas `ExtensionDtype` or `None` if the default conversion should be used for that type. If you have a dictionary mapping, you can pass `dict.get` as function. Returns: `pandas.Series` or `pandas.DataFrame`: `pandas.Series` or `pandas.DataFrame` depending on type of object )r2 to_pandasrs rrTable.to_pandassxzz##T4V44rc:URR"U0UD6$r$)r2 to_stringrs rrTable.to_string1szz##T4V44r max_chunksizec4URRUS9$)aq Convert the Table to a RecordBatchReader. Note that this method is zero-copy, it merely exposes the same data under a different API. Args: max_chunksize (`int`, defaults to `None`) Maximum size for RecordBatch chunks. Individual chunks may be smaller depending on the chunk layout of individual columns. Returns: `pyarrow.RecordBatchReader` r)r2 to_reader)rcrs rrTable.to_reader4szz##-#@@rc:URR"U0UD6$)z Select a schema field by its column name or numeric index. Args: i (`Union[int, str]`): The index or name of the field to retrieve. Returns: `pyarrow.Field` )r2fieldrs rr Table.fieldDszz000rc:URR"U0UD6$)z Select a column by its column name, or numeric index. Args: i (`Union[int, str]`): The index or name of the column to retrieve. Returns: `pyarrow.ChunkedArray` )r2columnrs rr Table.columnQszz  $1&11rc:URR"U0UD6$)zY Iterator over all columns in their numerical order. Yields: `pyarrow.ChunkedArray` )r2 itercolumnsrs rrTable.itercolumns^szz%%t6v66rc.URR$)zE Schema of the table and its columns. Returns: `pyarrow.Schema` r2r9rcs rr9 Table.schemagszz   rc.URR$)zO List of all columns in numerical order. Returns: `List[pa.ChunkedArray]` )r2columnsrs rr Table.columnsqszz!!!rc.URR$)z4 Number of columns in this table. Returns: int )r2 num_columnsrs rrTable.num_columns{szz%%%rc.URR$)z~ Number of rows in this table. Due to the definition of a table, all columns have the same number of rows. Returns: int )r2num_rowsrs rrTable.num_rowsszz"""rc.URR$)zo Dimensions of the table: (#rows, #columns). Returns: `(int, int)`: Number of rows and number of columns. )r2shapers rr Table.shapeszzrc.URR$)z> Total number of bytes consumed by the elements of the table. )r2nbytesrs rr Table.nbytess zz   rc.URR$)z Names of the table's columns. )r2 column_namesrs rrTable.column_namess zz&&&rc$URU5$r$)r)rcothers r__eq__ Table.__eq__s{{5!!rc URU$r$r2)rcrTs r __getitem__Table.__getitem__szz!}rc,[UR5$r$)rRr2rs r__len__ Table.__len__s4::rc~URR5RSURR5$Nr)r2__repr__rr?r}rs rrTable.__repr__s,zz""$,,_dnn>U>UVVrc~URR5RSURR5$r)r2__str__rr?r}rs rr Table.__str__s,zz!!#++OT^^=T=TUUrc[5e)$ Compute zero-copy slice of this Table. Args: offset (`int`, defaults to `0`): Offset from start of table to slice. length (`int`, defaults to `None`): Length of slice (default is until end of table starting from offset). Returns: `datasets.table.Table` NotImplementedErrorrs rrq Table.slice "##rc[5eK Select records from a Table. See `pyarrow.compute.filter` for full usage. rrs rfilter Table.filter "##rc[5e0 Flatten this Table. Each column with a struct type is flattened into one column per struct field. Other columns are left unchanged. Args: memory_pool (`MemoryPool`, defaults to `None`): For memory allocations, if required, otherwise use default pool. Returns: `datasets.table.Table` rrs rflatten Table.flatten "##rc[5eaL Make a new table by combining the chunks this table has. All the underlying chunks in the `ChunkedArray` of each column are concatenated into zero or one chunk. Args: memory_pool (`MemoryPool`, defaults to `None`): For memory allocations, if required, otherwise use default pool. Returns: `datasets.table.Table` rrs rcombine_chunksTable.combine_chunksrrc[5e Cast table values to another schema. Args: target_schema (`Schema`): Schema to cast to, the names and order of fields must match. safe (`bool`, defaults to `True`): Check for overflows or other unsafe conversions. Returns: `datasets.table.Table` rrs rcast Table.cast "##rc[5e)a EXPERIMENTAL: Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata Args: metadata (`dict`, defaults to `None`): Returns: `datasets.table.Table`: shallow_copy rrs rreplace_schema_metadataTable.replace_schema_metadatarrc[5e Add column to Table at position. A new table is returned with the column added, the original table object is left unchanged. Args: i (`int`): Index to place the column at. field_ (`Union[str, pyarrow.Field]`): If a string is passed then the type is deduced from the column data. column (`Union[pyarrow.Array, List[pyarrow.Array]]`): Column data. Returns: `datasets.table.Table`: New table with the passed column added. rrs r add_columnTable.add_column  &"##rc[5e)aF Append column at end of columns. Args: field_ (`Union[str, pyarrow.Field]`): If a string is passed then the type is deduced from the column data. column (`Union[pyarrow.Array, List[pyarrow.Array]]`): Column data. Returns: `datasets.table.Table`: New table with the passed column added. rrs r append_columnTable.append_columnrrc[5e)z Create new Table with the indicated column removed. Args: i (`int`): Index of column to remove. Returns: `datasets.table.Table`: New table without the column. rrs r remove_columnTable.remove_column/s "##rc[5e)a| Replace column in Table at position. Args: i (`int`): Index to place the column at. field_ (`Union[str, pyarrow.Field]`): If a string is passed then the type is deduced from the column data. column (`Union[pyarrow.Array, List[pyarrow.Array]]`): Column data. Returns: `datasets.table.Table`: New table with the passed column set. rrs r set_columnTable.set_column<s "##rc[5e: Create new table with columns renamed to provided names. rrs rrename_columnsTable.rename_columnsNrrc[5e)a! Drop one or more columns and return a new table. Args: columns (`List[str]`): List of field names referencing existing columns. Raises: `KeyError` : if any of the passed columns name are not existing. Returns: `datasets.table.Table`: New table without the columns. rrs rdrop Table.dropTrrc[5e)a% Select columns of the table. Returns a new table with the specified columns, and metadata preserved. Args: columns (:obj:`Union[List[str], List[int]]`): The column names or integer indices to select. Returns: `datasets.table.Table`: table with only a subset of the columns rrs rselect Table.selectdrrrr$)3r}r~rrrr%rrfdictrrrr]rrrrr rrrrrpropertyr9rrrrrrrrrrrrqrrrrrrrr rrrrr __classcell__r?s@rrrsV bhh%%4 2" 655<5|5Ax}A 1 27!!""&& # #  !! '' "WV$ $ $$ $ $$*$ $$$$ $ $ $rrc\rSrSrSrSrg) TableBlockitz `TableBlock` is the allowed class inside a `ConcanetationTable`. Only `MemoryMappedTable` and `InMemoryTable` are `TableBlock`. This is because we don't want a `ConcanetationTable` made out of other `ConcanetationTables`. rN)r}r~rrrrrrrr r ts   rr c\rSrSrSr\S\4Sj5r\S\R4Sj5r \S5r \S5r \S 5r \S 5r\S 5rSS jrSrSrSrSrSrSrSrSrSrSrSrSrSrg ) InMemoryTablei~a; The table is said in-memory when it is loaded into the user's RAM. Pickling it does copy all the data using memory. Its implementation is simple and uses the underlying pyarrow Table methods directly. This is different from the `MemoryMapped` table, for which pickling doesn't copy all the data in memory. For a `MemoryMapped`, unpickling instead reloads the table from the disk. `InMemoryTable` must be used when data fit in memory, while `MemoryMapped` are reserved for data bigger than memory or when you want the memory footprint of your application to stay low. r!c([U5nU"U5$r$)r-)rHr!r2s r from_fileInMemoryTable.from_files0:5zrr.c([U5nU"U5$r$)r3)rHr.r2s r from_bufferInMemoryTable.from_buffers26:5zrcNU"[RR"U0UD65$)a Convert pandas.DataFrame to an Arrow Table. The column types in the resulting Arrow Table are inferred from the dtypes of the pandas.Series in the DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the case of `object`, we need to guess the datatype by looking at the Python objects in this Series. Be aware that Series of the `object` dtype don't carry enough information to always lead to a meaningful Arrow type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only contains `None/nan` objects, the type is set to null. This behavior can be avoided by constructing an explicit schema and passing it to this function. Args: df (`pandas.DataFrame`): schema (`pyarrow.Schema`, *optional*): The expected schema of the Arrow Table. This can be used to indicate the type of columns if we cannot infer it automatically. If passed, the output will have exactly this schema. Columns specified in the schema that are not found in the DataFrame columns or its index will raise an error. Additional columns or index levels in the DataFrame which are not specified in the schema will be ignored. preserve_index (`bool`, *optional*): Whether to store the index as an additional column in the resulting `Table`. The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. Use `preserve_index=True` to force it to be stored as a column. nthreads (`int`, defaults to `None` (may use up to system CPU count threads)) If greater than 1, convert columns to Arrow in parallel using indicated number of threads. columns (`List[str]`, *optional*): List of column to be converted. If `None`, use all columns. safe (`bool`, defaults to `True`): Check for overflows or other unsafe conversions, Returns: `datasets.table.Table`: Examples: ```python >>> import pandas as pd >>> import pyarrow as pa >>> df = pd.DataFrame({ ... 'int': [1, 2], ... 'str': ['a', 'b'] ... }) >>> pa.Table.from_pandas(df) ``` )r%r from_pandasrHrrs rr*InMemoryTable.from_pandass$p288''8899rcNU"[RR"U0UD65$)a$ Construct a Table from Arrow arrays. Args: arrays (`List[Union[pyarrow.Array, pyarrow.ChunkedArray]]`): Equal-length arrays that should form the table. names (`List[str]`, *optional*): Names for the table columns. If not passed, schema must be passed. schema (`Schema`, defaults to `None`): Schema for the created table. If not passed, names must be passed. metadata (`Union[dict, Mapping]`, defaults to `None`): Optional metadata for the schema (if inferred). Returns: `datasets.table.Table` )r%r from_arraysr+s rr.InMemoryTable.from_arrayss#$288''8899rcNU"[RR"U0UD65$)a Construct a Table from Arrow arrays or columns. Args: mapping (`Union[dict, Mapping]`): A mapping of strings to Arrays or Python lists. schema (`Schema`, defaults to `None`): If not passed, will be inferred from the Mapping values metadata (`Union[dict, Mapping]`, defaults to `None`): Optional metadata for the schema (if inferred). Returns: `datasets.table.Table` )r%r from_pydictr+s rr1InMemoryTable.from_pydicts# 288''8899rcVU"[RR"U/UQ70UD65$)a Construct a Table from list of rows / dictionaries. Args: mapping (`List[dict]`): A mapping of strings to row values. schema (`Schema`, defaults to `None`): If not passed, will be inferred from the Mapping values metadata (`Union[dict, Mapping]`, defaults to `None`): Optional metadata for the schema (if inferred). Returns: `datasets.table.Table` )r%r from_pylist)rHmappingrrs rr4InMemoryTable.from_pylists( 288''A$A&ABBrcNU"[RR"U0UD65$)a Construct a Table from a sequence or iterator of Arrow `RecordBatches`. Args: batches (`Union[Sequence[pyarrow.RecordBatch], Iterator[pyarrow.RecordBatch]]`): Sequence of `RecordBatch` to be converted, all schemas must be equal. schema (`Schema`, defaults to `None`): If not passed, will be inferred from the first `RecordBatch`. Returns: `datasets.table.Table`: )r%rror+s rroInMemoryTable.from_batches s#288(($9&9::rNc2[URXS95$)rrwrx)r"rz)rcrwrxs rrqInMemoryTable.slicesT__F_JKKrcL[URR"U0UD65$r)r"r2rrs rrInMemoryTable.filter*s#TZZ..??@@rcF[[UR/UQ70UD65$r)r" table_flattenr2rs rrInMemoryTable.flatten0s"]4::GGGHHrcL[URR"U0UD65$r)r"r2rrs rrInMemoryTable.combine_chunks>s#TZZ66GGHHrcF[[UR/UQ70UD65$r)r" table_castr2rs rrInMemoryTable.castNs"Z DTDVDEErcL[URR"U0UD65$a EXPERIMENTAL: Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be `None`, which deletes any existing metadata). Args: metadata (`dict`, defaults to `None`): Returns: `datasets.table.Table`: shallow_copy )r"r2rrs rr%InMemoryTable.replace_schema_metadata]s#TZZ??PPQQrcL[URR"U0UD65$r)r"r2rrs rrInMemoryTable.add_columnks#&TZZ22DCFCDDrcL[URR"U0UD65$M Append column at end of columns. Args: field_ (`Union[str, pyarrow.Field]`): If a string is passed then the type is deduced from the column data. column (`Union[pyarrow.Array, List[pyarrow.Array]]`): Column data. Returns: `datasets.table.Table`: New table with the passed column added. )r"r2rrs rrInMemoryTable.append_columns#TZZ55tFvFGGrcL[URR"U0UD65$ Create new Table with the indicated column removed. Args: i (`int`): Index of column to remove. Returns: `datasets.table.Table`: New table without the column. )r"r2r rs rr InMemoryTable.remove_columns#TZZ55tFvFGGrcL[URR"U0UD65$ Replace column in Table at position. Args: i (`int`): Index to place the column at. field_ (`Union[str, pyarrow.Field]`): If a string is passed then the type is deduced from the column data. column (`Union[pyarrow.Array, List[pyarrow.Array]]`): Column data. Returns: `datasets.table.Table`: New table with the passed column set. )r"r2rrs rrInMemoryTable.set_columns#"TZZ22DCFCDDrcL[URR"U0UD65$r)r"r2rrs rrInMemoryTable.rename_columnss#TZZ66GGHHrcL[URR"U0UD65$) Drop one or more columns and return a new table. Args: columns (`List[str]`): List of field names referencing existing columns. Raises: `KeyError` : if any of the passed columns name are not existing. Returns: `datasets.table.Table`: New table without the columns. )r"r2rrs rrInMemoryTable.drops!TZZ__d=f=>>rcL[URR"U0UD65$B Select columns of the table. Returns a new table with the specified columns, and metadata preserved. Args: columns (:obj:`Union[List[str], List[int]]`): The column names or integer indices to select. Returns: :class:`datasets.table.Table`: New table with the specified columns, and metadata preserved. )r"r2rrs rrInMemoryTable.selects#TZZ..??@@rrr|)r}r~rrr classmethodstrr$r%Bufferr'r*r.r1r4rorqrrrrrrrr rrrrrrrrr"r"~s 7:7:r::&::"CC" ; ;L"A II F RE*H" HE&I ?" Arr"c J^\rSrSrSrSS\R S\S\\ \ 4U4Sjjjr \ SS\4Sjj5r S rS r\SS\R S\\ \ S \R 4S jj5rS \ S \ \ 4SjrSSjrSrSrSrSrSrSrSrSrSrSrSrSrSr U=r!$)MemoryMappedTableia The table is said memory mapped when it doesn't use the user's RAM but loads the data from the disk instead. Pickling it doesn't copy the data into memory. Instead, only the path to the memory mapped arrow file is pickled, as well as the list of transforms to "replay" when reloading the table from the disk. Its implementation requires to store an history of all the transforms that were applied to the underlying pyarrow Table, so that they can be "replayed" when reloading the Table from the disk. This is different from the `InMemoryTable` table, for which pickling does copy all the data in memory. `InMemoryTable` must be used when data fit in memory, while `MemoryMapped` are reserved for data bigger than memory or when you want the memory footprint of your application to stay low. r2pathreplaysc>[TU]U5 [RR U5UlUbX0lg/Ulgr$)rrfosrfabspathrg)rcr2rfrgr?s rrfMemoryMappedTable.__init__s4 GGOOD) 070CW  rr!cL[U5nURX25nU"X1U5$r$)r<_apply_replays)rHr!rgr2s rr$MemoryMappedTable.from_files)4X>""525G,,rc4URURS.$)Nrfrgrprs r __getstate__MemoryMappedTable.__getstate__s dll;;rcxUSnUSn[U5nURXC5n[RXX#S9 g)Nrfrgrp)r<rmrerf)rcstaterfrgr2s r __setstate__MemoryMappedTable.__setstate__sBV} "4T:##E3""4T"Krr"cUbKUHEup#nUS:Xa[U/UQ70UD6nMUS:Xa[U/UQ70UD6nM4[X5"U0UD6nMG U$)Nrr)rDr?getattr)r2rgnamerrs rrm MemoryMappedTable._apply_replayssi  &-"F6>&u>t>v>EY&)%A$A&AE#E0$A&AE '. rreplaych[R"UR5nURU5 U$r$)rErFrgappend)rcr{rgs r_append_replay MemoryMappedTable._append_replays%-- -vrcxSX404nURU5n[URXS9URU5$)rrqr:)r~rerzrf)rcrwrxr{rgs rrqMemoryMappedTable.slicesEF+R0%%f- !NPTPYPY[bccrcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)rr)rErFr~rer2rrfrcrrr{rgs rrMemoryMappedTable.filter+sZDMM$/v1FG%%f- !2!2D!CF!CTYYPWXXrcS[R"U5[R"U54nURU5n[[ UR /UQ70UD6UR U5$)rr)rErFr~rer?r2rfrs rrMemoryMappedTable.flatten3sYT]]40$--2GH%%f- tzz!KD!KF!KTYYX_``rcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)aJ Make a new table by combining the chunks this table has. All the underlying chunks in the ChunkedArray of each column are concatenated into zero or one chunk. Args: memory_pool (`MemoryPool`, defaults to `None`): For memory allocations, if required, otherwise use default pool. Returns: `datasets.table.Table` r)rErFr~rer2rrfrs rr MemoryMappedTable.combine_chunksCsZ#DMM$$7v9NO%%f- !:!:D!KF!KTYYX_``rcS[R"U5[R"U54nURU5n[[ UR /UQ70UD6UR U5$)a Cast table values to another schema Args: target_schema (`Schema`): Schema to cast to, the names and order of fields must match. safe (`bool`, defaults to `True`): Check for overflows or other unsafe conversions. Returns: `datasets.table.Table` r)rErFr~rerDr2rfrs rrMemoryMappedTable.castUsY$---t}}V/DE%%f- DJJ!H!H!H$))U\]]rcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)a EXPERIMENTAL: Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata. Args: metadata (`dict`, defaults to `None`): Returns: `datasets.table.Table`: shallow_copy r)rErFr~rer2rrfrs rr)MemoryMappedTable.replace_schema_metadatafs^,T]]4-@$--PVBWX%%f- !C!CT!TV!TVZV_V_ahiircS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)rr)rErFr~rer2rrfrs rrMemoryMappedTable.add_columnvsZ& d 3T]]65JK%%f- !6!6!G!GT[\\rcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)rMr)rErFr~rer2rrfrs rrMemoryMappedTable.append_columnsZ"4==#6 f8MN%%f- !9!94!J6!JDIIW^__rcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)rQr )rErFr~rer2r rfrs rr MemoryMappedTable.remove_columnsZ"4==#6 f8MN%%f- !9!94!J6!JDIIW^__rcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)rUr)rErFr~rer2rrfrs rrMemoryMappedTable.set_columnsZ" d 3T]]65JK%%f- !6!6!G!GT[\\rcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)rr)rErFr~rer2rrfrs rr MemoryMappedTable.rename_columnssZ#DMM$$7v9NO%%f- !:!:D!KF!KTYYX_``rcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)r[r)rErFr~rer2rrfrs rrMemoryMappedTable.dropsW$---t}}V/DE%%f- $!A&!A499gVVrcS[R"U5[R"U54nURU5n[URR "U0UD6UR U5$)r_r)rErFr~rer2rrfrs rrMemoryMappedTable.selectsZDMM$/v1FG%%f- !2!2D!CF!CTYYPWXXrrpr$r|)"r}r~rrrr%rrbr rReplayrfrar$rqru staticmethodrmr~rqrrrrrrrr rrrrrrrs@rreres(LbhhLcLHT&\[TU]U5 X lUH9nUH0n[U[5(aM[ S[ U5S35e M; g)Nz_The blocks of a ConcatenationTable must be InMemoryTable or MemoryMappedTable objects, but got rQ)rrfrrr  TypeError _short_str)rcr2r subtablessubtabler?s rrfConcatenationTable.__init__s\   I%!(J77#%%/%9$:!=& rcHURURRS.$)N)rr9)rr2r9rs rrqConcatenationTable.__getstate__ s++1B1BCCrcUSnUSnURU5nUbDURU:wa4[RR /US9n[R "XE/SS9n[ RXUS9 g)Nrr9rldefaultpromote_optionsr)*_concat_blocks_horizontally_and_verticallyr9r%rro concat_tablesrrf)rcrtrr9r2 empty_tables rruConcatenationTable.__setstate__#svxx??G  %,,&"8((//6/BK$$e%99UE##D#?raxisr"cvUVs/sH"n[US5(a UROUPM$ nnUS:Xa[R"USS9$US:XaZ[ U5HIupBUS:XaUnM[ UR UR5HupgWRXg5nM MK W$[S5es snf)Nr2rrrr z'axis' must be either 0 or 1) rr2r%r enumeraterprrrrm)rrr2 pa_tablesrTr,rycols r_concat_blocks!ConcatenationTable._concat_blocks.sTZ[TZ5GE7$;$;U[[FTZ [ 19##IyI I QY%i06$H%(););U]]%K #+#9#9$#D&L 1 O;< <\s)B6c/n[U5H/up4U(dMURUSS9nURU5 M1 URUSS9$)Nr rr)rrr})rHrpa_tables_to_concat_verticallyrTtables"pa_table_horizontally_concatenateds rr=ConcatenationTable._concat_blocks_horizontally_and_vertically?s^)+&"6*IA141C1CFQR1C1S . * 1 12T U + !!"@q!IIrcUbP/n[USS9H=upEU(a#[UR[U5US95/nU[U5- nM? U$UVs/sHo`R USS9PM nn[ SU55(a+UR UVVs/sH ofHowPM M snnSS9nU$s snfs snnf)Nc"[U[5$r$)rr")rGs r2ConcatenationTable._merge_blocks..Ms :VWYfKgr)keyrr c3># UHn[U5S:Hv M g7f)r N)rR)r row_blocks rr3ConcatenationTable._merge_blocks..SsF 93y>Q& sr)rr"rr _merge_blocksall)rHrr merged_blocks is_in_memory block_grouprblocks rr ConcatenationTable._merge_blocksIs  M-4VAg-h) #01C1CDDU\`1C1a#b"cKk!22 .i TZZSYi..yq.ASYMZF FFF # 1 1,9QMyyeUyUMQXY!2!  [Rs B7B< c[U[5(aU$[US[5(aURUSS9$URU5$Nrr)rr r)rHrs r_consolidate_blocks&ConcatenationTable._consolidate_blocksYsL fj ) )M q : . .$$V!$4 4$$V, ,rc<URU5n[U[5(aUnU"URU//5$[US[5(a*UR USS9nUVs/sHo3/PM nnU"X!5$UR U5nU"X!5$s snfr)rrr r2rr)rHrr2ts r from_blocksConcatenationTable.from_blocksbs((0 fj ) )Eu{{eWI. . q : . .&&vA&6E#)*6ac6F*u% %BB6JEu% % +s* Brc ^^ S[[R[4S[[[4SjnS[[S[ S[ [[[[44SjmS[[[S[[[S[ [[[[[[44U4S jjm SS[[[S[[[S [ S[[[4U 4S jjjnU"US 5nUS SHnU"U5nU"XWUS9nM URU5$)a`Create `ConcatenationTable` from list of tables. Args: tables (list of `Table` or list of `pyarrow.Table`): List of tables. axis (`{0, 1}`, defaults to `0`, meaning over rows): Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns (horizontally). r2r"c[U[R5(a [U5//$[U[5(a [ R "UR5$U//$r$)rr%rr"rrErFrrs r to_blocks1ConcatenationTable.from_tables..to_blocks~sO%**&u-.//E#566}}U\\22y rrrxc UVs/sHo"RSU5PM nnUVs/sH#o"RU[US5U- 5PM% nnX44$s snfs snfNr)rqrR)rrxr2sliced remainders r_slice_row_block8ConcatenationTable.from_tables.._slice_row_blocks^:CD)kk!V,)FDV_`V_UVS1->-GHV_I`$ $E`s A*ArIrc>[U5[U5p//p2U(GaEU(Ga=[USS5[USS5:aRURUS5 T"US[URS5S55uo@S'URU5 O[USS5[USS5:aRURUS5 T"US[URS5S55uoAS'URU5 O@URURS55 URURS55 U(a U(aGM=U(dU(a [ S5eX#4$)a Make sure each row_block contain the same num_rows to be able to concatenate them on axis=1. To do so, we modify both blocks sets to have the same row_blocks boundaries. For example, if `result` has 2 row_blocks of 3 rows and `blocks` has 3 row_blocks of 2 rows, we modify both to have 4 row_blocks of size 2, 1, 1 and 2: [ x x x | x x x ] + [ y y | y y | y y ] ----------------------------- = [ x x | x | x | x x ] [ y y | y | y | y y ] rzQFailed to concatenate on axis=1 because tables don't have the same number of rows)rrRr}poprm)rIr new_result new_blocksrrs r_split_both_like8ConcatenationTable.from_tables.._split_both_likesM""&\4US:XaURU5 U$US:Xa1T"X5up[U5Hup4XRU5 M U$)Nrr )extendr)rIrrrTrrs r_extend_blocks6ConcatenationTable.from_tables.._extend_blockssZqy f% M !1&!A$-f$5LAI$$Y/%6Mrr Nrr)r r%rrr rrr) rHrrrrrr2 table_blocksrrs @@r from_tablesConcatenationTable.from_tablespsJ !U288U?3 !T*=M8N ! %Z(8 %# %%PTU_P`bfgqbrPrJs % # *j)*# *48j9I4J# * 4Z()4Z0@+AA B# *LYZ j)* 48j9I4J RU  $z" #  6!9%ABZE$U+L#FtDF v&&rc#d# SnURHn[US5nX4v X- nM g7fr)rrR)rcrwrrxs r_slicesConcatenationTable._slicess6kkF^F" "  F"s.0c URRXS9nUbUOURU- n/nURHn[ US5nUS:Xa OXa::aX- nM$XaU-::a:UR UVs/sHowRU5PM sn5 X!-U- SpMfUR UVs/sHowRX5PM sn5 Sup!M [ X45$s snfs snf)r)rxr)rr)r2rqrrrRr}r)rcrwrxr2rrn_rowsrs rrqConcatenationTable.slices    7!-4==63IkkF^F{!F?* ?1wwv?@!'6!91 G1wwv6GH!%""%00 @Hs 7C% 1C* c JURR"U/UQ70UD6n/n[URUR5HNuupgnUR Xg5n UR UV s/sHoR"U /UQ70UD6PM sn 5 MP [XE5$s sn fr)r2rrprrrqr}r) rcmaskrrr2rrwrxrsubmaskrs rrConcatenationTable.filters !!$888(+DLL$++(F $ Vfjj0G MMvNv!88G=d=f=vN O)G"%00Os+B c [UR/UQ70UD6n/nURH4nURUVs/sHofR"U0UD6PM sn5 M6 [ X45$s snfr)r?r2rr}rrrcrrr2rrrs rrConcatenationTable.flattensgdjj:4:6:kkF MMvFv!99d5f5vF G"!%00GsA+ c URR"U0UD6n/nURH4nURUVs/sHofR"U0UD6PM sn5 M6 [ X45$s snfr)r2rrr}rrs rr!ConcatenationTable.combine_chunkssj ))4:6:kkF MMfMf++T# UHupURT:XdMUv M g7fr$)ry)rrTrrys rr*ConcatenationTable.cast..2s&4oGX81\a\f\fjn\nQQGXs& &)featuresrrDr2from_arrow_schemarrrr}rnextrry arrow_schemarr)rc target_schemarrrr2target_featuresrr new_tablesfieldsr subfieldssubfield subfeatures subschemarys @rrConcatenationTable.casts '4::}FtFvF"44]CIJ-(F% $11D$$VZZ4oyQWGX4o0o%pq2&fo'pfoZb }}7U(Ufo'pq '44 !!(-- "KD"KF"KL & MM* %%"%00 (qs."D, c URR"U0UD6n/nURH4nURUVs/sHofR"U0UD6PM sn5 M6 [ X0R5$s snfrG)r2rrr}rrs rr*ConcatenationTable.replace_schema_metadata9sn 22DCFCkkF MMvVv!44dEfEvV W"!%55WsA8 c[5errrs rrConcatenationTable.add_columnKrrc[5erLrrs rr ConcatenationTable.append_column`s "##rc ~URR"U/UQ70UD6nURRUn/nURHcnUR UVs/sHCnXXR;a/UR"URR U5/UQ70UD6OUPME sn5 Me [ XF5$s snfrP)r2r rrr}indexr) rcrTrrr2ryrrrs rr  ConcatenationTable.remove_columnqs ((PPPpqq# ""%00 sA B: c[5erTrrs rrConcatenationTable.set_columns ""##rcURR"U/UQ70UD6n[[URRU55n/nUR HWnUR UVVs/sH5owR"URVs/sHoUPM sn/UQ70UD6PM7 snn5 MY [XE5$s snfs snnfr)r2rrrprrr}r) rcnamesrrr2rrrrys rr!ConcatenationTable.rename_columnss ))%A$A&AS00%89kkF MMflmflab!!1>>"J>4;>"J\T\U[\flm ""%00#Kms+C B>C >C cJURR"U/UQ70UD6n/nURH\nURUVVs/sH:owR"UVs/sHoUR;dMUPM sn/UQ70UD6PM< snn5 M^ [ XE5$s snfs snnfrZ)r2rrr}rr rcrrrr2rrrcs rrConcatenationTable.drops 9$9&9kkF MMiopiode66g"Mgann9L1g"M_PT_X^_iop q"!%00#NpB B.B4B B cJURR"U/UQ70UD6n/nURH\nURUVVs/sH:owR"UVs/sHoUR;dMUPM sn/UQ70UD6PM< snn5 M^ [ XE5$s snfs snnfr^)r2rrr}rrrs rrConcatenationTable.selects !!';D;F;kkF MMkqrkqfg88$O1;NQ$OaRVaZ`akqr s"!%00%Prr rrr$r|))r}r~rrrr%rrr rfrqrurr rrrarrr rrrrrrrqrrrrrrrr rrrrrrrs@rrrs2 bhh T*5E0F D @=tE*bhh*>$?@==TVT\T\== JT*EU@VJ[][c[cJJ #6 hsm Wj  -)<-AT-- &!4 &9M & &O'eBHHeO& 11$1(1>6$$*$"10$& 11*11rrrrcf[U5n[U5S:XaUS$[RXS9$)a Concatenate tables. Args: tables (list of `Table`): List of tables to be concatenated. axis (`{0, 1}`, defaults to `0`, meaning over rows): Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns (horizontally). Returns: `datasets.table.Table`: If the number of input tables is > 1, then the returned table is a `datasets.table.ConcatenationTable`. Otherwise if there's only one table, it is returned as is. r rr)rrRrr)rrs rrrs7"&\F 6{aay  ) )& ) <[U[R5(a;[R"URVs/sHnT"U/UQ70UD6PM sn5$T"U/UQ70UD6$s snfr$)rr% ChunkedArray chunked_arraychunks)arrayrrchunkfuncs rr)_wrap_for_chunked_arrays..wrappersb eR__ - -##u||$\|eT%%A$%A&%A|$\] ]/// /%]sA(r)r.rs` r_wrap_for_chunked_arraysr0s0 Nrr,rxc[R"[R"UR5U55R 5=(d UR [ U5:H$)zICheck if all the sub-lists of a `pa.ListArray` have the specified length.)pcrequal value_lengthsas_py null_countrR)r,rxs r_are_list_values_of_lengthr7 sE 66"((5..0&9 : @ @ B deFVFVZ]^cZdFddrc $URnURS:as[R"[R "USSUR 5[R"[U5[R"555USS/5nU$)z7Add the null bitmap to the offsets of a `pa.ListArray`.rNrO) offsetsr6r% concat_arraysr2replace_with_maskis_nullnullsrRint32)r,r9s r%_combine_list_array_offsets_with_maskr?svmmG !""$$WSb\5==?BHHSQVZY[YaYaYcDde    Nrtypec [U[R5(a[UR5$[U[R 5(aY[R "UVs/sH7n[R"UR[UR55PM9 sn5$[U[R5(a)[R"[UR55$[U[R5(a4[R"[UR5UR5$U$s snf)zCConvert a (possibly nested) `pa.ExtensionType` to its storage type.)rr% ExtensionType _storage_type storage_type StructTypestructrryr@ListTypelist_ value_typeFixedSizeListType list_size)r@rs rrCrCs$(())T..// D"-- ( (yyW[\W[e"((5::}UZZ/HIW[\]] D"++ & &xx doo677 D".. / /xx doo6GG K ]s(>E valuecV[U5n[U5S:aUSSS-USS-nU$)Ni iz ... i$)rbrR)rLouts rrr)s6 e*C 3x$%4j9$s56{2 Jrpa_typeallow_primitive_to_strallow_decimal_to_strc j[[X#S9n[U[R5(a UR n[U[R 5(a!URU"XR55$URU:XaU$[RRUR5(a[RRU5(aUVs1sHoURiM snURVs1sHoURiM sn:XaURRS:XaU$UVs/sH.oT"URUR5UR5PM0 nn[RR!U[#U5UR%5S9$GO[RR'UR5(d/[RR)UR5(Gak[RR+U5(Gar[-XR.5(GaUUR0S:aURn[3U5nXx:wa2U"X5n[4R6"USUR.SS9nU"X5nO![4R6"USUR.SS9nUR8n [R:R!U"XR<5UR.UR%5S9$UR8UR>UR.-UR>[AU5-UR.-n [R:R!U"XR<5UR.5$GO[RR'U5(aE[CU5n [RDR!X"UR8UR<55$[RR)U5(aE[CU5n [RFR!X"UR8UR<55$GO[RR+UR5(GaR[RR+U5(aUR.URR.:XaUR8UR>URR.-UR>[AU5-URR.-n [R:R!U"XR<5UR.UR%5S9$GO[RR'U5(a[HRJ"[AU5S-5UR>-URR.-n [RDR!X"UR8UR<5UR%5S9$[RR)U5(a[HRJ"[AU5S-5UR>-URR.-n [RFR!X"UR8UR<5UR%5S9$GO[RRMU5(aU(d_[RROUR5(a1[QS[SUR5S [SU5S US 35eU(d^[RRUUR5(a0[QS[SUR5S [SU5S U35e[RR%U5(a[[RR%UR5(d-[QS[SUR5S [SU535eURWU5$[QS[SUR5S [SU535es snfs snfs snf) a/Improved version of `pa.Array.cast` It supports casting `pa.StructArray` objects to re-order the fields. It also let you control certain aspects of the casting, e.g. whether to disable casting primitives (`booleans`, `floats` or `ints`) or disable casting decimals to strings. Args: array (`pa.Array`): PyArrow array to cast pa_type (`pa.DataType`): Target PyArrow type allow_primitive_to_str (`bool`, defaults to `True`): Whether to allow casting primitives to strings. Defaults to `True`. allow_decimal_to_str (`bool`, defaults to `True`): Whether to allow casting decimals to strings. Defaults to `True`. Raises: `pa.ArrowInvalidError`: if the arrow data casting fails `TypeError`: if the target type is not supported according, e.g. - if a field is missing - if casting from primitives to strings and `allow_primitive_to_str` is `False` - if casting from decimals to strings and `allow_decimal_to_str` is `False` Returns: `List[pyarrow.Array]`: the casted array rPrQr)rrTreturn_fixed_size_listrr zCouldn't cast array of type z to z( since allow_primitive_to_str is set to  z$ and allow_decimal_to_str is set to ),r array_castrr%ExtensionArraystoragerB wrap_arrayrDr@types is_structry num_fieldsr StructArrayr.rr<is_list is_large_listis_fixed_size_listr7rKr6rCr2 list_slicevaluesFixedSizeListArrayrIrwrRr? ListArrayLargeListArrayr_arange is_string is_primitiverr is_decimalr) r,rOrPrQ_crarrays array_typerD array_values array_offsetss rrXrX0sD 4J vB%**++ '2++,,!!"U,@,@"ABB w    EJJ ' ' 88  g & &W,MWEZZW,MjojtjtQujtafR\R\jtQu,uzz$$) KRS7%bUZZ0%**=7FS>>--fT']QVQ^Q^Q`-a a   %** % %)?)? )K)K 88 & &w / /)%1B1BCC##a'!&J#0#O>OV[VcVcVe=$)<< w'8'88ELL3u:>T=UUVX(BHH,?,? ,K,K2:ejj3I2J$zZaObNcd::N9OQ 88  G $ $RXX-=-=ejj-I-I::ejj;Q:RRVWabiWjVklm mzz'"" 2:ejj3I2J$zZaObNcd eeI-NQuTs"b&b+5b0featurerc JSSKJnJnJn [ [ UUS9n[ U[R5(a URn[US5(aURU5$[RRUR5(a[ U[5(aURVs1sHoR iM sn=n [#U5::a[R$"S/['U5-5n UR)5V V s/sH$upU"X;aUR+U 5OU U 5PM& n n n [R,R/U [1U5UR35S9$GO[RR5UR5(d/[RR7UR5(Ga[ X5(aU"UR8UR:5n[RR7UR5(a&URUR8R:XaU$[=U5n[R>R/X5$[ X5(Ga*UR@S:Ga~[CXR@5(GaaURDS:aURn[GU5nUU:wa:[IUUUUS9n[JRL"USUR@S S 9n[IUUUUS9nO![JRL"USUR@S S 9nUR8nU"UUR:5n[RNR/XR@UR35S 9$UR8URPUR@-URP['U5-UR@-n[RNR/U"UUR:5UR@5$GOU"UR8UR:5n[RR5UR5(a&URUR8R:XaU$[=U5n[RRR/X5$GOE[RRUUR5(Ga[ X5(a[VRX"['U5S-5URP-URRZ-n[R>R/X"UR8UR:5UR35S 9$[ X5(GahUR@S:aUR@URRZ:XaUR8URPURRZ-URP['U5-URRZ-nU"UUR:5n[RNR/XR@UR35S 9$O[VRX"['U5S-5URP-URRZ-n[RRR/X"UR8UR:5UR35S 9$[RR3UR5(a[IUU"U5UUS9$[ XU[45(d[IUU"5UUS9$[]S [_UR5S [_U535es snfs sn n f)a Cast an array to the arrow type that corresponds to the requested feature type. For custom features like [`Audio`] or [`Image`], it takes into account the "cast_storage" methods they defined to enable casting from other arrow types. Args: array (`pa.Array`): The PyArrow array to cast. feature (`datasets.features.FeatureType`): The target feature type. allow_primitive_to_str (`bool`, defaults to `True`): Whether to allow casting primitives to strings. Defaults to `True`. allow_decimal_to_str (`bool`, defaults to `True`): Whether to allow casting decimals to strings. Defaults to `True`. Raises: `pa.ArrowInvalidError`: if the arrow data casting fails `TypeError`: if the target type is not supported according, e.g. - if a field is missing - if casting from primitives and `allow_primitive_to_str` is `False` - if casting from decimals and `allow_decimal_to_str` is `False` Returns: array (`pyarrow.Array`): the casted array r ) LargeListListget_nested_typerS cast_storageNrrrOrTrTrVzCouldn't cast array of type  to )0features.featuresrsrtrurcast_array_to_featurerr%rYrZrrvr\r]r@rrysetr,rRrCrr_r.rr<r`rardrqr?rgrxr7r6rCrXr2rcrerwrfrbr_rhrKrr)r,rqrPrQrsrtrurlr array_fields null_arrayry subfeaturermcasted_array_valuesrprnrDros rrzrzs>DC 51 B %**++ w''##E** xx%**%% gt $ $SXS]S]:^S]%::S]:^*^,cfgnco)o4&3u:"56J)0 (7$D(<5;;t$*jY(7 >>--fDMPUP]P]P_-` `   %** % %)?)? )K)K g ) )"$U\\7??"C xx%%ejj116I6N6NRWR^R^RcRc6c !Fe L ((44]XX  & &~~"-e^^DD''!+%*ZZ '4Z'@ %5$. % ,7M5I %E %'MM%GNNcg$hE$. % *7M5I %E%'MM%GNNcg$hE',|| .0w.O+!44@@/emmo A (-||!LL7>>9U\\CPUJ=VZaZhZh>UZZ%9%99#(<< uzz';';;u||cRWj?X\a\f\f\p\p>p$L+-\7??*K'00<<=PR`R`glgtgtgv<ww :"$3u:>!:U\\!IUZZMaMa a ||// r%,,PWP_P_?`glgtgtgv/ww xx ##  G $#9!5    4!8 9 9  I#9!5   3Juzz4J3K6R\]dReQfg hh{;_s 8^+^c SSKJnJn [[US9n[ U[ R5(a URn[US5(aURXS9$[ RRUR5(a[ U[5(apUR5VVs/sHupgU"UR!U5U5PM nnn[ R"R%U['U5UR)5S9$GO[ RR+UR5(ag[-U5n [ X5(aJUR.S:Xa:[ R0R%X"UR2UR455$GOf[ RR7UR5(aE[-U5n [ R8R%X"UR2UR455$[ RR;UR5(a[ X5(aUR.S:aUR2UR<URR>-UR<[AU5-URR>-n U"XR45n [ RBR%XR.UR)5S9$[ XU[45(dU$[ES[GUR5S [GU535es snnf) a!Embed data into an arrays's storage. For custom features like Audio or Image, it takes into account the "embed_storage" methods they define to embed external data (e.g. an image file) into an array. Args: array (`pa.Array`): The PyArrow array in which to embed data. feature (`datasets.features.FeatureType`): Array features. Raises: `TypeError`: if the target type is not supported according, e.g. - if a field is missing Returns: array (`pyarrow.Array`): the casted array r )rsrttoken_per_repo_id embed_storagerwrOrVzCouldn't embed array of type z with )$rrsrtrembed_array_storagerr%rYrZrrr\r]r@rrCrr_r.rr<r`r?rxrfrdrqrargrbrwrKrRrerr) r,rqrrsrt_eryr~rmrproembedded_array_valuess rrr/s,* $8I JB%**++ w(($$U$PP   EJJ ' ' gt $ $PWP]P]P_`P_>--fDMPUP]P]P_-` ` %   %** % %>eD g $ $2)=<<++M2ellGOO;\] ]    + +>eD   ,,]Bu||W__<]^^  $ $UZZ 0 0 g $ $")< << uzz333u||c%j7PTYT^T^ThTh6hL%'|__$E !((445JNNafananap4q q gi6 7 7 4Z 5K4LHU_`gUhTij kk/as8$McT^\rSrSrSrS\\S\\SS4U4SjjrSrS r S r U=r $) CastErrorijzUWhen it's not possible to cast an Arrow table to a specific schema or set of featurestable_column_namesrequested_column_namesr"Nc8>[TU]"U6 XlX lgr$)rrfrr)rcrrrr?s rrfCastError.__init__ms $"4&<#rcL[[URURS9S4$)Nrrr)rrrrrs r __reduce__CastError.__reduce__rs+ $*A*AZ^ZuZu  rc [UR5[UR5- n[UR5[UR5- nU(a:U(a3S[U5S[ U5S[U5S[ U5S3 $U(aS[U5S[ U5S3$S[U5S[ U5S3$)Nz there are z new columns (z) and z missing columns (z).))r{rrrRr)rc new_columnsmissing_columnss rdetailsCastError.detailsxs$112S9T9T5UU d99:SAXAX=YY ?K 01 ;@W?XX^_bcr_s^tuGHRSbHcGddfg g K 01 ;@W?XXYZ ZO 455G SbHcGddef fr)rr) r}r~rrrrrbrfrrrrrs@rrrjs=_=$s)=UYZ]U^=cg=  ggrrrrc[UR5[U5:waA[S[UR5S[U5S3UR[ U5S9eUR 5VVs/sHup#[XU5PM nnn[RRXARS9$s snnf)zCast a table to the arrow schema that corresponds to the requested features. Args: table (`pyarrow.Table`): PyArrow table to cast. features ([`Features`]): Target features. Returns: table (`pyarrow.Table`): the casted table Couldn't cast rx! because column names don't matchrrl) sortedrrrr9rrCrzr%rr.r)r2rryrqrms rcast_table_to_featuresrse  !VH%55j67vj>R=SSu v$11#'>  PX~~O_ `O_md#EK9O_F ` 88  /D/D  EEas7B;r9c 6SSKJn UR"U5n[UR5nU[UR 5::dA[ S[UR5S[U5S3UR[U5S9eUR5VVs/sHSupV[XT;aXO:[R"S/[U5-URU5R S9U5PMU nnn[R"R%XqS 9$s snnf) aCast a table to the arrow schema. Different from `cast_table_to_features`, this method can preserve nullability. Args: table (`pa.Table`): PyArrow table to cast. features ([`Features`]): Target features. Returns: `pa.Table`: the casted table r rrrxrrN)r@rl)rrrr{rrrrr9rrCrzr%r,rRrr@rr.)r2r9rrrryrqrms rcast_table_to_schemars#))&1HU//0 V\\!2 2j67vj>R=SSu v$11#'>  &^^-  .MD 5EK288TFSQVZDW^d^j^jko^p^u^u;v  .   88    66sADc "SSKJnJn UR"UR5nUR 5VVs/sH"upVU"U5(a [ XXaS9OXPM$ nnn[RRXtRS9$s snnf)zEmbed external data into a table's storage. Args: table (`pyarrow.Table`): PyArrow table in which to embed data. Returns: table (`pyarrow.Table`): the table with embedded data r )rrequire_storage_embedrrl) ryrrrr9rCrr%rr.r)r2rrrrryrqrms rembed_table_storagersC))%,,7H &^^- .MD ! ) ) EKV [ .   88  /D/D  EE s)B cURU:wa [X5$URRUR:waURUR5$U$)a#Improved version of `pa.Table.cast`. It supports casting to feature types stored in the schema metadata. Args: table (`pyarrow.Table`): PyArrow table to cast. schema (`pyarrow.Schema`): Target PyArrow schema. Returns: table (`pyarrow.Table`): the casted table )r9rmetadatarrs rrDrDsK ||v#E22   &// 1,,V__== rc SSKJn UR"UR5n[ SUR 555(GaE/n/nURGHnUR UR5nX%Rn[RRUR5(a[US5(aUR5U:waeURUR55 URURVs/sHoRSUR3PM sn5 MURU5 URUR5 GM [R R#UUS9n OUR5n URSS9n U"U R$V s0sHoX_M sn 5n U R'U R(R*5$s snfs sn f) aImproved version of `pa.Table.flatten`. It behaves as `pa.Table.flatten` in a sense it does 1-step flatten of the columns with a struct type into one column per struct field, but updates the metadata and skips decodable features unless the `decode` attribute of these features is set to False. Args: table (`pa.Table`): PyArrow table to flatten. Returns: `Table`: the flattened table r rc3n# UH+n[US5=(a UR5U:Hv M- g7f)rN)rr)rr~s rr table_flatten..s0 uctU_7:y ) Pj.@.@.Bj.P Pcts35rrQ)r) max_depth)rrrr9anyrdrryr%r\r]r@rrrr}rr.rrrr) r2rr flat_arraysflat_column_namesrr,r~r  flat_table flat_features column_names rr?r?s#))%,,7H uckcrcrct uuu \\ELL,E!**-Jxx!!%**--J 22j6H6H6Jj6X""5==?3!((Z_ZdZd)eZdhZZL(--*IZd)ef""5)!((4"XX)) #* ]]_ $$q$1MYcYpYpqYp+=+EEYpqrM  - -m.H.H.Q.Q RR*frs $G7 =G<functionc^^^^SSKJnJmJm UR"UR 5nUUUU4SjmUR 5HupET"XU5 M g)zVisit all arrays in a table and apply a function to them. Args: table (`pyarrow.Table`): PyArrow table to visit. function (`Callable[[pa.Array], None]`): Function to apply to each array. r )rrsrtc>[U[R5(aURH nT"X!5 M g[U[R5(a UR nT"X5 [R RUR5(aC[US5(d2UR5Hup4T"URU5U5 M g[R RUR5(a2[UTT45(aT"URUR5 ggg)Nrv)rr%r)r+rYrZr\r]r@rrCrr`rdrq) r,rqr-ryr~rsrt_visitrs rrtable_visitor.._visit s eR__ - -u&&%!2!233  U $xx!!%**--gg~6V6V(/ $D5;;t,j9)8!!%**--g 4'8995<<9:.rN)rrrsrtrr9rC) r2rrrryrqrsrtrs ` @@@r table_visitorr sJ43))%,,7H : :") u{G$*r batch_sizec## /nSnURUS9GHn[U5S:XaMU[U5-U:a!URU5 U[U5- nMHU[U5-U:Xa8URU5 [RR U5v /nSnMX- nURUR SU55 [RR U5v UR U[U5U- 5/n[U5U- nGM U(d*U(a"[RR U5v ggg7f)a-Iterate over sub-tables of size `batch_size`. Args: table (`pyarrow.Table`): PyArrow table to iterate over. batch_size (`int`): Size of each sub-table to yield. drop_last_batch (`bool`, defaults to `False`): Drop the last batch if it is smaller than `batch_size`. rrN)rrRr}r%rrorq)r2rdrop_last_batch chunks_bufferchunks_buffer_sizer-cropped_chunk_lengths r table_iterr1 s.Mz: u:?  #e* ,z 9   ' #e* ,   #e* , :   '(('' 6 6M!" #-#B  Q0D!E F(('' 6 6"[[)=s5zL`?`abM!$U.B!B #;$ }hh##M22 -?sEEr)TTr$)F)OrEricollections.abcr functoolsr itertoolsrtypingrrrr r r numpyr_pyarrowr%pyarrow.computecomputer2 utils.loggingr ryrrr}loggerr rbrr-rcr3RecordBatchStreamReaderr7Schemar:r<rrLrrrVrXr r"rrrerrrr%r0rfboolr7Arrayr?DataTyperCrrer_rYrXrzrrmrrrrrDr?rrrrrrs $II%8 H srxxryyRXX4342C]C]4 CBII3288 tYtCyYSYSY4.C.CbX$ X$v  XAJXAx sE4 RY RYr3ZjAQSWX\]gXhSijV1V1r=$u+=S==.%DI* eblleCeDe  "((    ccmqmf 88mf kkmfCGmffjmf 288R**BLL".."J[J[ [\mfmf`osJi 88Ji+JiEIJihlJiXXJiJiZ7lrxx7l-7l7ltg g2F"((FjF,77"))7@FrxxF0bhh ,'S'ST%%Xrxxj$6F-G%@ 3e 3 3QSQYQYHZ 3r