hh SSKrSSKrSSKJrJrJr SSKJr SSKJ r J r J r J r J r Jr SSKrSSKrSSKrSSKJr SSKJrJrJrJr SSKJr SS KJr \ "S 5r \ "S 5r!\ "S 5r"\ "S 5r#S\$S\%4Sjr&S\ 4Sjr'S\S\\(\)\$\*\4S\S\R:4Sjr+S\S\\(\)\$\*\4S\R:4Sjr,S\RZS\%4Sjr."SS\ \!\"\#45r/S\0\*\1\ 4S\0\*\ 44Sjr2"SS\/\R:\RZ\R:45r3"SS\/\0\1\045r4"S S!\/\0\Rj\045r6"S"S#\/\Rn\Rp\Rn45r9"S$S%5r:"S&S'5r;"S(S)\5r<"S*S+\<5r="S,S-\<5r>"S.S/\ \!\"\#45r?"S0S1\?\!\"\#45r@"S2S3\?\!\"\#45rA"S4S5\A\R:\RZ\R:45rB"S6S7\?\\1\45rC"S8S9\A\Rn\Rp\Rn45rD"S:S;\?\0\"\045rES\*S<\1\*SS4S=jrFS\\(\)\$\4S>\(SS4S?jrGS\\(\)\$\*\4S\*4S@jrHSES\S\\(\)\$\*\4S\ \S\R:4SAjjrISFS\S\\(\)\$\*\4SB\?SC\ \14SDjjrJg)GN)IterableMappingMutableMapping)partial)AnyCallableGenericOptionalTypeVarUnion)Features)_ArrayXDExtensionType_is_zero_copy_onlydecode_nested_examplepandas_types_mapper)Table)no_op_if_value_is_nullT RowFormat ColumnFormat BatchFormatkeyreturnc`URS:H=(a URUR:$)N)stepstopstartrs X/home/james-whalen/.local/lib/python3.13/site-packages/datasets/formatting/formatting.py_is_range_contiguousr"(s" 88q= 2SXX22c8[SUS[U5S35e)NzWrong key type: 'z ' of type 'z6'. Expected one of int, slice, range, str or Iterable.) TypeErrortyper s r!_raise_bad_key_typer',s&  C5 DI;6lm r#tableindicesc [U[5(aIURXR-S5R S5SR 5n[ X5$[U[5(a![URUR56n[U[5(a[U5(ayURS:ai[ XRURURUR- 5R S5Vs/sHo3R 5PM sn5$[U[5(a:URU/5n[ XR S5R55$[U[ 5(aK[ XVs/sH4o2RUS5R S5SR 5PM6 sn5$[#U5 gs snfs snf)a1 Query a pyarrow Table to extract the subtable that correspond to the given key. The :obj:`indices` parameter corresponds to the indices mapping in case we cant to take into account a shuffling or an indices selection for example. The indices table must contain one column named "indices" of type uint64. rrN) isinstanceint fast_slicenum_rowscolumnas_py _query_tablesliceranger)r"rrstrselect to_pylistrr')r(rr)is r!!_query_table_with_indices_mappingr82s#s  '7'7!7;BB1EaHNNPE''#uS[[!1!123#u  $ $a+=+=ciiTWT]T]I]+^+e+efg+hi+ha +hi  #s cU#E>>!#4#>#>#@AA#x  E[^#_[^VW$6$6q!$<$C$CA$Fq$I$O$O$Q[^#_``j$`s G/ !;G4 cp[U[5(aURXR-S5$[U[5(a![ UR UR56n[U[ 5(aT[U5(aCURS:a3URURURUR- 5$[U[5(a=URRURVs/sH o"U:wdM UPM sn5$[U[5(am[R "U[R"5n[%U5S:XaURR SS5$UR'XR-5$[)U5 gs snf)zQ Query a pyarrow Table to extract the subtable that correspond to the given key. rrN)r+r,r-r.r2r3r)r"rrr4r(drop column_namesrnpfromiterint64len fast_gatherr')r(rr/s r!r1r1Ps?#snn 4a88#uS[[01#u  $ $a##CIIsxx#))/CD D #s{{e6H6H Z6HFVYM6H Z[[#x  kk#rxx( s8q=;;$$Q* *  ~~!566![s F3F3pa_arrayc URS:$Nr) null_count)rAs r!_is_array_with_nullsrEis    ""r#c\rSrSrSrS\R S\4SjrS\R S\ 4Sjr S\R S\ 4Sjr Sr g ) BaseArrowExtractormz Arrow extractor are used to extract data from pyarrow tables. It makes it possible to extract rows, columns and batches. These three extractions types have to be implemented. pa_tablerc[eNNotImplementedErrorselfrIs r! extract_rowBaseArrowExtractor.extract_rowt!!r#c[erKrLrNs r!extract_column!BaseArrowExtractor.extract_columnwrRr#c[erKrLrNs r! extract_batch BaseArrowExtractor.extract_batchzrRr#N)__name__ __module__ __qualname____firstlineno____doc__parrrPrrTrrW__static_attributes__rYr#r!rGrGmsL "BHH"""rxx"L""bhh";"r#rGpy_dictc^UR5VVs0sH upXS_M snn$s snnf)z:Return the first element of a batch (dict) as a row (dict)r)items)rararrays r!_unnestre~s(,3MMO >J_```s/rAc^[U[R5(a[UR[5(aT[ URR SS9nURVVs/sHo3RUS9HoDPM M snnmGO[ UR5=(a [SUR55nURVVs/sHo3RUS9HoDPM M snnmO[UR[5(a.[ URR SS9nURUS9mOI[ UR5=(a [U5(+nURUS9R5m[T5S:a~[U4SjT55(ad[RR![R"5S:a[R$"T[&S9$[R("TS [&S 9$[RR![R"5S:a[R$"T5$[R("TS S 9$s snnfs snnf) NT)unnest)zero_copy_onlyc3B# UHn[U5(+v M g7frK)rE).0chunks r! .s KAP,U333src3:># UHn[U[R5=(a6 UR[:H=(d UR TSR :g=(d- [U[ 5=(a [R"U5v M g7f)rN)r+r<ndarraydtypeobjectshapefloatisnan)rxrds r!rrsoAArzz*_60A0^QWWPUVWPXP^P^E^:q%(8RXXa[:sBBz2.0.0b1)rF)copyrr)r+r_ ChunkedArrayr&rr storage_dtypechunksto_numpyallrEtolistr?anyr<lib NumpyVersion __version__asarrayrrd)rOrArrrowrds @r!r)NumpyArrowExtractor._arrow_array_to_numpys h 0 0(--)>??!3HMM4O4OX\!]%-__%4E^l@mC@mC_"4HMM!B"sKAIKH&.__%4E^l@mC@mC_(--)>??!3HMM4O4OX\!]&//~/N!3HMM!B!iK_`hKiGi&//~/NUUW u:> 66&&r~~6)C::e6::xxE@@ 66  r~~ .) ;::e$ $88E. .?s -!J!JrN)rZr[r\r]rr_rr{rPr<rrTrWrprr`rYr#r!r~r~sq/5BHH55NrxxNBJJNabhha4a$/bhh$/2::$/r#r~c\rSrSrS\R S\R4SjrS\R S\R4Sjr S\R S\R4Sjr Sr g) PandasArrowExtractorrIrcBURSS9R[S9$)Nr)length types_mapper)r2 to_pandasrrNs r!rP PandasArrowExtractor.extract_rows"~~Q~'11?R1SSr#chURS/5R[S9URS$)Nrr)r5rrr;rNs r!rT#PandasArrowExtractor.extract_columns4s#--;N-OPXPePefgPhiir#c(UR[S9$)Nr)rrrNs r!rW"PandasArrowExtractor.extract_batchs!!/B!CCr#rYN) rZr[r\r]r_rpd DataFramerPSeriesrTrWr`rYr#r!rrs\TBHHTTjrxxjBIIjDbhhD2<<Dr#rc \rSrSrSS\\S\\\\\\ S444Sjjr S\S\4Sjr S \ S \S\ 4S jr S \S\4S jrSrg)PythonFeaturesDecoderNfeaturestoken_per_repo_idcXlX lgrKrrrOrrs r!rPythonFeaturesDecoder.__init__s! !2r#rrcnUR(a#URRXRS9$U$N)r)rdecode_exampler)rOrs r! decode_row PythonFeaturesDecoder.decode_rows.^b^k^kt}}++CCYCY+Ztqttr#r/ column_namecpUR(a$URRXURS9$U$r)r decode_columnr)rOr/rs r!r#PythonFeaturesDecoder.decode_columns9}} MM ' 'tOeOe ' f  r#batchcnUR(a#URRXRS9$U$r)r decode_batchrrOrs r!r"PythonFeaturesDecoder.decode_batchs.^b^k^kt}}))%CYCY)Zvqvvr#rrK)rZr[r\r]r rr{r4r boolrrr|rrr`rYr#r!rrsmq3 *3?GSRWX[]acgXgRhMhHi?j3 udutu D s t w$w4wr#rc\rSrSrS\\4SjrS\RS\R4Sjr S\RS\ S\R4S jr S \RS\R4S jr S rg )PandasFeaturesDecoderrcXlgrKr)rOrs r!rPandasFeaturesDecoder.__init__s r#rrc jUR(agURR5VVs0sH?up#URRU(dM%U[[ [ U55_MA snnO0nU(a+UR U5U[UR55'U$s snnfrK) rrc_column_requires_decodingrrr transformr|keys)rOrrfeaturedecodes r!r PandasFeaturesDecoder.decode_rows}}-1MM,?,?,A ,A(K==::;G] 3G[TU]X5 X lgrK)superrlazy)rOrrhrrs r!rPythonFormatter.__init__s 5 r#rIrcUR(a [X5$UR5RU5nURR U5nU$rK)rhr0rNrPr<rrOrIrs r!rBPythonFormatter.format_rowsH 998* *))+77A**55c: r#cUR5RU5nURRX!RS5nU$rC)rNrTr<rr;rOrIr/s r!r3PythonFormatter.format_columnB,,.==hG--;;FDYDYZ[D\] r#cUR(a [X5$UR5RU5nURR U5nU$rK)rhr7rNrWr<rrOrIrs r!rCPythonFormatter.format_batchsH 99X, ,++-;;HE,,99%@ r#)rh)NFN)rZr[r\r]rr_rrrBr|r3rCr` __classcell__rs@r!reresR288bhh4 RXX'r#rec\rSrSrSrSrS\RS\R4Sjr S\RS\R4Sjr S\RS\R4Sjr S rg ) PandasFormatterizpandas dataframez pandas seriesrIrczUR5RU5nURRU5nU$rK)rPrPr=rrks r!rBPandasFormatter.format_rows6))+77A**55c: r#cUR5RU5nURRX!RS5nU$rC)rPrTr=rr;rns r!r3PandasFormatter.format_columnrpr#czUR5RU5nURRU5nU$rK)rPrWr=rrks r!rCPandasFormatter.format_batchs6))+99(C**77< r#rYN)rZr[r\r]rYrZr_rrrrBrr3rCr`rYr#r!rwrws]#J!K288  bhh299 RXX",,r#rwc^\rSrSrSrS S\\/\44U4SjjjrS\RS\4Sjr S\RS\ 4Sjr S\RS\4S jr S rU=r$) CustomFormatteria A user-defined custom formatter function defined by a ``transform``. The transform must take as input a batch of data extracted for an arrow table using the python extractor, and return a batch. If the output batch is not a dict, then output_all_columns won't work. If the output batch has several fields, then querying a single column won't work since we don't know which field to return. rc ,>[TU]X#S9 Xlg)Nr)rgrr)rOrrrkwargsrs r!rCustomFormatter.__init__s (P"r#rIrc~URU5n[U5$![an[SU35UeSnAff=f)Nz]Custom formatting function must return a dict of sequences to be able to pick a row, but got )rCre Exceptionr%rOrIformatted_batchexcs r!rBCustomFormatter.format_rowsU++H5 ?+ + oppAB  s  <7<cPURU5n[US5(aD[UR55S:a&[ S[ UR55S35eO[ SU35eX!R S$![an[ SU35UeSnAff=f)NrrzTried to query a column but the custom formatting function returns too many columns. Only one column was expected but got columns .zPCustom formatting function must return a dict to be able to pick a row, but got r)rChasattrr?rr%r|r;rrs r!r3CustomFormatter.format_columns++H5 ?F + +?'')*Q.DDHI]I]I_D`Caabd/ bcrbst  "#8#8#;< < bcrbst  s6B B%B  B%cUR5RU5nURRU5nUR U5$rK)rNrWr<rrrrs r!rCCustomFormatter.format_batchs@++-;;HE,,99%@~~e$$r#)rrL)rZr[r\r]r^rr{rr_rrBrr3rCr`rtrus@r!rrsn#(D64<"8##288bhh<&%RXX%$%%r#rcolumnsc0X;a[SUSU35eg)NzColumn z5 not in the dataset. Current columns in the dataset: )KeyError)rrs r!_check_valid_column_keyr s( %Z[bZcdeer#sizec[U[5(a%US:aX-S:dX:a[SUSU35eg[U[5(ag[U[5(a7[ U5S:a'[ [U5US9 [ [U5US9 gg[U[5(aI[ U5S:a9[ [[U55US9 [ [[U55US9 gg[U5 g)Nrz Invalid key: z is out of bounds for size )r) r+r, IndexErrorr2r3r?_check_valid_index_keymaxminrr')rrs r!rr%s#s !G QCK}SE1LTFST T C   C   s8a< "3s8$ 7 "3s8$ 7  C " " s8a< "3s3x=t < "3s3x=t <  C r#c[U[R5(ag[U[5(ag[U[[ [ 45(ag[U5 grA)r+numbersIntegralr4r2r3rr'r s r!key_to_query_typer8sG#w''(( C   C%1 2 2r#c[U[[[[[ 45(d[ R"U5n[U[5(a[XR5 O&Ub URO URn[X5 Uc [X5nU$[XUS9nU$![a [U5 Nf=f)a Query a Table to extract the subtable that correspond to the given key. Args: table (``datasets.table.Table``): The input Table to query from key (``Union[int, slice, range, str, Iterable]``): The key can be of different types: - an integer i: the subtable containing only the i-th row - a slice [i:j:k]: the subtable containing the rows that correspond to this slice - a range(i, j, k): the subtable containing the rows that correspond to this range - a string c: the subtable containing all the rows but only the column c - an iterable l: the subtable that is the concatenation of all the i-th rows for all i in the iterable indices (Optional ``datasets.table.Table``): If not None, it is used to re-map the given key to the table rows. The indices table must contain one column named "indices" of type uint64. This is used in case of shuffling or rows selection. Returns: ``pyarrow.Table``: the result of the query on the input table )r))r+r,r2r3r4roperatorindexr%r'rr;r.rr1r8)r(rr)r pa_subtables r! query_tablerBs2 cCsH= > > %..%C#s%7%78#*#6wENNs)"5.  8GT  %  $ %sB//CCrformat_columnsc^[U[5(a URnOUn[U5n[ UR S9nTcU"XVS9$US:XaUT;aU"XV5$U"XVS9$UR U4SjUR55nU"XS9n U(aa[U [5(a>UR U4SjUR55n U"XS9n U RU 5 U $[SU 35eU $)a Format a Table depending on the key that was used and a Formatter object. Args: table (``datasets.table.Table``): The input Table to format key (``Union[int, slice, range, str, Iterable]``): Depending on the key that was used, the formatter formats the table as either a row, a column or a batch. formatter (``datasets.formatting.formatting.Formatter``): Any subclass of a Formatter such as PythonFormatter, NumpyFormatter, etc. format_columns (:obj:`List[str]`, optional): if not None, it defines the columns that will be formatted using the given formatter. Other columns are discarded (unless ``output_all_columns`` is True) output_all_columns (:obj:`bool`, defaults to False). If True, the formatted output is completed using the columns that are not in the ``format_columns`` list. For these columns, the PythonFormatter is used. Returns: A row, column or batch formatted object defined by the Formatter: - the PythonFormatter returns a dictionary for a row or a batch, and a list for a column. - the NumpyFormatter returns a dictionary for a row or a batch, and a np.array for a column. - the PandasFormatter returns a pd.DataFrame for a row or a batch, and a pd.Series for a column. - the TorchFormatter returns a dictionary for a row or a batch, and a torch.Tensor for a column. - the TFFormatter returns a dictionary for a row or a batch, and a tf.Tensor for a column. r)r?r/c36># UHoT;dM Uv M g7frKrYrrrs r!rformat_table..s*m:O3^lSl33:O  c36># UHoT;dM Uv M g7frKrYrs r!rrs@#8C>?   rtDsEF r#rK)NF)Krrcollections.abcrrr functoolsrtypingrrr r r r numpyr<pandasrpyarrowr_rrfeatures.featuresrrrrr(rutils.py_utilsrrrrrr3rr"r'r,r2r4r8r1rprErGr{r|rergrrrr~rrrrrrr0r7rrRrXr]rerwrrrrrrrYr#r!rs==DCuu3 CL K ~& m$ 3e33S  S%X=>INXX<E#ueS(*J$KPRPXPX2#288##"L+!EF""=T#tAw,'=DaL= -bhh"((.JK$-dD$.>?$1/,T2::t-CD1/hD-bllBIIr||.STDww*&&@j$~j$ZLhL II $" <<=$"N"i < DE" Yy, CD E^BHHbhh$@A Eiw 672nR\\299bll%JK(-%ilD 89-%`fftCyfTf !c5%&A B!#!RV!&5eUC!ABs $( ( sE5#x/ 0(e_(XX (^&* 9 9 sE5#x/ 09 9 TN 9 r#