hSSKrSSKJrJr SSKJr SSKJrJr \R"\ 5r "SS5r \S5r S rS r\\R S \4S j55rg) N)PoolRLock)tqdm) experimentalloggingc\rSrSrSrSrg)ParallelBackendConfig N)__name__ __module__ __qualname____firstlineno__ backend_name__static_attributes__r T/home/james-whalen/.local/lib/python3.13/site-packages/datasets/parallel/parallel.pyr r sLrr c `[Rc[XX#XEXgU5 $[XX#XEXgU5 $)a **Experimental.** Apply a function to iterable elements in parallel, where the implementation uses either multiprocessing.Pool or joblib for parallelization. Args: function (`Callable[[Any], Any]`): Function to be applied to `iterable`. iterable (`list`, `tuple` or `np.ndarray`): Iterable elements to apply function to. num_proc (`int`): Number of processes (if no backend specified) or jobs (using joblib). types (`tuple`): Additional types (besides `dict` values) to apply `function` recursively to their elements. disable_tqdm (`bool`): Whether to disable the tqdm progressbar. desc (`str`): Prefix for the tqdm progressbar. single_map_nested_func (`Callable`): Map function that applies `function` to an element from `iterable`. Takes a tuple of function, data_struct, types, rank, disable_tqdm, desc as input, where data_struct is an element of `iterable`, and `rank` is used for progress bar. )r r_map_with_multiprocessing_pool_map_with_joblib) functioniterablenum_procbatched batch_sizetypes disable_tqdmdescsingle_map_nested_funcs r parallel_mapr!sD"))1- :lZp   Hz,Vl rc U[U5::aUO [U5n/n [U5HTn [U5U-n [U5U-n X-[X5-n X-X:aSOS-nU RXXX4XZXg45 MV [U5[ SU 55:wa*[ S[U5S[ SU 5535e[ RSUS[U5S U Vs/sHn[US5PM sn35 S unnU(d[54[Rnn[UUUS 9nURX5nSSS5 [ RS US 35 WVVs/sHnUHnUPM M nnn[ RS[U5S35 U$s snf!,(df  Nn=fs snnf)Nrc3># UHn[US5v M g7fr#Nlen.0is r 1_map_with_multiprocessing_pool..7s:z!C!IIzzHError dividing inputs iterable among processes. Total number of objects z , length: c3># UHn[US5v M g7fr%r&r(s rr+r,;s9j3qt99jr-z Spawning z processes for z objects in slices of )NN)initargs initializerz Finished z processesz Unpacked z objects) r'rangeminappendsum ValueErrorloggerinforrset_lockrmap)rrrrrrrrr split_kwdsindexdivmodstartendr*r/r0poolmappedproc_resobjs rrr+s$s8}4x#h-HJx(mx'(mh& c%o-k%+Q158e%8'u]ipq ! 8}:z:::''*8}o69j99: <   KK H:_S]O;QfpRqfpabSVWXYZW[S\fpRqQrs'Hk !& DMM+ h{ Ct0= D KK)H:Z01"( =&hHScHc&F = KK)CK=12 MSr D C>s.F6F;9G ; G c ^^^^^^ SSKm T R[RUS9 T R 5"UUUU UU4SjU55sSSS5 $!,(df  g=f)Nr)n_jobsc 3b># UH$nTRT5"TUTTTSSS45v M& g7f)NT)delayed)r)rCrrrjoblibr rs rr+#_map_with_joblib..Us?!  NN1 2Hc7JX]_ceiko3p q qs,/)rHparallel_backendr rParallel) rrrrrrrrr rHs ` ``` `@rrrMsP   !6!C!CH U ! ! !   V U Us %A A&rc## U[lUS:Xa SSKJn U"5 Sv S[lg!S[lf=f7f)ag **Experimental.** Configures the parallel backend for parallelized dataset loading, which uses the parallelization implemented by joblib. Args: backend_name (str): Name of backend for parallelization implementation, has to be supported by joblib. Example usage: ```py with parallel_backend('spark'): dataset = load_dataset(..., num_proc=2) ``` sparkr)register_sparkN)r r joblibsparkrN)rrNs rrJrJ[s; *6&w. 2 -1*T*sA2 A ?A) contextlibmultiprocessingrr tqdm.autorutilsrr get_loggerr r6r r!rrcontextmanagerstrrJr rrrWsu')   H %4D   2322r