΅i0SSKJr SSKJrJrJr SSKJrJrJ r SSK J r SSK J r Jr SSKJr /SQr\ "SS S 9rS \S \4S jr\ "S5"SS\\ 55r\ "S5"SS\55r\ "S5"SS\\ 55rg)) defaultdict)CallableIteratorSized)AnyNoReturnTypeVar)functional_datapipe) DataChunk IterDataPipe)_check_unpickable_fn)BatcherIterDataPipeGrouperIterDataPipeUnBatcherIterDataPipe_T_coT) covariantnamereturnc,[S[SU35e)Nzmodule z has no attribute )AttributeError__name__)rs b/home/james-whalen/.local/lib/python3.13/site-packages/torch/utils/data/datapipes/iter/grouping.py __getattr__rs 78*,>tfE FFbatchc ^\rSrSr%Sr\\S'\\S'\\S'S\ 4S\S\S\S\ \ SS 4 U4S jjjr S\ \ 4S jr S\4S jrS rU=r$)ra Creates mini-batches of data (functional name: ``batch``). An outer dimension will be added as ``batch_size`` if ``drop_last`` is set to ``True``, or ``length % batch_size`` for the last batch if ``drop_last`` is set to ``False``. Args: datapipe: Iterable DataPipe being batched batch_size: The size of each batch drop_last: Option to drop the last batch if it's not full wrapper_class: wrapper to apply onto each batch (type ``List``) before yielding, defaults to ``DataChunk`` Example: >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> dp = IterableWrapper(range(10)) >>> dp = dp.batch(batch_size=3, drop_last=True) >>> list(dp) [[0, 1, 2], [3, 4, 5], [6, 7, 8]] datapipe batch_size drop_lastF wrapper_classrNct>US::a [S5e[TU] 5 XlX lX0lX@lg)Nrz+Batch size is required to be larger than 0!)AssertionErrorsuper__init__rrr r!)selfrrr r! __class__s rr%BatcherIterDataPipe.__init__5s8 ? !NO O  $"*rc#"# /nURHDnURU5 [U5UR:XdM/UR U5v /nMF [U5S:a&UR (dUR U5v ggg7fNr)rappendlenrr!r )r&rxs r__iter__BatcherIterDataPipe.__iter__DsxA LLO5zT__,((//  u:>>>((//" s :BABcL[UR[5(aeUR(a"[ UR5UR -$[ UR5UR -S- UR -$[ [U5RS35e)Nz# instance doesn't have valid length) isinstancerrr r,r TypeErrortyperr&s r__len__BatcherIterDataPipe.__len__Osv dmmU + +~~4==)T__<<DMM*T__\rSrSrSrS S\S\SS4SjjrSrS r S r g) rYa Undos batching of data (functional name: ``unbatch``). In other words, it flattens the data up to the specified level within a batched DataPipe. Args: datapipe: Iterable DataPipe being un-batched unbatch_level: Defaults to ``1`` (only flattening the top level). If set to ``2``, it will flatten the top two levels, and ``-1`` will flatten the entire DataPipe. Example: >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> source_dp = IterableWrapper([[[0, 1], [2]], [[3, 4], [5]], [[6]]]) >>> dp1 = source_dp.unbatch() >>> list(dp1) [[0, 1], [2], [3, 4], [5], [6]] >>> dp2 = source_dp.unbatch(unbatch_level=2) >>> list(dp2) [0, 1, 2, 3, 4, 5, 6] r unbatch_levelrNcXlX lgNrrD)r&rrDs rr%UnBatcherIterDataPipe.__init__qs  *rc#x# URH$nURXRS9ShvN M& gN 7f)NrD)r_diverD)r&elements rr.UnBatcherIterDataPipe.__iter__us0}}Gzz'9K9KzL L L% Ls ,:8 :c## US:a [S5eUS:XaB[U[[45(a"UHnUR USS9ShvN M gUv gUS:XaUv g[U[[45(a$UHnUR X2S- S9ShvN M g[ SUR S35eNqN$7f)Nz unbatch_level must be -1 or >= 0rJrr1zunbatch_level z" exceeds the depth of the DataPipe) ValueErrorr2listr rK IndexErrorrD)r&rLrDitems rrKUnBatcherIterDataPipe._diveys 2 ?@ @ B 'D)#455#D#zz$bzAAA$ a M'D)#455#D#zz$a>OzPPP$!$T%7%7$88Z[BQs%A C B?ACC#CCrG)r1) rr8r9r:r;r r=r%r.rKr?rrrrYs,,++c+$+Mrrgroupbyc\rSrSrSrSSSSSS.S\\S\\/\4S \ S \ S \ S-S \ S-S \ SS4Sjjr Sr Sr SSjrSrSrSSjrSrg)ra Groups data from IterDataPipe by keys from ``group_key_fn``, yielding a ``DataChunk`` with batch size up to ``group_size``. (functional name: ``groupby``). The samples are read sequentially from the source ``datapipe``, and a batch of samples belonging to the same group will be yielded as soon as the size of the batch reaches ``group_size``. When the buffer is full, the DataPipe will yield the largest batch with the same key, provided that its size is larger than ``guaranteed_group_size``. If its size is smaller, it will be dropped if ``drop_remaining=True``. After iterating through the entirety of source ``datapipe``, everything not dropped due to the buffer capacity will be yielded from the buffer, even if the group sizes are smaller than ``guaranteed_group_size``. Args: datapipe: Iterable datapipe to be grouped group_key_fn: Function used to generate group key from the data of the source datapipe keep_key: Option to yield the matching key along with the items in a tuple, resulting in `(key, [items])` otherwise returning [items] buffer_size: The size of buffer for ungrouped data group_size: The max size of each group, a batch is yielded as soon as it reaches this size guaranteed_group_size: The guaranteed minimum group size to be yielded in case the buffer is full drop_remaining: Specifies if the group smaller than ``guaranteed_group_size`` will be dropped from buffer when the buffer is full Example: >>> import os >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> def group_fn(file): ... return os.path.basename(file).split(".")[0] >>> source_dp = IterableWrapper( ... ["a.png", "b.png", "a.json", "b.json", "a.jpg", "c.json"] ... ) >>> dp0 = source_dp.groupby(group_key_fn=group_fn) >>> list(dp0) [['a.png', 'a.json', 'a.jpg'], ['b.png', 'b.json'], ['c.json']] >>> # A group is yielded as soon as its size equals to `group_size` >>> dp1 = source_dp.groupby(group_key_fn=group_fn, group_size=2) >>> list(dp1) [['a.png', 'a.json'], ['b.png', 'b.json'], ['a.jpg'], ['c.json']] >>> # Scenario where `buffer` is full, and group 'a' needs to be yielded since its size > `guaranteed_group_size` >>> dp2 = source_dp.groupby( ... group_key_fn=group_fn, ... buffer_size=3, ... group_size=3, ... guaranteed_group_size=2, ... ) >>> list(dp2) [['a.png', 'a.json'], ['b.png', 'b.json'], ['a.jpg'], ['c.json']] Fi'N)keep_key buffer_size group_sizeguaranteed_group_sizedrop_remainingr group_key_fnrYrZr[r\r]rcT[U5 XlX lX0lX@l[ [ 5UlSUlXPl SUl Ub#Ub SUs=:aU::d O [S5eXPl Ub#UbSUs=:aU::d O [S5eX`l Xpl [Ulg)Nrz)group_size must be > 0 and <= buffer_sizezNguaranteed_group_size must be > 0 and <= group_size and group_size must be set)r rr^rYmax_buffer_sizerrQbuffer_elementscurr_buffer_sizer[r\r#r]r r!)r&rr^rYrZr[r\r]s rr%GrouperIterDataPipe.__init__s \* ( *7B47H !$%)"  !k&= 1k1$%PQQ)3 & ,!!.C*Qz*Q$d*? &,&rcSnSnSnURH;n[URU5U:dM![URU5nUnM= URbBX R:a3UR(d"[ S[ URU55eURbX R:aURUnU=R U-slURU U$)NrzFailed to group items)rar,r\r] RuntimeErrorstrrb)r& biggest_key biggest_sizeresult_to_yieldfindkeys r_remove_biggest_key'GrouperIterDataPipe._remove_biggest_keys  ++G4''01L@"4#7#7#@A % ,  & & 2999'''T-A-A+-N)O   & & .999"22;?O -   -rc## URGHEnURU5nURURU5 U=RS- slUR bUR [ URU5:XanURURU5nUR(aX#4OUv U=R[ URU5-slURU URUR:XdGMUR5nUcGMURU5nUR(aX#4OUv GMH [URR55HcnURURRU55nU=R[ U5-slUR(aX#4OUv Me g7f)Nr1)rr^rar+rbr[r,r!rYr`rktuplekeyspop)r&r-keyresultris rr.GrouperIterDataPipe.__iter__sA##A&C   % , ,Q /  ! !Q & !*t#$$S)C0*.););Dr=r%rkr.rur}rrr?rUrrrrs1p !%,0$$'u%$'w|,$'  $'  $'$J$' #Tz$'$' $'L:=41"1 %rrN) collectionsrcollections.abcrrrtypingrrr %torch.utils.data.datapipes._decoratorr #torch.utils.data.datapipes.datapiper r 'torch.utils.data.datapipes.utils.commonr __all__rrfrrrrrUrrrs#55))EGH  4(GcGhGW<Y,y1<Y<Y~Y1L1 1hYw%,y1w% w%r