h#SSKrSSKJr SSKJrJrJrJrJrJ r J r J r J r SSK JrJr \ \ \\4r\ "S5r\\\/\\\4rSS.S\S \S \S \ \\/\4S \4 S jjrSS.S\S\S \S \ \\/\4S \4 SjjrSS\S \ \\/\4S \4SjjrSS\4S\\S\S \S \S \S \\\4 SjjrSS\4S\\S\S \\\4Sjjr\4S\ \S\S \\\4Sjjrg)N)partial) AnyCallableIterableIteratorListOptionalSequenceTypeVarUnion) minibatchregistryItemT) get_lengthsizebufferdiscard_oversizerreturnc<UbSU0O0n[[4UUUS.UD6$)aUCreate a batcher that uses the `batch_by_padded_size` strategy. The padded size is defined as the maximum length of sequences within the batch multiplied by the number of sequences in the batch. size (int or Sequence[int]): The largest padded size to batch sequences into. Can be a single integer, or a sequence, allowing for variable batch sizes. buffer (int): The number of sequences to accumulate before sorting by length. A larger buffer will result in more even sizing, but if the buffer is very large, the iteration order will be less random, which can result in suboptimal training. discard_oversize (bool): Whether to discard sequences that are by themselves longer than the largest padded batch size. get_length (Callable or None): Function to get the length of a sequence item. The `len` function is used by default. r)rrr)rminibatch_by_padded_size)rrrr optionalss Q/home/james-whalen/.local/lib/python3.13/site-packages/spacy/training/batchers.py"configure_minibatch_by_padded_sizers=0/9.Dz*"I    )      tolerancec<UbSU0O0n[[4UUUS.UD6$)aCreate a batcher that uses the "minibatch by words" strategy. size (int or Sequence[int]): The target number of words per batch. Can be a single integer, or a sequence, allowing for variable batch sizes. tolerance (float): What percentage of the size to allow batches to exceed. discard_oversize (bool): Whether to discard sequences that by themselves exceed the tolerated size. get_length (Callable or None): Function to get the length of a sequence item. The `len` function is used by default. r)rrr)rminibatch_by_words)rrrrrs rconfigure_minibatch_by_wordsr8s="/9.Dz*"I   )     rc8UbSU0O0n[[4SU0UD6$)zCreate a batcher that creates batches of the specified size. size (int or Sequence[int]): The target number of items per batch. Can be a single integer, or a sequence, allowing for variable batch sizes. rr)rr)rrrs rconfigure_minibatchr!Ss*/9.Dz*"I 9 54 59 55rFseqsc#~# [U[5(a[R"U5nO [ U5n[ XS9Hpn[ U5n[U5n[XgU5HGnUV s/sHoU PM n n [SU 55[U 5-n U(aX:aMCU v MI Mr gs sn f7f)a=Minibatch a sequence by the size of padded batches that would result, with sequences binned by length within a window. The padded size is defined as the maximum length of sequences within the batch multiplied by the number of sequences in the batch. size (int or Sequence[int]): The largest padded size to batch sequences into. buffer (int): The number of sequences to accumulate before sorting by length. A larger buffer will result in more even sizing, but if the buffer is very large, the iteration order will be less random, which can result in suboptimal training. discard_oversize (bool): Whether to discard sequences that are by themselves longer than the largest padded batch size. get_length (Callable or None): Function to get the length of a sequence item. The `len` function is used by default. )rc38# UHn[U5v M g7fNlen).0seqs r +minibatch_by_padded_size..s;(3c#hh(N) isinstanceint itertoolsrepeatiterrlistnext_batch_by_lengthmaxr() r#rrrrsize_ outer_batch target_sizeindicesisubbatch padded_sizes rrr_s.$  &T  3 ;' 5k ' *MG0781AH8;(;;c(mKKK$> N49sA-B=/B8=AB=g?c## [U[5(a[R"U5nO [ U5n[ U5nXb-n/n/n Sn Sn UHn U"U 5n XU-:aU(dU /v M!M#U S:XaX-U::aUR U 5 X- n MHX-U -Xg-::aU R U 5 X- n MlU(aUv [ U5nXb-nU nU n /n Sn X-U::aUR U 5 X- n MX-Xg-::aU R U 5 X- n MU(aUv [ U5nXb-nU /nU n M URU 5 U(aUv gg7f)aCreate minibatches of roughly a given number of words. If any examples are longer than the specified batch length, they will appear in a batch by themselves, or be discarded if discard_oversize=True. seqs (Iterable[Sequence]): The sequences to minibatch. size (int or Sequence[int]): The target number of words per batch. Can be a single integer, or a sequence, allowing for variable batch sizes. tolerance (float): What percentage of the size to allow batches to exceed. discard_oversize (bool): Whether to discard sequences that by themselves exceed the tolerated size. get_length (Callable or None): Function to get the length of a sequence item. The `len` function is used by default. rN)r.r/r0r1r2r4appendextend)r#rrrrr7r9tol_sizebatchoverflow batch_size overflow_sizer*n_wordss rrrs($  &T u+K&H EHJMS/ 8+ +#e $a Z%9k$I LL   !J(72 8N O OOC  $M u+K".HE&JHM$4 S!% &K,BC$( K"5k &2$ QR LL   sE E" max_wordsc.[U5VVs/sHup4U"U5U4PM nnnUR5 /n/nUH[upU(dURU5 MU[U5S--U::aURU5 MGURU5 U/nM] U(aURU5 [ SU55[U5:XdeUVs/sHn[ [ U55PM nnUR5 U$s snnfs snf)zGiven a list of sequences, return a batched list of indices into the list, where the batches are grouped by length, in descending order. Batches may be at most max_words in size, defined as max sequence length * size. c38# UHn[U5v M g7fr&r')r)bs rr+#_batch_by_length..s'w!s1vvwr-) enumeratesortr?r(sumr3sortedreverse) r#rGrr;r*lengths_indicesbatchesrBlengths rr5r5s;DD/J/ 3+/OJGE$  LLO s5zA~ &) 3 LLO NN5 !CE% u 'w' '3t9 44 4078utF5M"G8 OO N%K 9s D Dr&)r0 functoolsrtypingrrrrrr r r r utilrrr/SizingrBatcherTboolrfloatrr!r(rrr5rrr]s   ' x}c! " Xe_%xU '<< =48     5'3,/0   N48    5'3,/0   8BF 6  6&x '=> 6  6" $ 5/$ $ $ $  $ d5k $T I 5/I I d5k IZ58 3-$' $s)_r