ViSrSSKrSSKrSSKrSSKrSSKrSSKrSSKJrJ r J r J r J r SSKJr \"S5r\"S5rSSKJr SSKJr S rS rS r S r!S9Sjr"S9Sjr#S:Sjr$S:Sjr%S:Sjr&S:Sjr'S:Sjr(S:Sjr)S:Sjr*Sr+Sr,Sr-Sr.Sr/S;Sjr0S;Sjr1SS*r?\?r@S+rAS,rB\?\A\B4S-jrC"S.S/\D\E\F5rG\4S0jrHS1S 4S2jrI"S3S4\5rJ"S5S6\J5rK\J"5rL\K"5rMS>S7jrNS=S8jrOg!\ a SSKJrJ r J r J r J r GN&f=f!\ a \"5r\"5rGN*f=f!\ a S r\\4r\r\\srrGN5f=f)?a:mod:`itertools` is full of great examples of Python generator usage. However, there are still some critical gaps. ``iterutils`` fills many of those gaps with featureful, tested, and Pythonic solutions. Many of the functions below have two versions, one which returns an iterator (denoted by the ``*_iter`` naming pattern), and a shorter-named convenience form that returns a list. Some of the following are based on examples in itertools docs. N)MappingSequenceSet ItemsViewIterable) make_sentinel_UNSET _REMAP_EXIT)filter)izipFTc<[U5 g![a gf=f)zSimilar in nature to :func:`callable`, ``is_iterable`` returns ``True`` if an object is `iterable`_, ``False`` if not. >>> is_iterable([]) True >>> is_iterable(object()) False .. _iterable: https://docs.python.org/2/glossary.html#term-iterable FT)iter TypeErrorobjs d/home/james-whalen/.local/share/pipx/venvs/semgrep/lib/python3.13/site-packages/boltons/iterutils.py is_iterablerMs' S   s  cP[U5(+=(d [U[5$)aNA near-mirror of :func:`is_iterable`. Returns ``False`` if an object is an iterable container type. Strings are considered scalar as well, because strings are more often treated as whole values as opposed to iterables of 1-character substrings. >>> is_scalar(object()) True >>> is_scalar(range(10)) False >>> is_scalar('hello') True r isinstance basestringrs r is_scalarr_s3  >:c:#>>cP[U5=(a [U[5(+$)zThe opposite of :func:`is_scalar`. Returns ``True`` if an object is an iterable other than a string. >>> is_collection(object()) False >>> is_collection(range(10)) True >>> is_collection('hello') False rrs r is_collectionros s  ?JsJ$? ??rc,[[XU55$)aSplits an iterable based on a separator. Like :meth:`str.split`, but for all iterables. Returns a list of lists. >>> split(['hi', 'hello', None, None, 'sup', None, 'soap', None]) [['hi', 'hello'], ['sup'], ['soap']] See :func:`split_iter` docs for more info. )list split_iter)srcsepmaxsplits rsplitr"}s  3X. //rc#^# [U5(d [S5eUb[U5nUS:XaU/v g[T5(aTnO([ T5(d[ T5mU4SjnOU4Sjn/nSnUHEnUbXR:aSnU"U5(aTc U(dM'US- nUv /nM4UR U5 MG U(dTbUv g7f)aDSplits an iterable based on a separator, *sep*, a max of *maxsplit* times (no max by default). *sep* can be: * a single value * an iterable of separators * a single-argument callable that returns True when a separator is encountered ``split_iter()`` yields lists of non-separator values. A separator will never appear in the output. >>> list(split_iter(['hi', 'hello', None, None, 'sup', None, 'soap', None])) [['hi', 'hello'], ['sup'], ['soap']] Note that ``split_iter`` is based on :func:`str.split`, so if *sep* is ``None``, ``split()`` **groups** separators. If empty lists are desired between two contiguous ``None`` values, simply use ``sep=[None]``: >>> list(split_iter(['hi', 'hello', None, None, 'sup', None])) [['hi', 'hello'], ['sup']] >>> list(split_iter(['hi', 'hello', None, None, 'sup', None], sep=[None])) [['hi', 'hello'], [], ['sup'], []] Using a callable separator: >>> falsy_sep = lambda x: not x >>> list(split_iter(['hi', 'hello', None, '', 'sup', False], falsy_sep)) [['hi', 'hello'], [], ['sup'], []] See :func:`split` for a list-returning version. expected an iterableNrc>UT;$Nxr s rsplit_iter.. Q#Xrc>UT:H$r&r'r(s rr*r+r,rcgNFr'r)s rr*r+sr)rrintcallabler frozensetappend)rr r!sep_func cur_group split_countss ` rrrsD s  .//x= q=%K }} s^^n%%IK   K$;&H A;;{9 1 KOI   Q CO sCCc*[[X55$)zStrips values from the beginning of an iterable. Stripped items will match the value of the argument strip_value. Functionality is analigous to that of the method str.lstrip. Returns a list. >>> lstrip(['Foo', 'Bar', 'Bam'], 'Foo') ['Bar', 'Bam'] )r lstrip_iteriterable strip_values rlstripr?s  H2 33rc#d# [U5nUHnX1:wdM Uv O UHnUv M g7f)a Strips values from the beginning of an iterable. Stripped items will match the value of the argument strip_value. Functionality is analigous to that of the method str.lstrip. Returns a generator. >>> list(lstrip_iter(['Foo', 'Bar', 'Bam'], 'Foo')) ['Bar', 'Bam'] N)r)r=r>iteratoris rr;r;s9H~H  G s00c*[[X55$)zStrips values from the end of an iterable. Stripped items will match the value of the argument strip_value. Functionality is analigous to that of the method str.rstrip. Returns a list. >>> rstrip(['Foo', 'Bar', 'Bam'], 'Bam') ['Foo', 'Bar'] )r rstrip_iterr<s rrstriprEs  H1 22rc## [U5nUHbnX1:XaV[5nURU5 SnUHnX1:XaURU5 MSn O U(d gUHnUv M Uv Md g7f)aStrips values from the end of an iterable. Stripped items will match the value of the argument strip_value. Functionality is analigous to that of the method str.rstrip. Returns a generator. >>> list(rstrip_iter(['Foo', 'Bar', 'Bam'], 'Bam')) ['Foo', 'Bar'] FTN)rrr5)r=r>rArBcachebrokents rrDrDsvH~H  FE LLOF#LLO!F  sA5A7c*[[X55$)aStrips values from the beginning and end of an iterable. Stripped items will match the value of the argument strip_value. Functionality is analigous to that of the method str.strip. Returns a list. >>> strip(['Fu', 'Foo', 'Bar', 'Bam', 'Fu'], 'Fu') ['Foo', 'Bar', 'Bam'] )r strip_iterr<s rstriprLs  80 11rc,[[X5U5$)a!Strips values from the beginning and end of an iterable. Stripped items will match the value of the argument strip_value. Functionality is analigous to that of the method str.strip. Returns a generator. >>> list(strip_iter(['Fu', 'Foo', 'Bar', 'Bam', 'Fu'], 'Fu')) ['Foo', 'Bar', 'Bam'] )rDr;r<s rrKrK#s {88 EErc t[X40UD6nUc [U5$[[R"XB55$)a0Returns a list of *count* chunks, each with *size* elements, generated from iterable *src*. If *src* is not evenly divisible by *size*, the final chunk will have fewer than *size* elements. Provide the *fill* keyword argument to provide a pad value and enable padding, otherwise no padding will take place. >>> chunked(range(10), 3) [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]] >>> chunked(range(10), 3, fill=None) [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, None, None]] >>> chunked(range(10), 3, count=2) [[0, 1, 2], [3, 4, 5]] See :func:`chunked_iter` for more info. ) chunked_iterr itertoolsislice)rsizecountkw chunk_iters rchunkedrV/s9 c.2.J }JI$$Z788rc+~# [U5(d [S5e[U5nUS::a [S5eSnUR S5nU(a[SUR 5-5eU(dgS n[U[5(a7[U5"54S jn[(a[U[5(aS n[U5n[[R"Xa55nU(dg[!U5nX:aU(a U/X- -XxS&U"U5v MU![ a SnSnNf=f7f) aGenerates *size*-sized chunks from *src* iterable. Unless the optional *fill* keyword argument is provided, iterables not evenly divisible by *size* will have a final chunk that is smaller than *size*. >>> list(chunked_iter(range(10), 3)) [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]] >>> list(chunked_iter(range(10), 3, fill=None)) [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, None, None]] Note that ``fill=None`` in fact uses ``None`` as the fill value. r$rz&expected a positive integer chunk sizeTfillFNz$got unexpected keyword arguments: %rcU$r&r'chks rr*chunked_iter..cscrc$URU5$r&)join)r[_seps rr*r\es DIIcNrc[U5$r&)bytesrZs rr*r\gseCjr)rrr2 ValueErrorpopKeyErrorkeysrrtype_IS_PY3rarrrPrQlen) rrRrTdo_fillfill_val postprocesssrc_iter cur_chunklcs rrOrOFs# s  .// t9D qyABBG66&> ?"'')KLL !K#z""'+Cy{B 7z#u--0KCyH ))(9:    ^ 9&Z495IcN)$$  s):D=D)CD=)D:7D=9D::D=c[US5$)a3Convenience function for calling :func:`windowed` on *src*, with *size* set to 2. >>> pairwise(range(5)) [(0, 1), (1, 2), (2, 3), (3, 4)] >>> pairwise([]) [] The number of pairs is always one less than the number of elements in the iterable passed in, except on empty inputs, which returns an empty list. )windowedrs rpairwisersts C rc[US5$)a8Convenience function for calling :func:`windowed_iter` on *src*, with *size* set to 2. >>> list(pairwise_iter(range(5))) [(0, 1), (1, 2), (2, 3), (3, 4)] >>> list(pairwise_iter([])) [] The number of pairs is always one less than the number of elements in the iterable passed in, or zero, when *src* is empty. rp) windowed_iterrrs r pairwise_iterrvs a  rc*[[X55$)zReturns tuples with exactly length *size*. If the iterable is too short to make a window of length *size*, no tuples are returned. See :func:`windowed_iter` for more. )rru)rrRs rrqrqs  c( ))rc[R"X5n[U5H"up4[U5Hn[ U5 M M$ [ U6$![ a [ /5s$f=f)aDReturns tuples with length *size* which represent a sliding window over iterable *src*. >>> list(windowed_iter(range(7), 3)) [(0, 1, 2), (1, 2, 3), (2, 3, 4), (3, 4, 5), (4, 5, 6)] If the iterable is too short to make a window of length *size*, then no window tuples are returned. >>> list(windowed_iter(range(3), 5)) [] )rPtee enumeratexrangenext StopIterationr )rrRteesrBrI_s rrurusb == #DdODAAYQ$ ; Bxs1AA)(A)c## U(d [S5eUcSUS-pO US-US-pUnX0:aUv X2- nX0:aMgg7f)zSame as :func:`frange`, but generator-based instead of returning a list. >>> tuple(xfrange(1, 3, step=0.75)) (1.0, 1.75, 2.5) See :func:`frange` for more details. step must be non-zeroN?)rb)stopstartstepcurs rxfrangersT 011 }4#:tck4#:e C *   *s :AAcU(d [S5eUcSUS-pO US-US-p[[R"X- U- 55nS/U-nU(dU$XS'[ SU5HnXES- U-XE'M U$)aA :func:`range` clone for float-based ranges. >>> frange(5) [0.0, 1.0, 2.0, 3.0, 4.0] >>> frange(6, step=1.25) [0.0, 1.25, 2.5, 3.75, 5.0] >>> frange(100.5, 101.5, 0.25) [100.5, 100.75, 101.0, 101.25] >>> frange(5, 0) [] >>> frange(5, 0, step=-1.25) [5.0, 3.75, 2.5, 1.25] rNrrrr1)rbr2mathceilr{)rrrrSretrBs rfrangers 011 }4#:tck4#:e  4<4/0 1E &5.C  F Au Ud" Jrc LUS:Xa [S5e[[XUX4S95$)amReturns a list of geometrically-increasing floating-point numbers, suitable for usage with `exponential backoff`_. Exactly like :func:`backoff_iter`, but without the ``'repeat'`` option for *count*. See :func:`backoff_iter` for more details. .. _exponential backoff: https://en.wikipedia.org/wiki/Exponential_backoff >>> backoff(1, 10) [1.0, 2.0, 4.0, 8.0, 10.0] repeatz/'repeat' supported in backoff_iter, not backoff)rSfactorjitter)rbr backoff_iter)rrrSrrs rbackoffrs3 JKK  U$*; <>> list(backoff_iter(1.0, 10.0, count=5)) [1.0, 2.0, 4.0, 8.0, 10.0] >>> list(backoff_iter(1.0, 10.0, count=8)) [1.0, 2.0, 4.0, 8.0, 10.0, 10.0, 10.0, 10.0] >>> list(backoff_iter(0.25, 100.0, factor=10)) [0.25, 2.5, 25.0, 100.0] A simplified usage example: .. code-block:: python for timeout in backoff_iter(0.25, 5.0): try: res = network_call() break except Exception as e: log(e) time.sleep(timeout) An enhancement for large-scale systems would be to add variation, or *jitter*, to timeout values. This is done to avoid a thundering herd on the receiving end of the network call. Finally, for *count*, the special value ``'repeat'`` can be passed to continue yielding indefinitely. Args: start (float): Positive number for baseline. stop (float): Positive number for maximum. count (int): Number of steps before stopping iteration. Defaults to the number of steps between *start* and *stop*. Pass the string, `'repeat'`, to continue iteration indefinitely. factor (float): Rate of exponential increase. Defaults to `2.0`, e.g., `[1, 2, 4, 8, 16]`. jitter (float): A factor between `-1.0` and `1.0`, used to uniformly randomize and thus spread out timeouts in a distributed system, avoiding rhythm effects. Positive values use the base backoff curve as a maximum, negative values use the curve as a minimum. Set to 1.0 or `True` for a jitter approximating Ethernet's time-tested backoff solution. Defaults to `False`. rzexpected start >= 0, not %rrzexpected factor >= 1.0, not %rzexpected stop >= 0zexpected stop >= start, not %rNr1rrz*count must be positive or "repeat", not %rgz%expected jitter -1 <= j <= 1, not: %r)floatrbrrlograndom) rrrSrrdenomrrBcur_rets rrrsxn %LE ;D 6]F s{6>?? |9FBCC s{-.. |9D@AA }ADIIdhhtz6:;;EAI UQYEMNN v%#%DvMN N A 8 qyG 6\FMMO;>> bucketize(range(5)) {False: [0], True: [1, 2, 3, 4]} >>> is_odd = lambda x: x % 2 == 1 >>> bucketize(range(5), is_odd) {False: [0, 2, 4], True: [1, 3]} *key* is :class:`bool` by default, but can either be a callable or a string or a list if it is a string, it is the name of the attribute on which to bucketize objects. >>> bucketize([1+1j, 2+2j, 1, 2], key='real') {1.0: [(1+1j), 1], 2.0: [(2+2j), 2]} if *key* is a list, it contains the buckets where to put each object >>> bucketize([1,2,365,4,98],key=[0,1,2,0,2]) {0: [1, 4], 1: [2], 2: [365, 98]} Value lists are not deduplicated: >>> bucketize([None, None, None, 'hello']) {False: [None, None, None], True: ['hello']} Bucketize into more than 3 groups >>> bucketize(range(10), lambda x: x % 3) {0: [0, 3, 6, 9], 1: [1, 4, 7], 2: [2, 5, 8]} ``bucketize`` has a couple of advanced options useful in certain cases. *value_transform* can be used to modify values as they are added to buckets, and *key_filter* will allow excluding certain buckets from being collected. >>> bucketize(range(5), value_transform=lambda x: x*x) {False: [0], True: [1, 4, 9, 16]} >>> bucketize(range(10), key=lambda x: x % 3, key_filter=lambda k: k % 3 != 1) {0: [0, 3, 6, 9], 2: [2, 5, 8]} Note in some of these examples there were at most two keys, ``True`` and ``False``, and each key present has a list with at least one item. See :func:`partition` for a version specialized for binary use cases. r$z&key and src have to be the same lengthc>[UTU5$r&getattrr)keys rr*bucketize..WQQ/rc US$)Nrr'r0s rr*rsQqTrz1expected key to be callable or a string or a listcU$r&r'r0s rr*rsArz*expected callable value transform functionc>T"US5$)Nr1r')r)fs rr*rs !AaD'r) rrrrrhrbziprr3 setdefaultr5) rrvalue_transform key_filterkey_funcrval key_of_valrs ` @r bucketizerXs` s  .// C   s8s3x EF F#sm#z""/ # C  !KLL% O $ $DEE#t ) Cc]  J!7!7 NN:r * 1 1/#2F G Jrc`[X5nURS/5URS/54$)aNo relation to :meth:`str.partition`, ``partition`` is like :func:`bucketize`, but for added convenience returns a tuple of ``(truthy_values, falsy_values)``. >>> nonempty, empty = partition(['', '', 'hi', '', 'bye']) >>> nonempty ['hi', 'bye'] *key* defaults to :class:`bool`, but can be carefully overridden to use either a function that returns either ``True`` or ``False`` or a string name of the attribute on which to partition objects. >>> import string >>> is_digit = lambda x: x in string.digits >>> decimal_digits, hexletters = partition(string.hexdigits, is_digit) >>> ''.join(decimal_digits), ''.join(hexletters) ('0123456789', 'abcdefABCDEF') TF)rget)rr bucketizeds r partitionrs/&3$J >>$ #Z^^E2%> >>rc*[[X55$)a ``unique()`` returns a list of unique values, as determined by *key*, in the order they first appeared in the input iterable, *src*. >>> ones_n_zeros = '11010110001010010101010' >>> ''.join(unique(ones_n_zeros)) '10' See :func:`unique_iter` docs for more details. )r unique_iter)rrs runiquers  C% &&rc#R^# [U5(d[S[U5-5eTcSnO=[T5(aTnO*[ T[ 5(aU4SjnO[ST-5e[ 5nUH'nU"U5nXS;dMURU5 Uv M) g7f)a'Yield unique elements from the iterable, *src*, based on *key*, in the order in which they first appeared in *src*. >>> repetitious = [1, 2, 3] * 10 >>> list(unique_iter(repetitious)) [1, 2, 3] By default, *key* is the object itself, but *key* can either be a callable or, for convenience, a string name of the attribute on which to uniqueify objects, falling back on identity when the attribute is not present. >>> pleasantries = ['hi', 'hello', 'ok', 'bye', 'yes'] >>> list(unique_iter(pleasantries, key=lambda x: len(x))) ['hi', 'hello', 'bye'] zexpected an iterable, not %rNcU$r&r'r0s rr*unique_iter..sQrc>[UTU5$r&rrs rr*rrr+"key" expected a string or callable, not %r)rrrfr3rrsetadd)rrrseenrBks ` rrrs" s  6cBCC { # C $ $/EKLL 5D  QK = HHQKG   s BB' B'c^TcO=[T5(aTnO*[T[5(aU4SjnO[ST-5e0n/n0nUH[nT(aW"U5OUnX;aXtU'MX;aU(aXhR U5 M@MBUR U5 XHU/Xh'M] U(dUVs/sH oUSPM n nU $UVs/sHoUPM n nU $s snfs snf)aThe complement of :func:`unique()`. By default returns non-unique/duplicate values as a list of the *first* redundant value in *src*. Pass ``groups=True`` to get groups of all values with redundancies, ordered by position of the first redundant value. This is useful in conjunction with some normalizing *key* function. >>> redundant([1, 2, 3, 4]) [] >>> redundant([1, 2, 3, 2, 3, 3, 4]) [2, 3] >>> redundant([1, 2, 3, 2, 3, 3, 4], groups=True) [[2, 2], [3, 3, 3]] An example using a *key* function to do case-insensitive redundancy detection. >>> redundant(['hi', 'Hi', 'HI', 'hello'], key=str.lower) ['Hi'] >>> redundant(['hi', 'Hi', 'HI', 'hello'], groups=True, key=str.lower) [['hi', 'Hi', 'HI']] *key* should also be used when the values in *src* are not hashable. .. note:: This output of this function is designed for reporting duplicates in contexts when a unique input is desired. Due to the grouped return type, there is no streaming equivalent of this function for the time being. c>[UTU5$r&rrs rr*redundant..rrrr1)r3rrrr5) rrgroupsrrredundant_orderredundant_groupsrBrrs ` r redundantrsD { # C $ $/EKLL DO HQKA =G$$'..q1 &&q)'+wl # />?!"1%? J-<>> one((True, False, False)) True >>> one((True, False, True)) >>> one((0, 0, 'a')) 'a' >>> one((0, False, None)) >>> one((True, True), default=False) False >>> bool(one(('', 1))) True >>> one((10, 20, 30, 42), key=lambda i: i > 40) 42 See `Martín Gaitán's original repo`_ for further use cases. .. _Martín Gaitán's original repo: https://github.com/mgaitan/one .. _XOR: https://en.wikipedia.org/wiki/Exclusive_or rpr1r)rrPrQr rh)rdefaultroness roner0s9>    !115 6D$i1n471'1rc,[[X 5U5$)a+Return first element of *iterable* that evaluates to ``True``, else return ``None`` or optional *default*. Similar to :func:`one`. >>> first([0, False, None, [], (), 42]) 42 >>> first([0, False, None, [], ()]) is None True >>> first([0, False, None, [], ()], default='ohai') 'ohai' >>> import re >>> m = first(re.match(regex, 'abc') for regex in ['b.*', 'a(.*)']) >>> m.group(1) 'bc' The optional *key* argument specifies a one-argument predicate function like that used for *filter()*. The *key* argument, if supplied, should be in keyword form. For example, finding the first even number in an iterable: >>> first([1, 1, 3, 4, 5], key=lambda x: x % 2 == 0) 4 Contributed by Hynek Schlawack, author of `the original standalone module`_. .. _the original standalone module: https://github.com/hynek/first )r|r )r=rrs rfirstrSs4 s%w //rc## UHIn[U[5(a-[U[5(d[U5HnUv M MEUv MK g7f)z``flatten_iter()`` yields all the elements from *iterable* while collapsing any nested iterables. >>> nested = [[1, 2], [[3], [4, 5]]] >>> list(flatten_iter(nested)) [1, 2, 3, 4, 5] N)rrr flatten_iter)r=itemsubitems rrrpsE dH % %jz.J.J'- .J sAAc*[[U55$)z``flatten()`` returns a collapsed list of all the elements from *iterable* while collapsing any nested iterables. >>> nested = [[1, 2], [[3], [4, 5]]] >>> flatten(nested) [1, 2, 3, 4, 5] )rr)r=s rflattenrs  X& ''rcn^[U5nT[La [UT5m[U4SjU55$)a``same()`` returns ``True`` when all values in *iterable* are equal to one another, or optionally a reference value, *ref*. Similar to :func:`all` and :func:`any` in that it evaluates an iterable and returns a :class:`bool`. ``same()`` returns ``True`` for empty iterables. >>> same([]) True >>> same([1]) True >>> same(['a', 'a', 'a']) True >>> same(range(20)) False >>> same([[], []]) True >>> same([[], []], ref='test') False c3,># UH oT:Hv M g7fr&r').0rrefs r same..s.XcczXs)rr r|all)r=rrAs ` rsamers2*H~H f}8S! .X. ..rcX4$r&r'pathrvalues r default_visitrs :rc\[U[5(aUS4$[U[5(aUR5[ U54$[U[ 5(aUR5[ U54$[U[5(aUR5[ U54$US4$r/)rrr __class__rrrzrrs r default_enterrs%$$e| E7 # # )E"222 E8 $ $ )E"222 E3   )E"222e|rc Un[U[5(aURU5 U$[U[5(a*UVVs/sHupgUPM nnnUR U5 U$[U[5(a*UVVs/sHupgUPM nnnURU5 U$[S[U5-5es snnf![ a UR U5nU$f=fs snnf![ a UR U5nU$f=f)Nzunexpected iterable type: %r) rrupdaterextendAttributeErrorrr RuntimeErrorrf) rr old_parent new_parent new_itemsrrBvvalss r default_exitrs C*g&&)$ J J ) )'(idai( -   d # J J $ $'(idai( -   d # J9D>> from pprint import pprint >>> reviews = {'Star Trek': {'TNG': 10, 'DS9': 8.5, 'ENT': None}, ... 'Babylon 5': 6, 'Dr. Who': None} >>> pprint(remap(reviews, lambda p, k, v: v is not None)) {'Babylon 5': 6, 'Star Trek': {'DS9': 8.5, 'TNG': 10}} Notice how both Nones have been removed despite the nesting in the dictionary. Not bad for a one-liner, and that's just the beginning. See `this remap cookbook`_ for more delicious recipes. .. _this remap cookbook: http://sedimental.org/remap.html remap takes four main arguments: the object to traverse and three optional callables which determine how the remapped object will be created. Args: root: The target object to traverse. By default, remap supports iterables like :class:`list`, :class:`tuple`, :class:`dict`, and :class:`set`, but any object traversable by *enter* will work. visit (callable): This function is called on every item in *root*. It must accept three positional arguments, *path*, *key*, and *value*. *path* is simply a tuple of parents' keys. *visit* should return the new key-value pair. It may also return ``True`` as shorthand to keep the old item unmodified, or ``False`` to drop the item from the new structure. *visit* is called after *enter*, on the new parent. The *visit* function is called for every item in root, including duplicate items. For traversable values, it is called on the new parent object, after all its children have been visited. The default visit behavior simply returns the key-value pair unmodified. enter (callable): This function controls which items in *root* are traversed. It accepts the same arguments as *visit*: the path, the key, and the value of the current item. It returns a pair of the blank new parent, and an iterator over the items which should be visited. If ``False`` is returned instead of an iterator, the value will not be traversed. The *enter* function is only called once per unique value. The default enter behavior support mappings, sequences, and sets. Strings and all other iterables will not be traversed. exit (callable): This function determines how to handle items once they have been visited. It gets the same three arguments as the other functions -- *path*, *key*, *value* -- plus two more: the blank new parent object returned from *enter*, and a list of the new items, as remapped by *visit*. Like *enter*, the *exit* function is only called once per unique value. The default exit behavior is to simply add all new items to the new parent, e.g., using :meth:`list.extend` and :meth:`dict.update` to add to the new parent. Immutable objects, such as a :class:`tuple` or :class:`namedtuple`, must be recreated from scratch, but use the same type as the new parent passed back from the *enter* function. reraise_visit (bool): A pragmatic convenience for the *visit* callable. When set to ``False``, remap ignores any errors raised by the *visit* callback. Items causing exceptions are kept. See examples for more details. remap is designed to cover the majority of cases with just the *visit* callable. While passing in multiple callables is very empowering, remap is designed so very few cases should require passing more than one function. When passing *enter* and *exit*, it's common and easiest to build on the default behavior. Simply add ``from boltons.iterutils import default_enter`` (or ``default_exit``), and have your enter/exit function call the default behavior before or after your custom logic. See `this example`_. Duplicate and self-referential objects (aka reference loops) are automatically handled internally, `as shown here`_. .. _this example: http://sedimental.org/remap.html#sort_all_lists .. _as shown here: http://sedimental.org/remap.html#corner_cases z visit expected callable, not: %rz enter expected callable, not: %rzexit expected callable, not: %r reraise_visitTz unexpected keyword arguments: %rr'NzDenter should return a tuple of (new_parent, items_iterator), not: %rFr1z!expected remappable root, not: %r) r3rrcreidr r5rreversedr_orig_default_visit Exception IndexError)rootvisitenterexitkwargsrrregistrystacknew_items_stackrrid_valuerrrres visited_items rremaprs^H E??:UBCC E??:UBCC D>>9D@AAJJ5M :V[[]JKKdD\NEDO YY[ e9 + */ 'CZ*~H-113ODJIFE!&X "#  !&E5)C C(+% %%/"&&bz2$FND kCU+CDELL$y/!:; ' 'A!BCC C( $ #  $ H?$FG G Hs*G G+,HG(+HHHc*\rSrSrSrSrSrSrSrg)PathAccessErrori{zAn amalgamation of KeyError, IndexError, and TypeError, representing what can occur when looking up a path in a nested object. c(XlX lX0lgr&)excsegr)selfrrrs r__init__PathAccessError.__init__s rcURRnU<SUR<SUR<SUR<S3$)N(z, ))r__name__rrr)rcns r__repr__PathAccessError.__repr__s, ^^ $ $#%txx499EErcZSUR<SUR<SUR<3$)Nzcould not access z from path z , got error: )rrrrs r__str__PathAccessError.__str__s88TYY2 3r)rrrN) r  __module__ __qualname____firstlineno____doc__rr r__static_attributes__r'rrrr{s F3rrc [U[5(aURS5nUnUHnX4nM U$![[4an[ XTU5eSnAf[ avn[U5nX4nSnAMM![[[[ 4a> [U5(d![ S[U5R-5n[ XTU5ef=fSnAff=f![ a U[LaeUs$f=f)a0Retrieve a value from a nested object via a tuple representing the lookup path. >>> root = {'a': {'b': {'c': [[1], [2], [3]]}}} >>> get_path(root, ('a', 'b', 'c', 2, 0)) 3 The path format is intentionally consistent with that of :func:`remap`. One of get_path's chief aims is improved error messaging. EAFP is great, but the error messages are not. For instance, ``root['a']['b']['c'][2][1]`` gives back ``IndexError: list index out of range`` What went out of range where? get_path currently raises ``PathAccessError: could not access 2 from path ('a', 'b', 'c', 2, 1), got error: IndexError('list index out of range',)``, a subclass of IndexError and KeyError. You can also pass a default that covers the entire operation, should the lookup fail at any level. Args: root: The target nesting of dictionaries, lists, or other objects supporting ``__getitem__``. path (tuple): A list of strings and integers to be successively looked up within *root*. default: The value to be returned should any ``PathAccessError`` exceptions be raised. .Nz%r object is not indexable) rrr"rdrrrr2rbrrfr r )rrrrrrs rget_pathrsB$ ##zz# CC :h( J#j) 6%c55 ::c(C(C"Hj)D:&s++'(D*.s)*<*<)=>)#D99 : :  f  sPC:CC A C$A93C9ACCCCC54C5cg)NTr')prrs rr*r*srcn^^^/m[T5(d[ST-5eUUU4Sjn[XS9 T$)aThe :func:`research` function uses :func:`remap` to recurse over any data nested in *root*, and find values which match a given criterion, specified by the *query* callable. Results are returned as a list of ``(path, value)`` pairs. The paths are tuples in the same format accepted by :func:`get_path`. This can be useful for comparing values nested in two or more different structures. Here's a simple example that finds all integers: >>> root = {'a': {'b': 1, 'c': (2, 'd', 3)}, 'e': None} >>> res = research(root, query=lambda p, k, v: isinstance(v, int)) >>> print(sorted(res)) [(('a', 'b'), 1), (('a', 'c', 0), 2), (('a', 'c', 2), 3)] Note how *query* follows the same, familiar ``path, key, value`` signature as the ``visit`` and ``enter`` functions on :func:`remap`, and returns a :class:`bool`. Args: root: The target object to search. Supports the same types of objects as :func:`remap`, including :class:`list`, :class:`tuple`, :class:`dict`, and :class:`set`. query (callable): The function called on every object to determine whether to include it in the search results. The callable must accept three arguments, *path*, *key*, and *value*, commonly abbreviated *p*, *k*, and *v*, same as *enter* and *visit* from :func:`remap`. reraise (bool): Whether to reraise exceptions raised by *query* or to simply drop the result that caused the error. With :func:`research` it's easy to inspect the details of a data structure, like finding values that are at a certain depth (using ``len(p)``) and much more. If more advanced functionality is needed, check out the code and make your own :func:`remap` wrapper, and consider `submitting a patch`_! .. _submitting a patch: https://github.com/mahmoud/boltons/pulls z query expected callable, not: %rc>T"XU5(aTRX4-U45 [XU5$![a T(aeN f=fr&)r5rr)rrrqueryreraisers rrresearch..entersU T&& D6M512T..  s$3AA)r)r3rr)rrr rrs `` @rresearchr"s9T C E??:UBCC/ $ JrcN\rSrSrSrS SjrSrSr\(aSr OSr \ r Sr g ) GUIDeratori aThe GUIDerator is an iterator that yields a globally-unique identifier (GUID) on every iteration. The GUIDs produced are hexadecimal strings. Testing shows it to be around 12x faster than the uuid module. By default it is also more compact, partly due to its default 96-bit (24-hexdigit) length. 96 bits of randomness means that there is a 1 in 2 ^ 32 chance of collision after 2 ^ 64 iterations. If more or less uniqueness is desired, the *size* argument can be adjusted accordingly. Args: size (int): character length of the GUID, defaults to 24. Lengths between 20 and 36 are considered valid. The GUIDerator has built-in fork protection that causes it to detect a fork on next iteration and reseed accordingly. cXlUS:dUS:a [S5eSSKnURUl[ R "5UlUR5 g)N$zexpected 20 < size <= 36r)rRrbhashlibsha1_sha1rPrSreseed)rrRr(s rrGUIDerator.__init__!sE "9r 78 8\\ __&  rc pSSKn[R"5UlSR [ UR5UR 5=(d S[ [R"55[R"[R"S5S5RS5/5Ul g)Nr-s  hex_codecascii) socketosgetpidpidr^str gethostnametimecodecsencodeurandomdecodesalt)rr2s rr+GUIDerator.reseed*sy99;HHc$((m$002Eo!$))+.$mmBJJqM,799? JK  rcU$r&r'rs r__iter__GUIDerator.__iter__6s rc8[R"5UR:waUR5 UR[ [ UR55-RS5nURU5R5SURnU$)Nutf8) r3r4r5r+r=r6r|rSr:r* hexdigestrR)r target_bytes hash_texts r__next__GUIDerator.__next__:slyy{dhh&  IID,<(==EEfML <0::Y[$))M Mr)r*rSr5r=rRN)) r rrrrrr+r@rgrGr|rr'rrr$r$ s-&   M Drr$cR^\rSrSrSr\(aU4SjrOU4SjrSr\rSr U=r $)SequentialGUIDeratoriJaOMuch like the standard GUIDerator, the SequentialGUIDerator is an iterator that yields a globally-unique identifier (GUID) on every iteration. The GUIDs produced are hexadecimal strings. The SequentialGUIDerator differs in that it picks a starting GUID value and increments every iteration. This yields GUIDs which are of course unique, but also ordered and lexicographically sortable. The SequentialGUIDerator is around 50% faster than the normal GUIDerator, making it almost 20x as fast as the built-in uuid module. By default it is also more compact, partly due to its 96-bit (24-hexdigit) default length. 96 bits of randomness means that there is a 1 in 2 ^ 32 chance of collision after 2 ^ 64 iterations. If more or less uniqueness is desired, the *size* argument can be adjusted accordingly. Args: size (int): character length of the GUID, defaults to 24. Note that with SequentialGUIDerator there is a chance of GUIDs growing larger than the size configured. The SequentialGUIDerator has built-in fork protection that causes it to detect a fork on next iteration and reseed accordingly. c&>[[U] 5 URURR S55R 5n[USURS5Ul U=RSURS-S- --sl g)NrCr1rp) superrLr+r*r=r:rDr2rRrr start_strrs rr+SequentialGUIDerator.reseedfsp & 4 6 499#3#3F#;<FFHIYz 2B7DJ JJ1$))a-1!45 6Jrc>[[U] 5 URUR5R 5n[ USURS5UlU=RSURS-S- --slg)NrNr1rOrp) rPrLr+r*r=rDr2rRrrQs rr+rSlse & 4 6 499-779IYz 2B7DJ JJ1$))a-1!45 6Jrc[R"5UR:waUR5 S[ UR 5UR --$)Nz%x)r3r4r5r+r|rSrrs rrGSequentialGUIDerator.__next__rs: 99;$(( " KKMtDJJ'$**455r)r) r rrrrrgr+rGr|r __classcell__)rs@rrLrLJs#4 7  7 6 DrrLc^^^T=(d /mT=(d /mT=(d Sm[U5nUVs/sH/nT(aT"U5T;aMT(aT"U5T;aM-UPM1 nnURTUS9 T(a.[UVs/sHnT"U5T;dMUPM snUU4SjS9mT(a.[UVs/sHnT"U5T;dMUPM snUU4SjS9mTU-T-$s snfs snfs snf)aFor when you care about the order of some elements, but not about others. Use this to float to the top and/or sink to the bottom a specific ordering, while sorting the rest of the elements according to normal :func:`sorted` rules. >>> soft_sorted(['two', 'b', 'one', 'a'], first=['one', 'two']) ['one', 'two', 'a', 'b'] >>> soft_sorted(range(7), first=[6, 15], last=[2, 4], reverse=True) [6, 5, 3, 1, 0, 2, 4] >>> import string >>> ''.join(soft_sorted(string.hexdigits, first='za1', last='b', key=str.lower)) 'aA1023456789cCdDeEfFbB' Args: iterable (list): A list or other iterable to sort. first (list): A sequence to enforce for elements which should appear at the beginning of the returned list. last (list): A sequence to enforce for elements which should appear at the end of the returned list. key (callable): Callable used to generate a comparable key for each item to be sorted, same as the key in :func:`sorted`. Note that entries in *first* and *last* should be the keys for the items. Defaults to passthrough/the identity function. reverse (bool): Whether or not elements not explicitly ordered by *first* and *last* should be in reverse order or not. Returns a new list in sorted order. cU$r&r'r0s rr*soft_sorted..sArrreversec2>TRT"U55$r&index)r)rrs rr*rZs%++VYZ[V\J]rrc2>TRT"U55$r&r^)r)rlasts rr*rZs SVWXSYHZr)rsortsorted)r=rrbrr\seqr)others ``` r soft_sortedrg~s@ KRE :2D +C x.C ^1Us1vQDSQRVW[^QE ^ JJ3J( 3:3a#a&E/3:@]^ #8#QQ4q#8>Z[ 5=4  _;8s)C/C/%C/ C4!C4C9C9c~^"U4SjS[5nTb[T5(d[ST-5e[XUS9$)aA version of :func:`sorted` which will happily sort an iterable of heterogenous types and return a new list, similar to legacy Python's behavior. >>> untyped_sorted(['abc', 2.0, 1, 2, 'def']) [1, 2.0, 2, 'abc', 'def'] Note how mutually orderable types are sorted as expected, as in the case of the integers and floats above. .. note:: Results may vary across Python versions and builds, but the function will produce a sorted list, except in the case of explicitly unorderable objects. c,>\rSrSrSrSrU4SjrSrg) untyped_sorted.._WrapperircXlgr&r)rrs rr)untyped_sorted.._Wrapper.__init__sHrcb>TbT"UR5O URnTbT"UR5O URnX!:nU$![aY [U5R[ [U55U4[U5R[ [U55U4:nU$f=fr&)rrrfr r)rrfrrrs r__lt__'untyped_sorted.._Wrapper.__lt__s#&?#dhh-C&)oC N599E JkJ JS **BtCyM3?;//DK%HIJ JsA AB.-B.N)r rrrslotsrrnrr`sr_Wrapperrjs   rrqz5expected function or callable object for key, not: %rr[)objectr3rrd)r=rr\rqs ` runtyped_sortedrssE$6  x}}O  (' ::r)NNr&)Nr)Ng@Fr/)NNNF)Prr3rr8r9rrPcollections.abcrrrrr ImportError collections typeutilsrr r rrfuture_builtinsr r rgr6rarunicoderranger{rrrr"rr?r;rErDrLrKrVrOrsrvrqrurrrrboolrrrrrrrrrrrrrrrrdrrrrr"r$rL guid_iter seq_guid_iterrgrsr'rrr~sB  HKK ' 8 $F .K &G$ ? @ 0E P 4$ 38 2 F9.+ \  ! *0,><"[ |TdM`?. '! H<~ 2F0: (/6 $  .$=|dN3h I3&"(9x.u9F::z-:-` L $& + \&;P,HGGGH(K XFGuJGLD& s4D9EE/9EEE,+E,/FF