rh(SSKJr /SQrSSKrSSKrSSSSS.SjrSSSSS.SjrS rS rS r "S S \R5r \ S:XaSSK r \ R"\ 5 gg)) annotations) runParallelrunNonParallelcpusNFupdateFunctionupdateMultiplyunpackIterableupdateSendsIterablec 4^^^^^^^^[5mTS:Xd$[R"5R(a[ TUTTUTS9$[ T5mSnTcTm/mUUUUUUU4SjnU"S5 SSKJnJn U"TS9n U "U5mUT:an[UTT--T5n [Xk5n U(aU "UU4SjU 55n OU "UU4S jU 55n U nTRU 5 U"U5 UT:aMnSSS5 T$!,(df  T$=f) aE runs parallelFunction over iterable in parallel, optionally calling updateFunction after each common.cpus * updateMultiply calls. Setting updateMultiply too small can make it so that cores wait around when they could be working if one CPU has a particularly hard task. Setting it too high can make it seem like the job has hung. updateFunction should take three arguments: the current position, the total to run, and the most recent results. It does not need to be pickleable, and in fact, a bound method might be very useful here. Or updateFunction can be "True" which just prints a generic message. If unpackIterable is True then each element in iterable is considered a list or tuple of different arguments to parallelFunction. If updateSendsIterable is True then the update function will get the iterable content, after the output. As of Python 3, partial functions are pickleable, so if you need to pass the same arguments to parallelFunction each time, make it a partial function before passing it to runParallel. Note that parallelFunction, iterable's contents, and the results of calling parallelFunction must all be pickleable, and that if pickling the contents or unpickling the results takes a lot of time, you won't get nearly the speedup from this function as you might expect. The big culprit here is definitely music21 streams. >>> files = ['bach/bwv66.6', 'schoenberg/opus19', 'AcaciaReel'] >>> def countNotes(fn): ... c = corpus.parse(fn) # this is the slow call that is good to parallelize ... return len(c.recurse().notes) >>> #_DOCS_SHOW outputs = common.runParallel(files, countNotes) >>> outputs = common.runNonParallel(files, countNotes) #_DOCS_HIDE cannot pickle doctest funcs. >>> outputs [165, 50, 131] Set updateFunction=True to get an update every 3 * numCpus (-1 if > 2) >>> #_DOCS_SHOW outputs = common.runParallel(files, countNotes, updateFunction=True) >>> outputs = common.runNonParallel(files, countNotes, updateFunction=True) #_DOCS_HIDE Done 0 tasks of 3 Done 3 tasks of 3 With a custom updateFunction that gets each output: >>> def yak(position, length, output): ... print("%s:%s %s is a lot of notes!" % (position, length, output)) >>> #_DOCS_SHOW outputs = common.runParallel(files, countNotes, updateFunction=yak) >>> outputs = common.runNonParallel(files, countNotes, updateFunction=yak) #_DOCS_HIDE 0:3 165 is a lot of notes! 1:3 50 is a lot of notes! 2:3 131 is a lot of notes! Or with updateSendsIterable, we can get the original files data as well: >>> def yik(position, length, output, fn): ... print("%s:%s (%s) %s is a lot of notes!" % (position, length, fn, output)) >>> #_DOCS_SHOW outputs = common.runParallel(files, countNotes, updateFunction=yik, >>> outputs = common.runNonParallel(files, countNotes, updateFunction=yik, #_DOCS_HIDE ... updateSendsIterable=True) 0:3 (bach/bwv66.6) 165 is a lot of notes! 1:3 (schoenberg/opus19) 50 is a lot of notes! 2:3 (AcaciaReel) 131 is a lot of notes! unpackIterable is useful for when you need to send multiple values to your function call as separate arguments. For instance, something like: >>> def pitchesAbove(fn, minPitch): # a two-argument function ... c = corpus.parse(fn) # again, the slow call goes in the function ... return len([p for p in c.pitches if p.ps > minPitch]) >>> inputs = [('bach/bwv66.6', 60), ... ('schoenberg/opus19', 72), ... ('AcaciaReel', 66)] >>> #_DOCS_SHOW outputs = common.runParallel(inputs, pitchesAbove, unpackIterable=True) >>> outputs = common.runNonParallel(inputs, pitchesAbove, unpackIterable=True) #_DOCS_HIDE >>> outputs [99, 11, 123] rrNc >TSLa[UT/5n[SUST35 gTS;aX[UT T-- U5HAnUS:aM U[T5:aSnOTUnT SLa T"UTU5 M3T"UTUTU5 MC gg)NTDone tasks of FNrFminprintrangelen) ii tasksDone thisPosition thisResult iterLengthiterablenumCpus resultsListr r r s Q/home/james-whalen/.local/lib/python3.13/site-packages/music21/common/parallel.py callUpdaterunParallel..callUpdate}s T !R,-I E)Jzl; < = 0 %bNW,D&Er J !#3{#33!%J!,\!:J&%/"<ZH"<ZR^I_`!K1)Paralleldelayed)n_jobsc34># UH nT"TU6v M g7fN.0i delayFunctionrs r runParallel..sHx!-!5xsc3:># UHnT"TU5v M g7fr(r)r*s r r.r/sGh- 44hs) rmultiprocessingcurrent_processdaemonrrjoblibr$r%rrextend)rparallelFunctionr r r r totalRunr!r$r%para endPositionrangeGen_rr-rrrs` `` ` @@@@r rrs"hfG!|668??h(8-;-;-;2E G G XJH#Kaa&qM(  !T 01 #h>)AA:NKX3HHxHHGhGG"H   r " x # "  " ! s A;D Dc^^^^^ ^ [T5m /m U UU UUU4SjnU"S5 [T 5H:nU(a U"TU6nO U"TU5nT RU5 U"US-5 M< T $)z This is intended to be a perfect drop in replacement for runParallel, except that it runs on one core only, and not in parallel. Used automatically if we're already in a parallelized function. c>UT-S:wagTSLa[UT/5n[SUST35 gTS;aU[UT- U5HAnUS:aM U[T5:aSnOTUnT SLa T"UTU5 M3T"UTUTU5 MC gg)NrTrrrFr) rrrrrrrr r r s r r!"runNonParallel..callUpdates  ! #  T !R,-I E)Jzl; < = 0 %b>&92 > !#3{#33!%J!,\!:J&%/"<ZH"<ZR^I_`!?1r#rr)rrappend) rr6r r r r r!r,r;rrs ` `` ` @@r rrsxXJKaa,qM :  !8A;/B!(1+.B21q5 r#cF[R"5nUS:aUS- $U$)z^ Returns the number of CPUs or if >= 3, one less (to leave something out for multiprocessing) rr)r1 cpu_count)cpuCounts r rrs(((*H1}!|r#cvSSKJn URU5n[UR 5R 5$)Nr)corpus)music21rDparserrecursenotes)fnrDcs r _countNrKs* RA qyy{  !!r#c US:agUS;agg)NrFz bach/bwv66.6zschoenberg/opus19 AcaciaReelTr))r,rIs r _countUnpackedrOsAv DD r#c&\rSrSrSrSrSrSrg)Testc@/SQnSSKJnJn [X5nUR U/SQ5 [XUR S9 [XUR SS9 [[[U55USS9nUR [U5S 5 URS U5 g) NrMr)rKrO2)r T)r r )r rF) music21.common.parallelrKrOr assertEqual_customUpdate1_customUpdate2list enumerater assertNotIn)selffilesrKrOoutputpasseds r &x_figure_out_segfault_testMultiprocess+Test.x_figure_out_segfault_testMultiprocesssCCU, 0E#'#6#6 8E#'#6#6(, .T)E"23^,02 Va( 'r#ctURUS5 URUS5 URU/SQ5 g)NrrT)rY assertLessassertIn)r_r,totalras r rZTest._customUpdate1 s. " 1 fn-r#c,URU/SQ5 g)NrM)rg)r_r, unused_total unused_outputrIs r r[Test._customUpdate2s bMNr#r)N)__name__ __module__ __qualname____firstlineno__rcrZr[__static_attributes__r)r#r rQrQs(". Or#rQ__main__) __future__r__all__r1unittestrrrrKrOTestCaserQrnrEmainTestr)r#r rys#   $A$%K^#'q"'U-`*" O8  O8 z Tr#