6biT}HSrSSKrSSKJs Jr SSKJr SSKJ r SSK J r SSK Jr SSKJr SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SrSSjrSr SSjr!SSjr"SSjr#"SS\RH5r%Sr&"SS\RH5r'g)zEPart of the TF-Keras training engine related to distributed training.N)backend) callbacks)distribute_coordinator_utils)distributed_training_utils_v1)partial_batch_padding_handler)training_arrays_v1)training_utils_v1)Progbar)ModeKeys) input_lib) tf_loggingcURU5nURURURUR4$N)_make_execution_functioninputsoutputs updates_opsession_kwargs)modelmode exec_funcs e/home/james-whalen/.local/lib/python3.13/site-packages/tf_keras/src/engine/training_distributed_v1.py_per_replica_execution_functionr#s@..t4I   cUR(a[R"XX#US9 g[R"XX#U5 g)N)rtargets)_compile_distribution dist_utilsclone_model_on_replicas_build_distributed_network)strategyrrrrs r _build_modelr"-s7 ""** T'  -- T7 rc ^^^^UUUU4SjnU$)a$Create step fn. Args: model: a TF-Keras Model instance. mode: One of ModeKeys.TRAIN/ModeKeys.TEST/ModeKeys.PREDICT. strategy: a `tf.distribute.Strategy` instance. output_labels: the output labels for the step function. Returns: A step function to run by `tf.distribute.Strategy`. c>[U[[45(a[U5S:XaUupOSn[U[5(aTR Vs/sHo1UPM nn[ TTTX5 TRR[[R"TT5T4S9unnnn[R"TUUUU5unn n n [R"UU 4U S[T5-S-S.U D6n [!TU R"5HfupU S:Xa%[$R&R(R*nO$[$R&R(R,nUR/XU5 Mh U R0$s snf)z"A step fn that returns update ops.Nargs distributed_ _function)updatesnameloss) isinstancetuplelistlendict_feed_input_namesr"extendedcall_for_each_replicarrget_distributed_model unwrap_valuesrfunctionstrziprtf distributeReduceOpSUMMEANset_last_step_outputr)ctxrr input_namegrouped_inputsgrouped_outputsgrouped_updatesgrouped_session_args all_inputs all_outputs all_updatesall_session_args combined_fnlabeloutput reduce_oprr output_labelsr!s r_step_fn%_make_train_step_fn.._step_fnEs fudm , ,V1A$OFGG fd # #5:5L5L5Lzz"5L  XudF<    3 3 +225$?F 4       $ $          &&    #d)+k9    ! 0C0CDMEMM2266 MM2277  $ $UI >E%%%_sF)rrr!rNrOs```` r_make_train_step_fnrR8s=&=&~ Orc  [Rn URn [XkRR 5n [ R"U SSS9n [R"X5n[R"U SS9nUR5 UR=(d /n[U[RU U5n0n[R"S5US'UR!5HKnUR#5n[R$"UR&UR(5UUR*'MM U RR-UUU US9nUR.nUR0n[3U5nUR4(a[R6"X 5 [8R:"UUUUUUS U S 9nU RR /XkRR --nXkRR -(a'UR=XkRR -5 [?U5nURAU 5 URCXZ5n[EXR5GHn[RF"U5 URIU5 0nS nS nS nUU:aUUn USU S .n!URKU SUU!5 UbU U:wa5[ RL"5ROU RQU 55 U n[ RR"UU/5un"n#U!R]U#5 URKU SUU!5 UU -nUS- nUR^R`(aOUU:aMU(a[bRd"U U5(a[XRf"SU5 UR4(a%[Rh"U[R5 [kUUUUUS9n$[mU$[n5(dU$/n$[qUU$5Hun%n&U&USU%-'M URsUU5 UR^R`(dGM O SUl:URwU 5 UR4(a%[Rh"U[R5 URyS S S 5 URz$![RTRVa! [XRZ"SU-U-5 GMf=f)aFit loop for training with TPU tf.distribute.Strategy. Args: model: TF-Keras Model instance. dataset: Dataset that returns inputs and targets epochs: Number of times to iterate over the data verbose: Integer, Verbosity mode, 0, 1 or 2 callbacks: List of callbacks to be called during training initial_epoch: Epoch at which to start training (useful for resuming a previous training run) steps_per_epoch: Total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch. Ignored with the default value of `None`. val_dataset: Dataset for validation data. validation_steps: Number of steps to run validation for (only if doing validation from data tensors). Ignored with the default value of `None`. validation_freq: Only relevant if validation data is provided. Integer or `collections.abc.Container` instance (e.g. list, tuple, etc.). If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. `validation_freq=2` runs validation every 2 epochs. If a Container, specifies the epochs on which to run validation, e.g. `validation_freq=[1, 2, 10]` runs validation at the end of the 1st, 2nd, and 10th epochs. Returns: Returns `None`. Raises: ValueError: in case of invalid arguments. int32 steps_per_run)valuedtyper+r!learning_phasegcAr,) iterationsinitial_loop_valuessteps do_validationepochssteps_per_epochverbose count_moderrN)batchsize num_stepsbeginzYour dataset iterator ran out of data; interrupting training. Make sure that your dataset can generate at least `steps_per_epoch * epochs` batches (in this case, %d batches).endz#Running validation at fit epoch: %s)r]rbrval_T)>r TRAIN_distribution_strategyminr3rUrvariabler get_iteratordistributed_scope __enter__ metrics_namesrRr:constant_get_training_eval_metricsresultzerosshaperWr+"experimental_run_steps_on_iteratorrun_oplast_step_outputsboolr"_copy_weights_to_distributed_modelcbksconfigure_callbacksappendr0_call_begin_hook#_maybe_load_initial_epoch_from_ckptrange_reset_metricson_epoch_begin_call_batch_hook get_sessionrunassignbatch_get_valueerrorsOutOfRangeErrorloggingwarningupdater stop_trainingr should_run_validationinfo_copy_weights_to_original_modelexperimental_tpu_test_loopr-r/r9 on_epoch_end_successful_loop_finish_call_end_hook__exit__history)'rdatasetr`rbr initial_epochra val_datasetvalidation_stepsvalidation_freqrcurrent_strategyiteration_valuerUiteratorscope out_labelsstep_fnr\mtensorr@train_opoutput_tensorsr_ steps_to_run target_stepsepoch epoch_logs step_indexprev_step_count current_step step_count batch_logs_rval_outsrKval_outs' rexperimental_tpu_fit_looprsV >>D3322@@O$$W?M &&wAH  ( (!! E OO$$*J! x~~/G "$++c"2  - - /&(hhv||V\\&JAFF#0  # # F F / G C zzH**N)*M ""55eB(( #'  I%--;;<44BBBL22@@@ 77EE E |$L t$==M}-!!%(  '   \)%l3J#'J  & &tWj* M&**G##%))-*>*>z*JK", $44h5OP 7   g &  & &tUJ K#j0J A L,,=\)@ .DD U   LL> F**::8>>2&# Hh--$:"%j(";w-4 6E>*#< uj1 ?? ( ( ( G.H%)E! T" ""225(..I NN4t$ ==o99,, :& &   s<R>>:S=<S=c ^^[RmTRn[R"X5n[R "USS9nUR 5 TRnUU4Sjn UR5n URX4S9n 0n [X5HfupU S:Xa%[RRRnO$[RRRnUR!XSS9X'Mh [R""[%U R'555nUS:a [)US 9nTR*(a[R,"TT5 [R."T5 [0R2"UTS SUUS [RS 9nUR5T5 S /[7TR5-nUbUnO [9S5eSnUU:aUSS.nUR;TSUU5 [<R>"UU /5unn[KTR5H&unn US:XaUU==UU - ss'MUU UU'M( URMTUUT5nUR;TSUU5 US:XaWROUS-5 US- nUU:aMUS:aWROU5 URQT5 URSSSS5 [7U5S:a US==U-ss'[7U5S:XaUS$U$![R@RBa0 SREU5n[FRH"SU-5 UnMf=f)aTest loop for evaluating with TPU tf.distribute.Strategy. Args: model: TF-Keras Model instance. dataset: Dataset for input data. verbose: Integer, Verbosity mode 0 or 1. steps: Total number of steps (batches of samples) before declaring predictions finished. Ignored with the default value of `None`. callbacks: List of callbacks to be called during training Returns: Scalar loss (if the model has a single output and no metrics) or list of scalars (if the model has multiple outputs and/or metrics). The attribute `model.metrics_names` will give you the display labels for the outputs. rrYc>[U[[45(a[U5S:XaUupOSn[R R 5R[TTX4S9 [[R"TT5T5up#pB[R"U/5 UVs/sHn[R"U5PM snsSSS5 $s snf!,(df  g=f)z-A fn that returns output of single test step.r%Nr&)r-r.r/r0r:r;get_replica_context merge_callr"rrr5control_dependenciesidentity)rrrrr*outrrs r _test_step_fn1experimental_tpu_test_loop.._test_step_fngs fudm , ,V1A$OFGG MM - - / : :E4#A ;  $C  , ,UD 94$  W $ $gY /078BKK$80 /80 /s!C& CCC C%r&r,NaxisrXtargetFr]r^TNumber of steps could not be inferred from the data, please pass the steps argument.rdrerg]Make sure that your dataset can generate at least `steps` batches (in this case, {} batches).@Your dataset iterator ran out of data; interrupting evaluation. rh)*r TESTrkrrnrorprqget_nextrr9r:r;r<r=r>reducegroupr/valuesr rr{rr|r}rr0 ValueErrorrrrrrformatrr enumerate make_logsrrr)rrrbr]rrrrrrtest_input_dataper_replica_outputsrrKrLrMtest_opprogbaroutsrrrr batch_outs warning_msgirs` @rrrHsW( ==D33&&wAH  ( (!! E OO$$J9&'')O*.../NZ=  F? ..22I ..33I 0 7 7 D!8! >hhtN11345G!|' ""55eTBe$((  ]] It$ 53u**+ +D   .  L  %+Q7 ""4, K #33Wn4MNMAz"%"5"56HAuAvQ:e,,%U+Q7(( D$G ""4 jI a< NN>DfUm  OO,.9: (L  sLA M('M(c ^^ [Rm [R"U5nSnU(d[R "TR 5n[R"U5upxn Xvl URURUR5Ul URUR5nUR5nUR!USS9nU bUR#U 5nTR$n [R&"X5n [R("U SS9n U R+5 U U4Sjn U R-5nU R/X4S9n[R0"X5nUS:a [3US 9nTR4(a[R6"TT 5 [R8"T5 [:R<"UTS SUUS T S 9nUR?T 5 [ATRB5n[EU5Vs/sHn/PM nnUbUnO [GS 5eSnUU:aUSS.nURIT SUU5 [JRL"U5n[NRP"UU/5unn[EU5H:nUU R\-nUU R\-nUUUnUUR_U5 M< URaTUUT 5nURIT SUU5 US:XaWRcUS-5 US- nUU:aMUS:aWRcU5 UReT 5 U RgSSS5 [AU5S:Xa[hRj"USSS9nO%UVs/sHn[hRj"USS9PM nnU(aURmU5nU$s snf![JRRRTa. SRWU5n[XRZ"SU-5 Mf=fs snf)a'Predict loop for predicting with TPU tf.distribute.Strategy. Args: model: TF-Keras Model instance. dataset: Dataset for input data. verbose: Integer, Verbosity mode 0 or 1. steps: Total number of steps (batches of samples) before declaring `_predict_loop` finished. Ignored with the default value of `None`. callbacks: List of callbacks to be called during training Returns: Array of predictions (if the model has a single output) or list of arrays of predictions (if the model has multiple outputs). NT)drop_remainderrrYcn>[RR5R[TTU4S9 [ [ R"TT5T5upp1[R"U/5 UVs/sHn[R"U5PM snsSSS5 $s snf!,(df  g=f)z3A fn that returns output of single prediction step.r&N) r:r;rrr"rrr5rr)rrrr*rrrs r_predict_step_fn7experimental_tpu_predict_loop.._predict_step_fns MM - - / : :E4#8 ;  $C  , ,UD 94$  W $ $gY /078BKK$80 /80 /s0B&5 B!B&!B&& B4r&rXrFr]r^rrrgrrrhr)7r PREDICTris_dataset_shape_fully_defined padding_utilPartialBatchPaddingHandler_feed_output_shapesr _get_dataset_attributespadded_batch_sizer padding_mask update_maskmap pad_batchunbatchrdprefetchrkrnrorprrflatten_per_replica_valuesr rr{rr|r}rr0 output_namesrrrr:rrrrrrrrnum_replicas_in_syncextendrrrrnp concatenate apply_mask)!rrrbr]rdataset_fully_shapedpadding_handler batch_sizerprefetch_bufferrrrrpredict_input_datarrrnum_model_outputsunconcatenated_outsrrr predict_opsrrroutput_start_indexoutput_end_indexsingle_model_outputprediction_resultrrs!` @rexperimental_tpu_predict_looprs(&   D%DDWMO 'AA  % % *3)J)J * & -7)'.~~  ( (/*E*E( $++o778//#-- 4-@  &&&7G33&&wAH  ( (!! E OO9&"**,*..4/ ::N!|' ""55eTBe$((   It$ E../',->'?@'?!2'?@   .  L  %+Q7 ""4, K ((>2K#33n-MAz$()A!"%5%J%J!J "%5%J%JJ #-"#3#   " ) )*= >*(( JM ""4 jI a< NNq+AJ4G 3FCBNN3Q '3F  +667HI GA&yy(( >>DfUm  OO,.9:   R s N6 1N;;P ;APPcj\rSrSrSrSSjrS SjrS SjrSrg) $DistributionSingleWorkerTrainingLoopiz;Training loop for distribution strategy with single worker.Nc N[R"XqRS9 [R"X#5 [R"UR UUU[ RUS9upNURXNU5nURUUU U UUU US9n[R"U5(d;UR R5 URUU U UUU S9un nSSS5 SnU (a}[R"U 5unnn[R"UU5 [R"UR UUU[ R5unnURUUUSUUU SS9nOU(a [!S5e["R$"UR 5(aR[R&"UUUUS S 9nUc [!S 5e[(R*"5(d[-UUUUUUU UUUS 9 $[.R0"UUUUUUUU U UUUS S 9 $!,(df  GN>=f)z%Fit loop for Distribution Strategies.)input_callbacks optimizer)validation_split) sample_weight class_weightrrshuffler`)rrrrrNT)rrrrrallow_partial_batchzHvalidation_split argument is not supported with distribution strategies.ra steps_namez^Number of steps could not be inferred from the data, please pass the steps_per_epoch argument.)r`rbrrrrarr) rr`rbr val_inputsrrrarrr)rvalidate_callbacksrvalidate_inputsprocess_batch_and_step_sizerkr rj_validate_or_infer_batch_size#_distribution_standardize_user_datais_distributing_by_cloningr_standardize_user_datar unpack_validation_datarrris_tpu_strategyinfer_steps_for_datasetr:executing_eagerlyrrfit_loop)selfrxyrr`rbrrvalidation_datarrrrrarrkwargsrrrval_xval_yval_sample_weightss rfit(DistributionSingleWorkerTrainingLoop.fitsx* %%%  ""1(&0&L&L  ( (   NN- ' # 88  ;; '%!-<  44U;;--335"'">">"/!-)%5# #?#!Q6  "88I  "  & &ue 4","H"H,,  # A CC0!%!1$(D K+   " "5#?#? @ @/GG, O& @ ''))0!#' +"/$3%5$3  "**  !"'+-+(  O65s H H$c [R"X#5 [R"URX$U[R 5upGUR XGU5nURUUUUSS9n [R"UR5(aK[R"XUSS9nUc [S5e[R"5(d [UU UUUS9$[ R""UU UUUUS9$)z*Evaluate loop for Distribution Strategies.T)rrrr]rrrbr]r)rrrbr]r)rr r rkr rr rrrr rrr:rrr test_loop) rrrrrrbrr]rrrs revaluate-DistributionSingleWorkerTrainingLoop.evaluates  ""1(&BB  ( (!   88AN ;; '! $ <   " "5#?#? @ @%=='E} 6 ''))1#' "++ !   rc [R"USS9 [R"URX#U[R 5up5UR X5U5nURX#SS9n[R"UR5(aK[R"XUSS9nUc [S5e[R"5(d [UUUUUS9$[ R""UUUUUUS 9$) z)Predict loop for Distribution Strategies.N)rrT)rrr]rrr!)rrbr]r)rr r rkr rr rrrr rrr:rrr predict_loop) rrrrrbr]rrrs rpredict,DistributionSingleWorkerTrainingLoop.predictHs ""Q$/&BB  ( (!@P@P  88AN ;; $<   " "5#?#? @ @%=='E} 6''))4#' "..  !   rrQ)NNNrXrXNrNTNNrNNrX)NNNrXNNN)NrNN) __name__ __module__ __qualname____firstlineno____doc__rr#r'__static_attributes__rQrrrrskE  #E T  5 v+ rrc^U4SjnU$)zCDecorator handles multi worker training with distribution strategy.cZ>^^UUU4Sjn[R"UTR5$)Ncr>TRSS5n[R"UT5nUTS'T"T40TD6$)Nr)poprfilter_distributed_callbacks)rrfiltered_callbacksrmethodrs r _worker_fn=_train_with_multi_worker..wrapper.._worker_fnzsE ;5I!+!H!H5" #5F; %*6* *r)dcrun_distribute_coordinatorrk)rrr6r5s`` rwrapper)_train_with_multi_worker..wrapperys( +,, 44  rrQ)r5r:s` r_train_with_multi_workerr<vs   Nrc0\rSrSrSrSrSrSrSrSr g) #DistributionMultiWorkerTrainingLoopiz=Training loop for distribution strategy with multiple worker.cXlgr_single_worker_loop)rsingle_worker_loops r__init__,DistributionMultiWorkerTrainingLoop.__init__s#5 rcL[URR5"U0UD6$r)r<rArrr'rs rr'DistributionMultiWorkerTrainingLoop.fits+'(@(@(D(DE    rcL[URR5"U0UD6$r)r<rAr#rFs rr#,DistributionMultiWorkerTrainingLoop.evaluates+'(@(@(I(IJ    rc:URR"U0UD6$r)rAr'rFs rr'+DistributionMultiWorkerTrainingLoop.predicts''//@@@rr@N) r)r*r+r,r-rCrr#r'r.rQrrr>r>sG6  Arr>r)drXNrNNNrX)rNN)(r-numpyrtensorflow.compat.v2compatv2r: tf_keras.srcrrr|tf_keras.src.distributerr8rrtf_keras.src.enginerrrr tf_keras.src.utils.generic_utilsr tf_keras.src.utils.mode_keysr tensorflow.python.distributer tensorflow.python.platformr rrr"rRrrr TrainingLooprr<r>rQrrrYsL!! *FOM21413< Ld  ~D6:L`6:m`l +<+I+Il ^&A*;*H*HAr