6bi*HSrSSKrSSKrSSKJr SSKJr SSK J r SSK J r SSK Jr SrS rS rS r\"S /S 9"SS55r\"S/S 9\R("S5"SS\555r\"S5"SS\ 55rg)z"Python module for evaluation loop.N) tf_logging) deprecation)ModelCheckpoint) optimizer) keras_exportgN@c[RRU5nUR5nUR 5Vs1sHo3R S5SiM sn$s snf)aLists all the attributes in a checkpoint. Checkpoint keys are paths in a checkpoint graph, and attribute is the first element in the path. e.g. with a checkpoint key "optimizer/iter/.ATTRIBUTES/VARIABLE_VALUE", optimizer is the attribute. The attribute is also used to save/restore a variable in a checkpoint, e.g. tf.train.Checkpoint(optimizer=optimizer, model=model). Args: ckpt_dir_or_file: Directory with checkpoints file or path to checkpoint. Returns: Set of attributes in a checkpoint. /r)tftrainload_checkpointget_variable_to_shape_mapkeyssplit)ckpt_dir_or_filereader variable_mapnames ^/home/james-whalen/.local/lib/python3.13/site-packages/tf_keras/src/utils/sidecar_evaluator.pylist_checkpoint_attributesr!sUXX % %&6 7F335L+7+<+<+> ?+>4JJsOA +> ?? ?sA"zkeras.utils.SidecarEvaluator)v1c4\rSrSrSrSSjrSrSrSrg) SidecarEvaluator5a A class designed for a dedicated evaluator task. `SidecarEvaluator` is expected to be run in a process on a separate machine from the training cluster. It is meant for the purpose of a dedicated evaluator, evaluating the metric results of a training cluster which has one or more workers performing the training, and saving checkpoints. The `SidecarEvaluator` API is compatible with both Custom Training Loop (CTL), and TF-Keras `Model.fit` to be used in the training cluster. Using the model (with compiled metrics) provided at `__init__`, `SidecarEvaluator` repeatedly performs evaluation "epochs" when it finds a checkpoint that has not yet been used. Depending on the `steps` argument, an eval epoch is evaluation over all eval data, or up to certain number of steps (batches). See examples below for how the training program should save the checkpoints in order to be recognized by `SidecarEvaluator`. Since under the hood, `SidecarEvaluator` uses `model.evaluate` for evaluation, it also supports arbitrary TF-Keras callbacks. That is, if one or more callbacks are provided, their `on_test_batch_begin` and `on_test_batch_end` methods are called at the start and end of a batch, and their `on_test_begin` and `on_test_end` are called at the start and end of an evaluation epoch. Note that `SidecarEvaluator` may skip some checkpoints because it always picks up the latest checkpoint available, and during an evaluation epoch, multiple checkpoints can be produced from the training side. Example: ```python model = tf.keras.models.Sequential(...) model.compile(metrics=tf.keras.metrics.SparseCategoricalAccuracy( name="eval_metrics")) data = tf.data.Dataset.from_tensor_slices(...) tf.keras.SidecarEvaluator( model=model, data=data, # dir for training-saved checkpoint checkpoint_dir='/tmp/checkpoint_dir', steps=None, # Eval until dataset is exhausted max_evaluations=None, # The evaluation needs to be stopped manually callbacks=[tf.keras.callbacks.TensorBoard(log_dir='/tmp/log_dir')] ).start() ``` `SidecarEvaluator.start` writes a series of summary files which can be visualized by tensorboard (which provides a webpage link): ```bash $ tensorboard --logdir=/tmp/log_dir/validation ... TensorBoard 2.4.0a0 at http://host:port (Press CTRL+C to quit) ``` If the training cluster uses a CTL, the `checkpoint_dir` should contain checkpoints that track both `model` and `optimizer`, to fulfill `SidecarEvaluator`'s expectation. This can be done by a `tf.train.Checkpoint` and a `tf.train.CheckpointManager`: ```python # Same `checkpoint_dir` supplied to `SidecarEvaluator`. checkpoint_dir = ... checkpoint = tf.train.Checkpoint(model=model, optimizer=optimizer) checkpoint_manager = tf.train.CheckpointManager( checkpoint, checkpoint_dir=..., max_to_keep=...) checkpoint_manager.save() ``` If the training cluster uses TF-Keras `Model.fit` API, a `tf.keras.callbacks.ModelCheckpoint` should be used, with `save_weights_only=True`, and the `filepath` should have 'ckpt-{epoch}' appended: ```python # Same `checkpoint_dir` supplied to `SidecarEvaluator`. checkpoint_dir = ... model_checkpoint = tf.keras.callbacks.ModelCheckpoint( filepath=os.path.join(checkpoint_dir, 'ckpt-{epoch}'), save_weights_only=True) model.fit(dataset, epochs, callbacks=[model_checkpoint]) ``` NcXlX lX0l[R"S[ [R S9UlXPlX@l U=(d /Ul g)aInitializes an `SidecarEvaluator` object. Args: model: Model to use for evaluation. The model object used here should be a `tf.keras.Model`, and should be the same as the one that is used in training, where `tf.keras.Model`s are checkpointed. The model should have one or more metrics compiled before using `SidecarEvaluator`. data: The input data for evaluation. `SidecarEvaluator` supports all data types that TF-Keras `model.evaluate` supports as the input data `x`, such as a `tf.data.Dataset`. checkpoint_dir: Directory where checkpoint files are saved. steps: Number of steps to perform evaluation for, when evaluating a single checkpoint file. If `None`, evaluation continues until the dataset is exhausted. For repeated evaluation dataset, user must specify `steps` to avoid infinite evaluation loop. max_evaluations: Maximum number of the checkpoint file to be evaluated, for `SidecarEvaluator` to know when to stop. The evaluator will stop after it evaluates a checkpoint filepath ending with '-'. If using `tf.train.CheckpointManager.save` for saving checkpoints, the kth saved checkpoint has the filepath suffix '-' (k=1 for the first saved), and if checkpoints are saved every epoch after training, the filepath saved at the kth epoch would end with '-. Thus, if training runs for n epochs, and the evaluator should end after the training finishes, use n for this parameter. Note that this is not necessarily equal to the number of total evaluations, since some checkpoints may be skipped if evaluation is slower than checkpoint creation. If `None`, `SidecarEvaluator` will evaluate indefinitely, and the user must terminate evaluator program themselves. callbacks: List of `keras.callbacks.Callback` instances to apply during evaluation. See [callbacks](/api_docs/python/tf/tf_keras/callbacks). iterations)r initial_valuedtypeN) modeldatacheckpoint_dirr Variable_ITERATIONS_UNINITIALIZEDint64 _iterationsmax_evaluationssteps callbacks)selfr r!r"r(r'r)s r__init__SidecarEvaluator.__init__sKX  ,;;3((  / "bc@[R"S[S35 g)Nz(No checkpoints appear to be found after a seconds. Please check if you are properly using a `tf.train.Checkpoint/CheckpointManager` or `tf.keras.callbacks.ModelCheckpoint(save_weights_only=True)` to save checkpoints by the training. See `tf.keras.SidecarEvaluator` doc for recommended flows of saving checkpoints.F)logginginfo_CHECKPOINT_TIMEOUT_SEC)r*s r _timeout_fnSidecarEvaluator._timeout_fns& 6&'(% % r-cURR(ap[URR[R5(a=[R R URURRS9nOO[R R URS9n[R R URUS9n[R RUR[URS9GHnURU5R5 [U5nSU;aURRU5 URR(al[URR[R5(d9URRR R#UR5 URR35[4:XaL[URR[R5(d[7S URS 35e[*R,"S U35 URR9UR:UR<UR>S S 9 0nURR@HInURC5n[U[D5(aURGU5 M;XURH'MK [*R,"SSRKURM5V V s/sHupU SU R353PM sn n 55 URN(dGMmURN[QURSS5S5::dGM[*R,"S5 g g![R$R&4ajn[U[R$R(5(aUe[*R,"SUSUR.R0SU35 SnAGM:SnAff=fs sn n f)zStarts the evaluation loop.)r r)iter)timeout timeout_fnr zESidecarEvaluator encountered an error when loading the checkpoint at z. Retrying. Error: z: NzCVariable `iterations` cannot be loaded from the checkpoint file at zQ. Please ensure `iterations` is included in the checkpoint saved during training.zDEvaluation starts: Model weights loaded from latest checkpoint file )r(r)verbosezEnd of evaluation. Metrics: %s =-rz2Last checkpoint evaluated. SidecarEvaluator stops.)*r r isinstance Optimizerr r Checkpointr&checkpoints_iteratorr"r1r2restoreexpect_partialr load_weightsrassignerrorsOpErrorUnavailableErrorr/r0 __class____name__numpyr$ RuntimeErrorevaluater!r(r)metricsresultdictupdaterjoinitemsr'intr) r* checkpointoptimizer_checkpointlatest_checkpointcheckpoint_attributesereturn_metricsmetricrNrvalues rstartSidecarEvaluator.starts ::  J JJ )"5"5% % ,,jjDJJ,@,@-J$&88#6#6D!>   +''"?"  . ""#45DDF(B%)%"77JJ++,=>::'' JJ((''11 JJ((33::4;K;KL.  &&(,EE"JJ(('' #**.*=*=)>?HH LL##4"57  JJ   t~~q   N**,,fd++"))&1286;;/ - LL0,:+?+?+A+AKD &%++-1+A ###$$,=,C,CC,H,L(MM HI" >II%%' a!;!;<<G %%6$78kk2232aS: ) ls CO>!QQ'AQ  Q)r&r)r"r!r'r r()NNN) rI __module__ __qualname____firstlineno____doc__r+r2r\__static_attributes__r-rrr5s%Pn6)p qr-rz#keras.experimental.SidecarEvaluatorc,^\rSrSrSrU4SjrSrU=r$)SidecarEvaluatorExperimentaliBzDeprecated. Please use `tf.keras.utils.SidecarEvaluator` instead. Caution: `tf.keras.experimental.SidecarEvaluator` endpoint is deprecated and will be removed in a future release. Please use `tf.keras.utils.SidecarEvaluator`. cR>[R"S5 [TU] "U0UD6 g)Nz`tf.keras.experimental.SidecarEvaluator` endpoint is deprecated and will be removed in a future release. Please use `tf.keras.utils.SidecarEvaluator`.)r/warningsuperr+)r*argskwargsrHs rr+%SidecarEvaluatorExperimental.__init__Ls( 1 $)&)r-rc)rIr^r_r`rar+rb __classcell__rHs@rrereBs**r-rez+keras.callbacks.SidecarEvaluatorModelExportc<^\rSrSrSrU4SjrSSjrSrSrU=r $)SidecarEvaluatorModelExportiUaCallback to save the best TF-Keras model. It expands the functionality of the existing ModelCheckpoint callback to enable exporting the best models after evaluation with validation dataset. When using the `SidecarEvaluatorModelExport` callback in conjunction with `keras.utils.SidecarEvaluator`, users should provide the `filepath`, which is the path for this callback to export model or save weights to, and `ckpt_filepath`, which is where the checkpoint is available to extract the epoch number from. The callback will then export the model that the evaluator deems as the best (among the checkpoints saved by the training counterpart) to the specified `filepath`. This callback is intended to be used by SidecarEvaluator only. Example: ```python model.compile(loss=..., optimizer=..., metrics=['accuracy']) sidecar_evaluator = keras.utils.SidecarEvaluator( model=model, data=dataset, checkpoint_dir=checkpoint_dir, max_evaluations=1, callbacks=[ SidecarEvaluatorModelExport( export_filepath=os.path.join(checkpoint_dir, 'best_model_eval', 'best-model-{epoch:04d}'), checkpoint_filepath=os.path.join(checkpoint_dir, 'ckpt-{epoch:04d}'), save_freq="eval", save_weights_only=True, monitor="loss", mode="min", verbose=1, ), ], ) sidecar_evaluator.start() # Model weights are saved if evaluator deems it's the best seen so far. Args: export_filepath: Path where best models should be saved by this `SidecarEvaluatorModelExport` callback. Epoch formatting options, such as `os.path.join(best_model_dir, 'best-model-{epoch:04d}')`, can be used to allow saved model to preserve epoch information in the file name. SidecarEvaluatorModelExport will use the "training epoch" at which the checkpoint was saved by training to fill the epoch placeholder in the path. checkpoint_filepath: Path where checkpoints were saved by training. This should be the same as what is provided to `filepath` argument of `ModelCheckpoint` on the training side, such as `os.path.join(checkpoint_dir, 'ckpt-{epoch:04d}')`. c 8>[TU]"SUSS.UD6 X lg)NT)filepathsave_best_onlyrc)rhr+_checkpoint_filepath)r*export_filepathcheckpoint_filepathrjrHs rr+$SidecarEvaluatorModelExport.__init__s-  $  %8!r-cURUR5nUb,[[R"SU5S5S- UlgSUlg)z.Updates export_index to the latest checkpoint.Nz.*ckpt-(?P\d+)ckptr)1_get_most_recently_modified_file_matching_patternrsrSrematch export_index)r*logsmost_recent_filepaths r on_test_begin)SidecarEvaluatorModelExport.on_test_beginsd  B B))   +HH46JK    !"D r-cdU=RS- slURURSUS9 g)z3Saves best model at the end of an evaluation epoch.ryN)epochbatchr~)epochs_since_last_save _save_modelr})r*r~s r on_test_end'SidecarEvaluatorModelExport.on_test_ends0 ##q(# t004Hr-)rsr})N) rIr^r_r`rar+rrrbrlrms@rroroUs6p8"(IIr-ro)rar{ tensorflowr tensorflow.python.platformrr/tensorflow.python.utilrtf_keras.src.callbacksrtf_keras.src.optimizersr tensorflow.python.util.tf_exportr_PRINT_EVAL_STEP_EVERY_SECr$r1rrdeprecated_endpointsrerorcr-rrs) =.2-9!@(,4II5IX3; !!"GH*#3*I<*";<ZI/ZI=ZIr-