6bilSrSSKrSSKrSSKJs Jr SrSrS Sjr S Sjr Sr "SS 5r g) zUtils for running models in a distribution setting. Mostly from https://github.com/tensorflow/models/blob/master/official/utils/misc/distribution_utils.py. Nc\[RRRR[RRRR [RRRR S.nX;a[SRU55eX$)a+Return a CollectiveCommunication based on all_reduce_alg. Args: all_reduce_alg: a string specifying which collective communication to pick, or None. Returns: tf.distribute.experimental.CollectiveCommunication object Raises: ValueError: if `all_reduce_alg` not in [None, "ring", "nccl"] )NringncclzqWhen used with `multi_worker_mirrored`, valid values for all_reduce_alg are [`ring`, `nccl`]. Supplied value: {}) tf distribute experimentalCollectiveCommunicationAUTORINGNCCL ValueErrorformat)all_reduce_alg collective_communication_optionss c/home/james-whalen/.local/lib/python3.13/site-packages/tf_keras/src/benchmarks/distribution_util.py_collective_communicationrsmm((@@EE **BBGG **BBGG($ = GGMvH  , ;;cUcg[RR[RRS.nX;a[ SR U55eX nU"US9$)axReturn a CrossDeviceOps based on all_reduce_alg and num_packs. Args: all_reduce_alg: a string specifying which cross device op to pick, or None. num_packs: an integer specifying number of packs for the cross device op. Returns: tf.distribute.CrossDeviceOps object or None. Raises: ValueError: if `all_reduce_alg` not in [None, "nccl", "hierarchical_copy"]. N)rhierarchical_copyzqWhen used with `mirrored`, valid values for all_reduce_alg are [`nccl`, `hierarchical_copy`]. Supplied value: {}) num_packs)rr NcclAllReduceHierarchicalCopyAllReducer r)rrmirrored_all_reduce_optionscross_device_ops_classs r_mirrored_cross_device_opsr7sn ++]]DD#8 AAGB  9H !I 66rcUS:a [S5eUR5nUS:Xa!US:a[SRU55egUS:Xa0[RR R [U5S9$US :XaUUS:Xa[RRS 5$US:a [S 5e[RRS 5$US :XaOUS:XaS /nO[U5Vs/sHnSU-PM nn[RRU[X#5S9$[SU35es snf)aReturn a DistributionStrategy for running the model. Args: distribution_strategy: a string specifying which distribution strategy to use. Accepted values are "off", "one_device", "mirrored", and "multi_worker_mirrored" -- case insensitive. "off" means not to use Distribution Strategy. num_gpus: Number of GPUs to run this model. Returns: tf.distribute.DistibutionStrategy object. Raises: ValueError: if `distribution_strategy` is "off" or "one_device" and `num_gpus` is larger than 1; or `num_gpus` is negative. rz`num_gpus` can not be negative.offzNWhen {} GPUs are specified, distribution_strategy flag cannot be set to `off`.Nmulti_worker_mirrored) communication one_devicez device:CPU:0z=`OneDeviceStrategy` can not be used for more than one device.z device:GPU:0mirroredz device:GPU:%d)devicescross_device_opsz$Unrecognized Distribution Strategy: ) r lowerrrrrMultiWorkerMirroredStrategyrOneDeviceStrategyrangeMirroredStrategyr)distribution_strategynum_gpusrrr#is rget_distribution_strategyr-Ws]*!|:;;1779% a<//5vh/?  77}}))EE3NCF   , q===22>B B a<O }}..~>> * q=%&G49(ODOq*OGD}}--7.    ./D.EF Es7D=c[R"[RR SS55nU(a?[ USR S/55[ USR S/55-nU$U(anUR S5n[ U5nUS:aUS:a [S 5eUS:XaSOUn[R"SU0SUS .S .5[RS'U$SnU$) zSet multi-worker cluster spec in TF_CONFIG environment variable. Args: worker_hosts: comma-separated list of worker ip:port pairs. Returns: Number of workers in the cluster. TF_CONFIGz{}clusterchiefworker,rrz2Must specify task_index when number of workers > 1)typeindex)r0task) jsonloadsosenvirongetlensplitr dumps) worker_hosts task_index tf_config num_workersworkerss rconfigure_clusterrDs 2::>>+t<=I)I.227B?@3 i $ $Xr 2D  & ! $$S)'l ?zA~D &*Q "&**$g.!)J? #  ;  rcLU(aUR5nU$[5nU$N)scopeDummyContextManager)strategystrategy_scopes rget_strategy_scoperKs)!) -. rc \rSrSrSrSrSrg)rHcgrF)selfs r __enter__DummyContextManager.__enter__ rcgrFrO)rPargss r__exit__DummyContextManager.__exit__rSrrON)__name__ __module__ __qualname____firstlineno__rQrV__static_attributes__rOrrrHrHs   rrH)r"rNr)N) __doc__r7r9tensorflow.compat.v2compatv2rrrr-rDrKrHrOrrrbsL  !!<87B%  >BB  r