6bicXSrSSKrSSKrSSKrSSKrSSKJs Jr SSK J r SSK J r SSK J r SSKJr SSKJr /SQr\R&"5qS r\"S /S 9"S S \R,R.55r\R2R\R4l\R6S5rSrSrSrSr Sr!Sr"Sr#Sr$g)z5Library for map layout and corresponding tf.Variable.N) dtensor_api) lazy_variable)utils) base_layer) keras_export)_self_tracked_trackables_trainable_weights_non_trainable_weights_captured_weight_regularizerc$[[SS5$)N layout_map)getattr _LAYOUT_MAPY/home/james-whalen/.local/lib/python3.13/site-packages/tf_keras/src/dtensor/layout_map.pyget_current_layout_mapr0s ; d 33rz$keras.dtensor.experimental.LayoutMap)v1cL\rSrSrSrS SjrSrSrSrSr S r S r S r S r g) LayoutMap4aA dict-like object that maps string to `Layout` instances. `LayoutMap` uses a string as key and a `Layout` as value. There is a behavior difference between a normal Python dict and this class. The string key will be treated as a regex when retrieving the value. See the docstring of `get` for more details. See below for a usage example. You can define the naming schema of the `Layout`, and then retrieve the corresponding `Layout` instance. To use the `LayoutMap` with a `Model`, please see the docstring of `tf.keras.dtensor.experimental.layout_map_scope`. ```python map = LayoutMap(mesh=None) map['.*dense.*kernel'] = layout_2d map['.*dense.*bias'] = layout_1d map['.*conv2d.*kernel'] = layout_4d map['.*conv2d.*bias'] = layout_1d layout_1 = map['dense_1.kernel'] # layout_1 == layout_2d layout_2 = map['dense_1.bias'] # layout_2 == layout_1d layout_3 = map['dense_2.kernel'] # layout_3 == layout_2d layout_4 = map['dense_2.bias'] # layout_4 == layout_1d layout_5 = map['my_model/conv2d_123/kernel'] # layout_5 == layout_4d layout_6 = map['my_model/conv2d_123/bias'] # layout_6 == layout_1d ``` Args: mesh: An optional `Mesh` that can be used to create all replicated layout as default when there isn't a layout found based on the input string query. NcD[R"5UlXlgN) collections OrderedDict _layout_map _default_mesh)selfmeshs r__init__LayoutMap.__init__Xs&224!rcXR;aURU$URH/n[R"X!5(dM URUs $ g)aRetrieve the corresponding layout by the string key. When there isn't an exact match, all the existing keys in the layout map will be treated as a regex and map against the input key again. The first match will be returned, based on the key insertion order. Return None if there isn't any match found. Args: key: the string key as the query for the layout. Returns: Corresponding layout based on the query. N)rrematch)rkeyks r __getitem__LayoutMap.__getitem__\sV "" "##C( (!!Axx''**"rcXR;a[USURUS35e[U[R5(d[US[ U535eX RU'g)Nz+ already exist in the LayoutMap with value z.. Please make sure to not use duplicated keys.z& should be a dtensor.Layout type, got )r ValueError isinstancedtensorLayouttype)rr%layouts r __setitem__LayoutMap.__setitem__rs "" "%))#./0++  &'..11(@fO !'rc8URRU5$r)rpop)rr%s r __delitem__LayoutMap.__delitem__s##C((rc,[UR5$r)lenrrs r__len__LayoutMap.__len__s4##$$rc,[UR5$r)iterrr8s r__iter__LayoutMap.__iter__sD$$%%rcUR$)zReturn the default `Mesh` set at instance creation. The `Mesh` can be used to create default replicated `Layout` when there isn't a match of the input string query. )rr8s rget_default_meshLayoutMap.get_default_meshs !!!rc[U5$)a Apply layout to all `tf.Variable` instances created under the scope. All `tf.Variable` instances created under this scope will be lazily initialized first. Once they are attached as the model or layer attributes, and there is a stable layout mapping for it, the variables will be reinitialized into a `tf.experimental.dtensor.DVariable` with corresponding layout. Note that the layout mapping will use object/attribute names as the keys to map the variable to the layout. For subclassed models, the full object/attribute name is used as the key. For Functional/Sequential models, we use `layer.name` as the key for the layer, followed by the attribute name. TF-Keras ensures name uniqueness among the layers within a Functional/Sequential model. See the following examples that show variable object names for different TF-Keras model types: ```python layout_map = layout_map_lib.LayoutMap(mesh=self.mesh) layout_map['d1.kernel'] = layout_1 layout_map['d1.bias'] = layout_2 layout_map['d2.kernel'] = layout_3 layout_map['d2.bias'] = layout_4 ## Subclassed model class SubclassModel(tf.keras.Model): def __init__(self, name=None): super().__init__(name=name) self.d1 = tf.keras.layers.Dense(1000) self.d2 = tf.keras.layers.Dense(1000) def call(self, inputs): x = self.d1(inputs) return self.d2(x) with layout_map.scope(): model = SubclassModel() inputs = tf.zeros((10, 10)) results = model(inputs) model.d1.kernel.layout == layout_1 model.d1.bias.layout == layout_2 model.d2.kernel.layout == layout_3 model.d2.bias.layout == layout_4 ## Functional model with layout_map.scope(): inputs = tf.keras.Input((10,), batch_size=10) x = tf.keras.layers.Dense(20, name='d1')(inputs) output = tf.keras.layers.Dense(30, name='d2')(x) model = tf.keras.Model(inputs, output) d1 = model.layers[1] d2 = model.layers[2] d1.kernel.layout == layout_1 d1.bias.layout == layout_2 d1.kernel.layout == layout_3 d1.bias.layout == layout_4 ## Sequential model with layout_map.scope(): model = tf.keras.Sequential([ tf.keras.layers.Dense(20, name='d1', input_shape=(10,)), tf.keras.layers.Dense(30, name='d2') ]) d1 = model.layers[0] d2 = model.layers[1] d1.kernel.layout == layout_1 d1.bias.layout == layout_2 d1.kernel.layout == layout_3 d1.bias.layout == layout_4 ``` Returns: A context that will lazily initialize all `tf.Variable` objects within the model, with their attributed layouts. )layout_map_scoper8s rscopeLayoutMap.scopesj %%r)rrr)__name__ __module__ __qualname____firstlineno____doc__r r'r0r4r9r=r@rD__static_attributes__rrrrr4s1 D", ')%&"U&rrc## [5nU[l[R"5 Sv U[lSSS5 g!U[lf=f!,(df  g=f7f)a Apply the layout to all the tf.Variables created under the scope. Create a scope that all the tf.Variable created under this scope will be lazily inited, and initialized later on with proper layout when the object path in the model is stable/finalized. Note that the layout mapping will use the object/attribute names as the key to map the variable against the layout. For subclassed models, the full object/attribute name is used as the key. For Functional/Sequential models, since the layers within the model do not get assigned to a meaningful attribute, we use `layer.name` as the key for the layer, followed by the attribute name. TF-Keras ensures name uniqueness among the layers in all Functional/Sequential models. See the following examples that show the variable object names for different TF-Keras model types: ```python layout_map = layout_map_lib.LayoutMap(mesh=self.mesh) layout_map['d1.kernel'] = layout_1 layout_map['d1.bias'] = layout_2 layout_map['d2.kernel'] = layout_3 layout_map['d2.bias'] = layout_4 ## Subclassed model class SubclassModel(tf.keras.Model): def __init__(self, name=None): super().__init__(name=name) self.d1 = tf.keras.layers.Dense(1000) self.d2 = tf.keras.layers.Dense(1000) def call(self, inputs): x = self.d1(inputs) return self.d2(x) with layout_map_scope(layout_map): model = SubclassModel() # Triggering the creation of weights within or outside of the scope works inputs = tf.zeros((10, 10)) results = model(inputs) model.d1.kernel.layout == layout_1 model.d1.bias.layout == layout_2 model.d2.kernel.layout == layout_3 model.d2.bias.layout == layout_4 ## Functional model with layout_map_scope(layout_map): inputs = tf.keras.Input((10,), batch_size=10) x = tf.keras.layers.Dense(20, name='d1')(inputs) output = tf.keras.layers.Dense(30, name='d2')(x) model = tf.keras.Model(inputs, output) d1 = model.layers[1] d2 = model.layers[2] d1.kernel.layout == layout_1 d1.bias.layout == layout_2 d1.kernel.layout == layout_3 d1.bias.layout == layout_4 ## Sequential model with layout_map_scope(layout_map): model = tf.keras.Sequential([ tf.keras.layers.Dense(20, name='d1', input_shape=(10,)), tf.keras.layers.Dense(30, name='d2') ]) d1 = model.layers[0] d2 = model.layers[1] d1.kernel.layout == layout_1 d1.bias.layout == layout_2 d1.kernel.layout == layout_3 d1.bias.layout == layout_4 ``` Args: layout_map: a LayoutMap which contains the variable_object_path (string) -> Layout. When a layout is not found for the variable, a default all replicated layout will be created for the variable. Yields: A context that will lazily initialize all `tf.Variable` objects within the model, with their attributed layouts. N)rrr rlazy_init_scope)r previous_layout_maps rrCrCsQv12'K  & & ( 9 %8K " ) (&9K " ) (s1+A)AA A A) AA A&"A)c  0nUR[SS9Hzup4[Vs/sH oUU;dM UPM sn(aM*SRUVs/sHn[ U5PM sn5n[ XU5n[ XU5 X[U5'M| URSS9Hn [X5 M UR[SS9Hup4U[U5n [ XU 5 M! [X5 [X5 U$s snfs snf)z0Map/Replace LazyInitVariable for subclass model.T predicate with_path.c6[U[R5$r)r+rLayeros r._map_subclass_model_variable..isJq**:*:;rrQ) _flatten_is_lazy_init_variable_KERAS_ATTRIBUTES_TO_SKIPjoinstr_create_dvariable_set_object_by_pathid _config_dvariable_regularization_init_state_variable_for_rng_update_trackable_reference) modelr %lazy_init_variable_to_tf_variable_mappathvariableaitem object_path new_variablelayer tf_variables r_map_subclass_model_variablerpSs,.) ..() 1 >0!IA0 > hhd;ddD d;< ((K E6>Jbl;;  )   ..().sJq**D*DErrZNzKeras is expected to use tf.random.Generator when using DTensor API. Please call `tf.keras.backend.experimental.enable_tf_random_generator` at the beginning of your program. _generator) r[_random_generator_builtrzr*hasattrr\ _state_varr`r, default_meshr@ _maybe_init)rfr lkeras_generators rrdrds^^E--  ! !o&@&@&H1  ?L 1 16L  & & 1 17 7 5FB : : E E5O & & 1%%j&A&A&CD++-ED-,EDs C22 D cURHDup#n[U5(d[SU35eU[U5nUR X%U5 MF /Ulg)a8Update the weights regularizer for newly created `DVariable`. The weight regularization usually happens when `layer.add_weight()` is called, at which point the library will first create a `LazyInitVariable`, and then replace it with a `DVariable`. We will defer the creation of those losses, until the DVariable is created. See `layer._captured_weight_regularizer` for more details. Args: layer: the layer instance for DVariable regularization config. lazy_init_variable_to_tf_variable_map: the dict between LazyInitVariable ID and newly created DVariable. zFExpect the regularization loss are created from LazyInitVariable, got N)r r\r*rb_handle_weight_regularization)rnrgrsri regualarizer d_variables rrcrcsk$).(J(J$ %h//))1 4 ;2h<H  ++DlK)K*,E&rcXnUcBURRn[RR UR 5US9nUR n[U5(a5[R"5 [R"XS5nSSS5 O[R"XS5nURnURS5(aUSSn[R"XRR US9nU$!,(df  NV=f)aCreate a new variable instead of using the LazyInitVariable. We choose to do this since even the LazyInitVariable might behavior like a normal tf.Variable/DVariable, it is not future proof for any new changes to variable class. It will also fail the instance type check in python, which could affect user's code when they do any filtering based on type to find any variables. Args: layout_map: a LayoutMap which contains the variable_object_path (string) -> Layout. object_path: string, the object attribute path for the variable. variable: LazyInitVariable which will be replaced by the newly created tf.Variable. Returns: A new tf.Variable with correct layout information. N)rrankz:0) trainablers)shaperr,r- replicatedr@_initial_valuecallablerdisable_init_variable_creatorrcall_with_layout copy_to_meshrsendswith DVariabler)r rlrir/ variable_rankinit_val variable_namerms rr`r`s( $F ~ ++ **,,.]+ &&H  8 8 :--h?H; : ''9MMMd##%cr* $$..]L ; :s ;C99 Dc[U5Hfup4U[U5S- :Xa)[U[5(aX U'M2[ XU5 M@[U[5(aXnM[[ X5nMh g)aSet the attribute of instance to the object. Args: object_to_set: the instance whose attribute should be set. path: the tuple/list of string and ints, representing the attribute names. Int means that the attribute to set is a item a list. value: the value of the attribute. N) enumerater7r+intsetattrr) object_to_setrhvaluei attr_names rrarasd"$  D A )S)),1i( %8)S)) - 8 ' A (rc8[RRRU5nUR 5up4UHYnUR 5R 5H4upg[U5(dMURU[U5USS9 M6 M[ g)aUpdate the trackable object references for the model. Note that this method is only needed because of a corner case for model checkpoint, where it could accidently catch a LazyInitVariable in checkpoint dependency and not visible to the model attribute graph itself. Args: model: the keras model instance whose checkpoint dependency will be examed. lazy_init_variable_to_tf_variable_map: the dict between LazyInitVariable ID and newly created DVariable. T) overwriteN) tf __internal__trackingObjectGraphViewbreadth_first_traversal_trackable_childrenitemsr\_track_trackablerb)rfrg object_graph trackables_ trackableref_namerefs rrere9s??++;;EBL 88:MJ &::<BBDMH%c****9"S'B"+E rc6[U[R5$r)r+rLazyInitVariable)objs rr\r\Ts c=99 ::r)%rJr contextlibr# threadingtensorflow.compat.v2compatv2rtf_keras.src.dtensorrr,rrtf_keras.src.enginer tensorflow.python.util.tf_exportrr]localrrabcMutableMappingrr'getcontextmanagerrCrprurdrcr`rarer\rrrrs< !!7.&*:oo 44<r& ..r&=r&j"--55  b9b9J&R+\&.R,<*ZB66;r