6bi=^SrSSKJs Jr SSKJr SSKJr \"S5"SS\55r g)zAdditive attention layer that can be used in sequence DNN/CNN models. This file follows the terminology of https://arxiv.org/abs/1706.03762 Figure 2. Attention is formed by three tensors: Query, Key and Value. N)BaseDenseAttention) keras_exportzkeras.layers.AdditiveAttentioncN^\rSrSrSrSU4SjjrU4SjrSrU4SjrSr U=r $) AdditiveAttentionaAdditive attention layer, a.k.a. Bahdanau-style attention. Inputs are `query` tensor of shape `[batch_size, Tq, dim]`, `value` tensor of shape `[batch_size, Tv, dim]` and `key` tensor of shape `[batch_size, Tv, dim]`. The calculation follows the steps: 1. Reshape `query` and `key` into shapes `[batch_size, Tq, 1, dim]` and `[batch_size, 1, Tv, dim]` respectively. 2. Calculate scores with shape `[batch_size, Tq, Tv]` as a non-linear sum: `scores = tf.reduce_sum(tf.tanh(query + key), axis=-1)` 3. Use scores to calculate a distribution with shape `[batch_size, Tq, Tv]`: `distribution = tf.nn.softmax(scores)`. 4. Use `distribution` to create a linear combination of `value` with shape `[batch_size, Tq, dim]`: `return tf.matmul(distribution, value)`. Args: use_scale: If `True`, will create a variable to scale the attention scores. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Defaults to `0.0`. Call arguments: inputs: List of the following tensors: * query: Query `Tensor` of shape `[batch_size, Tq, dim]`. * value: Value `Tensor` of shape `[batch_size, Tv, dim]`. * key: Optional key `Tensor` of shape `[batch_size, Tv, dim]`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: * query_mask: A boolean mask `Tensor` of shape `[batch_size, Tq]`. If given, the output will be zero at the positions where `mask==False`. * value_mask: A boolean mask `Tensor` of shape `[batch_size, Tv]`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. use_causal_mask: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. Defaults to `False`. Output: Attention outputs of shape `[batch_size, Tq, dim]`. [Optional] Attention scores after masking and softmax with shape `[batch_size, Tq, Tv]`. The meaning of `query`, `value` and `key` depend on the application. In the case of text similarity, for example, `query` is the sequence embeddings of the first piece of text and `value` is the sequence embeddings of the second piece of text. `key` is usually the same tensor as `value`. Here is a code example for using `AdditiveAttention` in a CNN+Attention network: ```python # Variable-length int sequences. query_input = tf.keras.Input(shape=(None,), dtype='int32') value_input = tf.keras.Input(shape=(None,), dtype='int32') # Embedding lookup. token_embedding = tf.keras.layers.Embedding(max_tokens, dimension) # Query embeddings of shape [batch_size, Tq, dimension]. query_embeddings = token_embedding(query_input) # Value embeddings of shape [batch_size, Tv, dimension]. value_embeddings = token_embedding(value_input) # CNN layer. cnn_layer = tf.keras.layers.Conv1D( filters=100, kernel_size=4, # Use 'same' padding so outputs have the same shape as inputs. padding='same') # Query encoding of shape [batch_size, Tq, filters]. query_seq_encoding = cnn_layer(query_embeddings) # Value encoding of shape [batch_size, Tv, filters]. value_seq_encoding = cnn_layer(value_embeddings) # Query-value attention of shape [batch_size, Tq, filters]. query_value_attention_seq = tf.keras.layers.AdditiveAttention()( [query_seq_encoding, value_seq_encoding]) # Reduce over the sequence axis to produce encodings of shape # [batch_size, filters]. query_encoding = tf.keras.layers.GlobalAveragePooling1D()( query_seq_encoding) query_value_attention = tf.keras.layers.GlobalAveragePooling1D()( query_value_attention_seq) # Concatenate query and document encodings to produce a DNN input layer. input_layer = tf.keras.layers.Concatenate()( [query_encoding, query_value_attention]) # Add DNN layers, and create Model. # ... ``` c 2>[TU]"S0UD6 Xlg)N)super__init__ use_scale)selfr kwargs __class__s j/home/james-whalen/.local/lib/python3.13/site-packages/tf_keras/src/layers/attention/additive_attention.pyr AdditiveAttention.__init__s "6""c>[R"US5nUSn[RRU5nUR(a$UR SU/SUR SS9UlOSUl[TU]%U5 g)Nscaleglorot_uniformT)nameshape initializerdtype trainable) tf TensorShapecompatdimension_valuer add_weightrrr build)r input_shapev_shapedimrs rr"AdditiveAttention.buildsx..Q0bkii'', >>e,jj )DJDJ  k"rc[R"USS9n[R"USS9nUR(a URnOSn[R"U[R "X4-5-SS9$)zCalculates attention scores as a nonlinear sum of query and key. Args: query: Query tensor of shape `[batch_size, Tq, dim]`. key: Key tensor of shape `[batch_size, Tv, dim]`. Returns: Tensor of shape `[batch_size, Tq, Tv]`. )axisg?r)r expand_dimsr r reduce_sumtanh)r querykey q_reshaped k_reshapedrs r_calculate_scores#AdditiveAttention._calculate_scoress[^^E3 ^^Cb1 >>JJEE}}URWWZ-D%EEBOOrc>SUR0n[TU] 5n[[ UR 55[ UR 55-5$)Nr )r r get_configdictlistitems)r config base_configrs rr5AdditiveAttention.get_configsGt~~.g(* D**,-V\\^0DDEEr)rr )T) __name__ __module__ __qualname____firstlineno____doc__r r"r2r5__static_attributes__ __classcell__)rs@rrrs&dL## P(FFrr) r@tensorflow.compat.v2rv2r2tf_keras.src.layers.attention.base_dense_attentionr tensorflow.python.util.tf_exportrrr rrrGsB"!Q:./RF*RF0RFr