hF5SSKJr SSKrSSKrSSKJr SSKJr "SS\R5r"SS \R5r g) ) annotationsN)nn) RoPEAttentionc^\rSrSrSrSS U4SjjjrS SjrS S SjjrS SSjjrSr U=r $)MemoryAttentionLayer aX Implements a memory attention layer with self-attention and cross-attention mechanisms for neural networks. This class combines self-attention, cross-attention, and feedforward components to process input tensors and generate memory-based attention outputs. Attributes: d_model (int): Dimensionality of the model. dim_feedforward (int): Dimensionality of the feedforward network. dropout_value (float): Dropout rate for regularization. self_attn (RoPEAttention): Self-attention mechanism using RoPE (Rotary Position Embedding). cross_attn_image (RoPEAttention): Cross-attention mechanism for image processing. linear1 (nn.Linear): First linear layer of the feedforward network. linear2 (nn.Linear): Second linear layer of the feedforward network. norm1 (nn.LayerNorm): Layer normalization for self-attention output. norm2 (nn.LayerNorm): Layer normalization for cross-attention output. norm3 (nn.LayerNorm): Layer normalization for feedforward network output. dropout1 (nn.Dropout): Dropout layer after self-attention. dropout2 (nn.Dropout): Dropout layer after cross-attention. dropout3 (nn.Dropout): Dropout layer after feedforward network. activation (nn.ReLU): Activation function for the feedforward network. pos_enc_at_attn (bool): Flag to add positional encoding at attention. pos_enc_at_cross_attn_queries (bool): Flag to add positional encoding to cross-attention queries. pos_enc_at_cross_attn_keys (bool): Flag to add positional encoding to cross-attention keys. Methods: forward: Performs the full memory attention operation on input tensors. _forward_sa: Performs self-attention on input tensor. _forward_ca: Performs cross-attention between target and memory tensors. Examples: >>> layer = MemoryAttentionLayer(d_model=256, dim_feedforward=2048, dropout=0.1) >>> tgt = torch.randn(1, 100, 256) >>> memory = torch.randn(1, 100, 64) >>> pos = torch.randn(1, 100, 256) >>> query_pos = torch.randn(1, 100, 256) >>> output = layer(tgt, memory, pos, query_pos) >>> print(output.shape) torch.Size([1, 100, 256]) c>[TU]5 XlX lX0l[ SSSS9Ul[ SSSSSS9Ul[R"X5Ul [R"U5Ul [R"X!5Ul [R"U5Ul[R"U5Ul[R"U5Ul[R"U5Ul[R"U5Ul[R"U5Ul[R*"5UlX@lX`lXPlg)a3 Initialize a memory attention layer with self-attention, cross-attention, and feedforward components. Args: d_model (int): Dimensionality of the model. dim_feedforward (int): Dimensionality of the feedforward network. dropout (float): Dropout rate for regularization. pos_enc_at_attn (bool): Whether to add positional encoding at attention. pos_enc_at_cross_attn_keys (bool): Whether to add positional encoding to cross-attention keys. pos_enc_at_cross_attn_queries (bool): Whether to add positional encoding to cross-attention queries. r) embedding_dim num_headsdownsample_rateT@) rope_k_repeatr r r kv_in_dimN)super__init__d_modeldim_feedforward dropout_valuer self_attncross_attn_imagerLinearlinear1Dropoutdropoutlinear2 LayerNormnorm1norm2norm3dropout1dropout2dropout3ReLU activationpos_enc_at_attnpos_enc_at_cross_attn_queriespos_enc_at_cross_attn_keys)selfrrrr'r)r( __class__s i/home/james-whalen/.local/lib/python3.13/site-packages/ultralytics/models/sam/modules/memory_attention.pyrMemoryAttentionLayer.__init__7s(  .$&SAWXY - ! yy: zz'* yy: \\'* \\'* \\'*  7+  7+  7+ '') /-J**D'cURU5nUR(aX2-OU=pEURXEUS9nXRU5-nU$)z^Perform self-attention on input tensor using positional encoding and RoPE attention mechanism.)v)rr'rr")r*tgt query_postgt2qks r, _forward_sa MemoryAttentionLayer._forward_saksLzz#$($8$8 dB~~ad~+MM$'' r.c(0nUS:a%[UR[5(deSU0nURU5nUR"SUR(aXs-OUUR (aX$-OUUS.UD6nXR U5-nU$)zXPerform cross-attention between target and memory tensors using RoPEAttention mechanism.rnum_k_exclude_rope)r4r5r0) isinstancerrr r(r)r#)r*r1memoryr2posr9kwdsr3s r, _forward_ca MemoryAttentionLayer._forward_cass  !d33]CC CC(*<=Dzz#$$ "&"D"Dd$"==fl6    MM$'' r.c URX5nURXXCU5nURU5nURUR UR UR U5555nXRU5-nU$)ae Process input tensors through self-attention, cross-attention, and feedforward network layers. Args: tgt (torch.Tensor): Target tensor for self-attention with shape (N, L, D). memory (torch.Tensor): Memory tensor for cross-attention with shape (N, S, D). pos (Optional[torch.Tensor]): Positional encoding for memory tensor. query_pos (Optional[torch.Tensor]): Positional encoding for target tensor. num_k_exclude_rope (int): Number of keys to exclude from rotary position embedding. Returns: (torch.Tensor): Processed tensor after attention and feedforward layers with shape (N, L, D). )r6r?r!rrr&rr$)r*r1r<r=r2r9r3s r,forwardMemoryAttentionLayer.forwardsu*s.sI