hdSSKJr SSKrSSKJr SSKJrJr "SS\R5r"SS\R5r g) ) annotationsN)nn)MLP LayerNorm2dc^\rSrSrSrS\R SS4S U4SjjjrS SjrS Sjr Sr U=r $) MaskDecoder a Decoder module for generating masks and their associated quality scores using a transformer architecture. This class predicts masks given image and prompt embeddings, utilizing a transformer to process the inputs and generate mask predictions along with their quality scores. Attributes: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. iou_token (nn.Embedding): Embedding for the IoU token. num_mask_tokens (int): Number of mask tokens. mask_tokens (nn.Embedding): Embedding for the mask tokens. output_upscaling (nn.Sequential): Neural network sequence for upscaling the output. output_hypernetworks_mlps (nn.ModuleList): Hypernetwork MLPs for generating masks. iou_prediction_head (nn.Module): MLP for predicting mask quality. Methods: forward: Predict masks given image and prompt embeddings. predict_masks: Internal method for mask prediction. Examples: >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> masks, iou_pred = decoder( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, multimask_output=True ... ) >>> print(f"Predicted masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") c >[TU]5 XlX lX0l[ R "SU5UlUS-Ul[ R "URU5Ul [ R"[ R"XS-SSS9[US-5U"5[ R"US-US-SSS9U"55Ul [ R"[UR5Vs/sHn[!XUS-S5PM sn5Ul[!XURU5Ulgs snf)a Initialize the MaskDecoder module for generating masks and their associated quality scores. Args: transformer_dim (int): Channel dimension for the transformer module. transformer (nn.Module): Transformer module used for mask prediction. num_multimask_outputs (int): Number of masks to predict for disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder)  kernel_sizestrider N)super__init__transformer_dim transformernum_multimask_outputsr Embedding iou_tokennum_mask_tokens mask_tokens SequentialConvTranspose2droutput_upscaling ModuleListrangeroutput_hypernetworks_mlpsiou_prediction_head) selfrrr activationiou_head_depthiou_head_hidden_dim_ __class__s a/home/james-whalen/.local/lib/python3.13/site-packages/ultralytics/models/sam/modules/decoders.pyrMaskDecoder.__init__)s2 .&%:"a94q8<<(<(>> decoder = MaskDecoder(transformer_dim=256, transformer=transformer_module) >>> image_emb = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_emb = torch.rand(1, 2, 256) >>> dense_emb = torch.rand(1, 256, 64, 64) >>> masks, iou_pred = decoder(image_emb, image_pe, sparse_emb, dense_emb, multimask_output=True) >>> print(f"Masks shape: {masks.shape}, IoU predictions shape: {iou_pred.shape}") )image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsr Nr) predict_masksslice) r$r-r.r/r0multimask_outputmasksiou_pred mask_slices r*forwardMaskDecoder.forwardYsg>,,-%=$; - (8U1d^U1a[ aQ)*AzM*c [R"URRURR/SS9nUR S5R URSSS5n[R"XS4SS9n[R"XRSSS9nXt-n[R"X&RSSS9nURuppURXxU5upU SS2SSS24nU SS2SSUR-2SS24nURSS5RXX5nURU5n[UR5Vs/sH"nURU"USS2USS245PM$ nn[R "USS9nURuppUURXX-5-RU SX5nUR#U5nUU4$s snf)z`Predict masks and quality scores using image and prompt embeddings via transformer architecture.rdimr Nr)torchcatrweightr unsqueezeexpandshaperepeat_interleaverr transposeviewrr!r"stackr#)r$r-r.r/r0 output_tokenstokenssrcpos_srcbchwhs iou_token_outmask_tokens_outupscaled_embeddingi hyper_in_listhyper_inr4r5s r*r1MaskDecoder.predict_maskss 4>>#8#8$:J:J:Q:Q"RXYZ %//299:R:X:XYZ:[]_acd MD!L%%&6 QQO+))(LLOKYY a""381a7 QQ)=)=%= >ABmmAq!&&qQ2!2237QVW[WkWkQl- QlAD * *1 -oaAg.F GQl - ;;}!4'-- a.33A!%@@FFq"aS++M:h- s!)G-) r#rrrrr"rrr)rintr nn.ModulerrXr%type[nn.Module]r&rXr'rXreturnNone) r- torch.Tensorr.r]r/r]r0r]r3boolr[!tuple[torch.Tensor, torch.Tensor]) r-r]r.r]r/r]r0r]r[r_) __name__ __module__ __qualname____firstlineno____doc__rGELUrr7r1__static_attributes__ __classcell__r)s@r*rr sB&'&(gg#&.s.s.s # .s $ .s  .s!.s .s.s`+&++#/ + ". +  + ++Z%&%%#/ % ". % + %%r9rc ^\rSrSrSrS\R SSSSSSSSSS4 SU4SjjjrSSS jjrSSS jjr S r S r S r U=r $)SAM2MaskDecodera Transformer-based decoder for predicting instance segmentation masks from image and prompt embeddings. This class extends the functionality of the MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Attributes: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. iou_token (nn.Embedding): Embedding for IOU token. num_mask_tokens (int): Total number of mask tokens. mask_tokens (nn.Embedding): Embedding for mask tokens. pred_obj_scores (bool): Whether to predict object scores. obj_score_token (nn.Embedding): Embedding for object score token. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. output_upscaling (nn.Sequential): Upscaling layers for output. use_high_res_features (bool): Whether to use high-resolution features. conv_s0 (nn.Conv2d): Convolutional layer for high-resolution features (s0). conv_s1 (nn.Conv2d): Convolutional layer for high-resolution features (s1). output_hypernetworks_mlps (nn.ModuleList): List of MLPs for output hypernetworks. iou_prediction_head (MLP): MLP for IOU prediction. pred_obj_score_head (nn.Linear | MLP): Linear layer or MLP for object score prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. Methods: forward: Predict masks given image and prompt embeddings. predict_masks: Predict instance segmentation masks from image and prompt embeddings. _get_stability_scores: Compute mask stability scores based on IoU between thresholds. _dynamic_multimask_via_stability: Dynamically select the most stable mask output. Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) r r Fg?g\(\?c <>[TU]5 XlX lX0l[ R "SU5UlUS-Ul[ R "URU5Ul Xl UR(a[ R "SU5Ul Xl [ R"[ R"XS-SSS9[US-5U"5[ R"US-US-SSS9U"55UlXplU(a<[ R$"XS-SSS9Ul[ R$"XS-SSS9Ul[ R*"[-UR5Vs/sHn[/XUS-S5PM sn5Ul[/UUURUUS9UlUR(a5[ R4"US5UlU (a[/XSS5UlXlXlXlgs snf) a^ Initialize the SAM2MaskDecoder module for predicting instance segmentation masks. This decoder extends the functionality of MaskDecoder, incorporating additional features such as high-resolution feature processing, dynamic multimask output, and object score prediction. Args: transformer_dim (int): Channel dimension of the transformer. transformer (nn.Module): Transformer used to predict masks. num_multimask_outputs (int): Number of masks to predict when disambiguating masks. activation (Type[nn.Module]): Type of activation to use when upscaling masks. iou_head_depth (int): Depth of the MLP used to predict mask quality. iou_head_hidden_dim (int): Hidden dimension of the MLP used to predict mask quality. use_high_res_features (bool): Whether to use high-resolution features. iou_prediction_use_sigmoid (bool): Whether to use sigmoid for IOU prediction. dynamic_multimask_via_stability (bool): Whether to use dynamic multimask via stability. dynamic_multimask_stability_delta (float): Delta value for dynamic multimask stability. dynamic_multimask_stability_thresh (float): Threshold for dynamic multimask stability. pred_obj_scores (bool): Whether to predict object scores. pred_obj_scores_mlp (bool): Whether to use MLP for object score prediction. use_multimask_token_for_obj_ptr (bool): Whether to use multimask token for object pointer. Examples: >>> transformer = nn.TransformerEncoder(nn.TransformerEncoderLayer(d_model=256, nhead=8), num_layers=6) >>> decoder = SAM2MaskDecoder(transformer_dim=256, transformer=transformer) >>> print(decoder) r rrrrr )sigmoidN)rrrrrrrrrrpred_obj_scoresobj_score_tokenuse_multimask_token_for_obj_ptrrrrruse_high_res_featuresConv2dconv_s0conv_s1r r!rr"r#Linearpred_obj_score_headdynamic_multimask_via_stability!dynamic_multimask_stability_delta"dynamic_multimask_stability_thresh)r$rrrr%r&r'rqiou_prediction_use_sigmoidrwrxryrnpred_obj_scores_mlprpr(r)s r*rSAM2MaskDecoder.__init__sX .&%:"a94q8<<(<(Hc URUUUUUUS9uppU(aUSS2SS2SS2SS24nU SS2SS24n ORUR(a%UR(dURX5upOUSS2SS2SS2SS24nU SS2SS24n U(aUR(a U SS2SS24n O U SS2SS24n XX4$)a Predict masks given image and prompt embeddings. Args: image_embeddings (torch.Tensor): Embeddings from the image encoder with shape (B, C, H, W). image_pe (torch.Tensor): Positional encoding with the shape of image_embeddings (B, C, H, W). sparse_prompt_embeddings (torch.Tensor): Embeddings of the points and boxes with shape (B, N, C). dense_prompt_embeddings (torch.Tensor): Embeddings of the mask inputs with shape (B, C, H, W). multimask_output (bool): Whether to return multiple masks or a single mask. repeat_image (bool): Flag to repeat the image embeddings. high_res_features (list[torch.Tensor] | None, optional): Optional high-resolution features. Returns: masks (torch.Tensor): Batched predicted masks with shape (B, N, H, W). iou_pred (torch.Tensor): Batched predictions of mask quality with shape (B, N). sam_tokens_out (torch.Tensor): Batched SAM token for mask output with shape (B, N, C). object_score_logits (torch.Tensor): Batched object score logits with shape (B, 1). Examples: >>> image_embeddings = torch.rand(1, 256, 64, 64) >>> image_pe = torch.rand(1, 256, 64, 64) >>> sparse_prompt_embeddings = torch.rand(1, 2, 256) >>> dense_prompt_embeddings = torch.rand(1, 256, 64, 64) >>> decoder = SAM2MaskDecoder(256, transformer) >>> masks, iou_pred, sam_tokens_out, obj_score_logits = decoder.forward( ... image_embeddings, image_pe, sparse_prompt_embeddings, dense_prompt_embeddings, True, False ... ) )r-r.r/r0 repeat_imagehigh_res_featuresNr r)r1rwtraining _dynamic_multimask_via_stabilityrp) r$r-r.r/r0r3r~rr4r5rRobject_score_logitssam_tokens_outs r*r7SAM2MaskDecoder.forward8sLAE@R@R-%=$;%/ ASA = !QRA+&E12H  1 1$--"CCETOE8!QqS!Q,'E1Q3'H  D D,QU3N-Q!V4NCCr9c SnUR(aW[R"URRUR RUR R/SS9nSnO?[R"UR RUR R/SS9nURS5RURSSS5n[R"X4SS9n U(a#[R"XRSSS9n O$URSU RS:XdeUn X-n URSS:XdS5e[R"X)RSSS9n U RuppURXU 5unn USS2USS24nUSS2US-US-UR-2SS24nU RSS5RXX5n UR(aUcUR!U 5nO?UR unnnnnUunnU"U"U"U 5U-55nU"U"U5U-5n[#UR5Vs/sH"nUR$U"USS2USS245PM$ nn[R&"USS9nURuppUURXX-5-RU SX5nUR)U5nUR(a$US:XdeUR+USS2SSS245n O"SUR-URSS5-n UUUU 4$s snf) zYPredict instance segmentation masks from image and prompt embeddings using a transformer.rr;r r=z@image_pe should have size 1 in batch dim (from `get_dense_pe()`)Nrg$@)rnr>r?ror@rrrArBrCrDrrrErFrqrr!r"rGr#rvnew_ones)!r$r-r.r/r0r~rsrHrIrJrKrLrMrNrOrPrQrRrSdc1ln1act1dc2act2feat_s0feat_s1rTrUrVr4r5rs! r*r1SAM2MaskDecoder.predict_masks}sE    !II((//NN))$$++  MA!IIt~~'<'N>N>U>U&V\]^M%//299:R:X:XYZ:[]_acd MD!L ))*:LLOQRSC#))!, Q? ??"C+~~a A%i'ii%))(LLOKYY a""38C1a7 QAQ1E1E)E FIJmmAq!&&qQ2))->-F!%!6!6s!; (,(=(= %CdC0 GW!%c#c(W*<&=!> !%c*<&=&G!H RWW[WkWkQl- QlAD * *1 -oaAg.F GQl - ;;}!4'-- a.33A!%@@FFq"aS++M:   6M6"&":":2aAg;"G #'):):8>>!;La)P"P h1DDD!- s,)Mc URS5nURn[R"X:SS9R 5n[R"X*:SS9R 5n[R "US:X4- S5$)zNCompute mask stability scores based on IoU between upper and lower thresholds.r=r;rg?)flattenrxr>sumfloatwhere)r$ mask_logitsstability_deltaarea_iarea_us r*_get_stability_scores%SAM2MaskDecoder._get_stability_scoressq!))"- @@;8bAGGI;)99rBHHJ{{6A:v<>> decoder = SAM2MaskDecoder(...) >>> all_mask_logits = torch.rand(2, 4, 256, 256) # 2 images, 4 masks each >>> all_iou_scores = torch.rand(2, 4) >>> mask_logits, iou_scores = decoder._dynamic_multimask_via_stability(all_mask_logits, all_iou_scores) >>> print(mask_logits.shape, iou_scores.shape) torch.Size([2, 1, 256, 256]) torch.Size([2, 1]) Nr r=r;r)device).NN) r>argmaxarangerCrrArryr expand_as)r$all_mask_logitsall_iou_scoresmultimask_logitsmultimask_iou_scoresbest_scores_inds batch_indsbest_multimask_logitsbest_multimask_iou_scoressinglemask_logitssinglemask_iou_scoresstability_scores is_stablemask_logits_outiou_scores_outs r*r0SAM2MaskDecoder._dynamic_multimask_via_stabilitysD6+1ab!Q;7-ae4 <<(<"E\\"6"<"rultralytics.nn.modulesrrModulerrjr9r*rs8# 3`"))`FS/biiS/r9