models mmd 3x mask rcnn_swin t p4 w7_fpn_1x_coco

mmd-3x-mask-rcnn_swin-t-p4-w7_fpn_1x_coco

Overview

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures.

The above abstract is from mmdetection website. Review the original-model-card to understand the data used to train the model, evaluation metrics, license, intended uses, limitations and bias before using the model.

Inference samples

Inference type	Python sample (Notebook)	CLI with YAML
Real time	image-instance-segmentation-online-endpoint.ipynb	image-instance-segmentation-online-endpoint.sh
Batch	image-instance-segmentation-batch-endpoint.ipynb	image-instance-segmentation-batch-endpoint.sh

Finetuning samples

Task	Use case	Dataset	Python sample (Notebook)	CLI with YAML
Image instance segmentation	Image instance segmentation	fridgeObjects	fridgeobjects-instance-segmentation.ipynb	fridgeobjects-instance-segmentation.sh

Model Evaluation

Task	Use case	Dataset	Python sample (Notebook)
Image instance segmentation	Image instance segmentation	fridgeObjects	image-instance-segmentation.ipynb

Sample inputs and outputs (for real-time inference)

Sample input

{
  "input_data": {
    "columns": [
      "image"
    ],
    "index": [0, 1],
    "data": ["image1", "image2"]
  }
}

Note: "image1" and "image2" string should be in base64 format or publicly accessible urls.

Sample output

[
    {
        "boxes": [
            {
                "box": {
                    "topX": 0.1,
                    "topY": 0.2,
                    "bottomX": 0.8,
                    "bottomY": 0.7
                },
                "label": "carton",
                "score": 0.98,
                "polygon": [
                    [ 0.576, 0.680,  …]
                ]
            }
        ]
    },
    {
        "boxes": [
            {
                "box": {
                    "topX": 0.2,
                    "topY": 0.3,
                    "bottomX": 0.6,
                    "bottomY": 0.5
                },
                "label": "can",
                "score": 0.97,
                "polygon": [
                    [ 0.58, 0.7,  …]
                ]
            }
        ]
    }
]

Note: Please refer to instance segmentation output data schema for more detail.

Model inference - visualization for a sample image

Version: 8

Tags

Preview license : apache-2.0 model_specific_defaults : ordereddict({'apply_deepspeed': 'false', 'apply_ort': 'false'}) task : image-segmentation

View in Studio: https://ml.azure.com/registries/azureml/models/mmd-3x-mask-rcnn_swin-t-p4-w7_fpn_1x_coco/version/8

License: apache-2.0

Properties

SHA: fff646d3dda72d8c794471bfaa75b4db0adb7610

datasets: COCO

evaluation-min-sku-spec: 4|1|28|176

evaluation-recommended-sku: Standard_NC6s_v3

finetune-min-sku-spec: 4|1|28|176

finetune-recommended-sku: Standard_NC6s_v3

finetuning-tasks: image-instance-segmentation

inference-min-sku-spec: 4|0|14|28

inference-recommended-sku: Standard_DS3_v2, Standard_D4a_v4, Standard_D4as_v4, Standard_DS4_v2, Standard_D8a_v4, Standard_D8as_v4, Standard_DS5_v2, Standard_D16a_v4, Standard_D16as_v4, Standard_D32a_v4, Standard_D32as_v4, Standard_D48a_v4, Standard_D48as_v4, Standard_D64a_v4, Standard_D64as_v4, Standard_D96a_v4, Standard_D96as_v4, Standard_FX4mds, Standard_F8s_v2, Standard_FX12mds, Standard_F16s_v2, Standard_F32s_v2, Standard_F48s_v2, Standard_F64s_v2, Standard_F72s_v2, Standard_FX24mds, Standard_FX36mds, Standard_FX48mds, Standard_E4s_v3, Standard_E8s_v3, Standard_E16s_v3, Standard_E32s_v3, Standard_E48s_v3, Standard_E64s_v3, Standard_NC4as_T4_v3, Standard_NC6s_v3, Standard_NC8as_T4_v3, Standard_NC12s_v3, Standard_NC16as_T4_v3, Standard_NC24s_v3, Standard_NC64as_T4_v3, Standard_NC24ads_A100_v4, Standard_NC48ads_A100_v4, Standard_NC96ads_A100_v4, Standard_ND96asr_v4, Standard_ND96amsr_A100_v4, Standard_ND40rs_v2

model_id: mmd-3x-mask-rcnn_swin-t-p4-w7_fpn_1x_coco