Skip to content

Latest commit

 

History

History
173 lines (126 loc) · 8.71 KB

README.md

File metadata and controls

173 lines (126 loc) · 8.71 KB
Raw

Model archiver for MMS

Contents of this Document

Other Relevant Documents

Overview

A key feature of MMS is the ability to package all model artifacts into a single model archive file. It is a separate command line interface (CLI), model-archiver, that can take model checkpoints and package them into a .mar file. This file can then be redistributed and served by anyone using MMS. It takes in the following model artifacts: a model composed of one or more files, the description of the model's inputs in the form of a signature file, a service file describing how to handle inputs and outputs, and other optional assets that may be required to serve the model. The CLI creates a .mar file that MMS's server CLI uses to serve the models.

Important: Make sure you try the Quick Start: Creating a Model Archive tutorial for a short example of using model-archiver.

MMS can support any arbitrary model file. It is the custom service code's responsibility to locate and load the model files. The following information is required to create a standalone model archive:

  1. Model name
  2. Model path
  3. Handler

Model Archiver Command Line Interface

Now let's cover the details on using the CLI tool: model-archiver.

Here is an example usage with the squeezenet_v1.1 model archive which you can download or create by following the example in the main README:

model-archiver --model-name squeezenet_v1.1 --model-path squeezenet --handler mxnet_vision_service:handle

Arguments

$ model-archiver -h
usage: model-archiver [-h] --model-name MODEL_NAME --model-path MODEL_PATH
                      --handler HANDLER [--runtime {python,python2,python3}]
                      [--export-path EXPORT_PATH] [-f]

Model Archiver Tool

optional arguments:
  -h, --help            show this help message and exit
  --model-name MODEL_NAME
                        Exported model name. Exported file will be named as
                        model-name.mar and saved in current working directory
                        if no --export-path is specified, else it will be
                        saved under the export path
  --model-path MODEL_PATH
                        Path to the folder containing model related files.
  --handler HANDLER     Handler path to handle custom MMS inference logic.
  --runtime {python,python2,python3}
                        The runtime specifies which language to run your
                        inference code on. The default runtime is
                        RuntimeType.PYTHON. At the present moment we support
                        the following runtimes python, python2, python3
  --export-path EXPORT_PATH
                        Path where the exported .mar file will be saved. This
                        is an optional parameter. If --export-path is not
                        specified, the file will be saved in the current
                        working directory.
  --archive-format {tgz,default}
                        The format in which the model artifacts are archived.
                        "tgz": This creates the model-archive in <model-name>.tar.gz format.
                        If platform hosting MMS requires model-artifacts to be in ".tar.gz"
                        use this option.
                        "no-archive": This option creates an non-archived version of model artifacts
                        at "export-path/{model-name}" location. As a result of this choice,
                        MANIFEST file will be created at "export-path/{model-name}" location
                        without archiving these model files
                        "default": This creates the model-archive in <model-name>.mar format.
                        This is the default archiving format. Models archived in this format
                        will be readily hostable on native MMS.
  -f, --force           When the -f or --force flag is specified, an existing
                        .mar file with same name as that provided in --model-
                        name in the path specified by --export-path will
                        overwritten

Artifact Details

MAR-INF

MAR-INF is a reserved folder name that will be used inside .mar file. This folder contains the model archive metadata files. Users should avoid using MAR-INF in their model path.

Runtime

Model name

A valid model name must begin with a letter of the alphabet and can only contains letters, digits, underscores (_), dashes (-) and periods (.).

Note: The model name can be overridden when you register the model with Register Model API.

Model path

A folder that contains all necessary files needed to run inference code for the model. All the files and sub-folders (except excluded files) will be packaged into the .mar file.

excluded files

The following types of file will be excluded during model archive packaging:

  1. hidden files
  2. Mac system files: __MACOSX and .DS_Store
  3. MANIFEST.json
  4. python compiled byte code (.pyc) files and cache folder pycache

handler

A handler is a python entry point that MMS can invoke to execute inference code. The format of a Python handler is:

  • python_module_name[:function_name] (for example: lstm-service:handle).

The function name is optional if the provided python module follows one of predefined conventions:

  1. There is a handle() function available in the module
  2. The module contains only one Class and that class contains a handle() function.

Further details and specifications are found on the custom service page.

Creating a Model Archive

1. Download these sample SqueezeNet model artifacts (if you don't have them handy)

mkdir squeezenet

curl -o squeezenet/squeezenet_v1.1-symbol.json https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/squeezenet_v1.1-symbol.json
curl -o squeezenet/squeezenet_v1.1-0000.params https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/squeezenet_v1.1-0000.params
curl -o squeezenet/signature.json https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/signature.json
curl -o squeezenet/synset.txt https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/synset.txt

The downloaded model artifact files are:

  • Model Definition (json file) - contains the layers and overall structure of the neural network.
  • Model Params and Weights (params file) - contains the parameters and the weights.
  • Model Signature (json file) - defines the inputs and outputs that MMS is expecting to hand-off to the API.
  • assets (text files) - auxiliary files that support model inference such as vocabularies, labels, etc. These vary depending on the model.

2. Download the model archiver source

git clone https://github.com/awslabs/multi-model-server.git

3. Prepare your model custom service code

You can implement your own model customer service code with a model archive entry point. Here we are going to use the MXNet vision service model_service_template. This template is one of several provided with MMS. Download the template and place it in your squeezenet folder.

cp -r multi-model-server/examples/model_service_template/* squeezenet/

4. Package your model

With the model artifacts available locally, you can use the model-archiver CLI to generate a .mar file that can be used to serve an inference API with MMS.

In this next step we'll run model-archiver and tell it our model's prefix is squeezenet_v1.1 with the model-name argument. Then we're giving it the model-path to the model's assets.

Note: For mxnet models, model-name must match prefix of the symbol and param file name.

model-archiver --model-name squeezenet_v1.1 --model-path squeezenet --handler mxnet_vision_service:handle

This will package all the model artifacts files located in the squeezenet directory and output squeezenet_v1.1.mar in the current working directory. This .mar file is all you need to run MMS, serving inference requests for a simple image recognition API. Go back to the Serve a Model tutorial and try to run this model archive that you just created!