Skip to content

Latest commit

 

History

History
470 lines (455 loc) · 54.3 KB

README.md

File metadata and controls

470 lines (455 loc) · 54.3 KB

Annotation Converters

Annotation converter is a function which converts annotation file to suitable for metric evaluation format. Each annotation converter expects specific annotation file format or data structure, which depends on original dataset. If converter for your data format is not supported by Accuracy Checker, you can provide your own annotation converter. Each annotation converter has parameters available for configuration.

Process of conversion can be implemented in two ways:

  • via configuration file
  • via command line

Describing Annotation Conversion in Configuration File

Annotation conversion can be provided in dataset section your configuration file to convert annotation in-place before every evaluation. Each conversion configuration should contain converter field filled selected converter name and provide converter specific parameters (more details in supported converters section). All paths can be prefixed via command line with -s, --source argument.

You can additionally use optional parameters like:

  • subsample_size - Dataset subsample size. You can specify the number of ground truth objects or dataset ratio in percentage. Please, be careful to use this option, some datasets does not support subsampling. You can also specify subsample_seed if you want to generate subsample with specific random seed.
  • annotation - path to store converted annotation pickle file. You can use this parameter if you need to reuse converted annotation to avoid subsequent conversions.
  • dataset_meta - path to store meta information about converted annotation if it is provided.
  • analyze_dataset - flag which allow to get statistics about converted dataset. Supported annotations: ClassificationAnnotation, DetectionAnnotation, MultiLabelRecognitionAnnotation, RegressionAnnotation. Default value is False.

Example of usage:

   annotation_conversion:
     # Converter name which will be called for conversion.
     converter: sample
     # Converter specific parameters, can be different depend on converter realization.
     data_dir: sample/sample_dataset
   # (Optional) subsample generation. Can be also used with prepared annotation file.
   subsample_size: 1000
   # (Optional) paths to store annotation files for following usage. In the next evaluation these files will be directly used instead running conversion.
   annotation: sample_dataset.pickle
   dataset_meta: sample_dataset.json

Conversing Process via Command Line

The command line for annotation conversion looks like:

convert_annotation <converter_name> <converter_specific parameters>

All converter specific options should have format --<parameter_name> <parameter_value> You may refer to -h, --help to full list of command line options. Some optional arguments are:

  • -o, --output_dir - directory to save converted annotation and meta info.
  • -a, --annotation_name - annotation file name.
  • -m, --meta_name - meta info file name.

Supported Converters

Accuracy Checker supports following list of annotation converters and specific for them parameters:

  • cifar - converts CIFAR classification dataset to ClassificationAnnotation
    • data_batch_file - path to pickle file which contain dataset batch (e.g. test_batch)
    • has_background - allows to add background label to original labels (Optional, default value is False).
    • convert_images - allows to convert images from pickle file to user specified directory (default value is False).
    • converted_images_dir - path to converted images location.
    • num_classes - the number of classes in the dataset - 10 or 100 (Optional, default 10)
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • mnist_csv - convert MNIST dataset for handwritten digit recognition stored in csv format to ClassificationAnnotation.
    • annotation_file - path to dataset file in csv format.
    • convert_images - allows to convert images from annotation file to user specified directory (default value is False).
    • converted_images_dir - path to converted images location if enabled convert_images.
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding). Optional, more details in Customizing dataset meta section.
  • fashion_mnist - convert Fashion-MNIST dataset to ClassificationAnnotation.
    • annotation_file - path to labels file in binary format.
    • data_file - path to images file in binary format.
    • convert_images - allows to convert images from data file to user specified directory (default value is False).
    • converted_images_dir - path to converted images location if enabled convert_images.
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding). Optional, more details in Customizing dataset meta section.
  • imagenet - convert ImageNet dataset for image classification task to ClassificationAnnotation.
    • annotation_file - path to annotation in txt format.
    • labels_file - path to file with word description of labels (synset_words).
    • has_background - allows to add background label to original labels and convert dataset for 1001 classes instead 1000 (default value is False).
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • voc_detection - converts Pascal VOC annotation for detection task to DetectionAnnotation.
    • imageset_file - path to file with validation image list.
    • annotations_dir - path to directory with annotation files.
    • images_dir - path to directory with images related to devkit root (default JPEGImages).
    • has_background - allows convert dataset with/without adding background_label. Accepted values are True or False. (default is True)
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • voc_segmentation - converts Pascal VOC annotation for semantic segmentation task to SegmentationAnnotation.
    • imageset_file - path to file with validation image list.
    • images_dir - path to directory with images related to devkit root (default JPEGImages).
    • mask_dir - path to directory with ground truth segmentation masks related to devkit root (default SegmentationClass).
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section. Note: Since OpenVINO 2020.4 the converter behaviour changed. data_source parameter of dataset should contains directory for images only, if you have segmentation mask in separated location, please use segmentation_masks_source for specifying gt masks location.
  • mscoco_detection - converts MS COCO dataset for object detection task to DetectionAnnotation.
    • annotation_file - path to annotation file in json format.
    • has_background - allows convert dataset with/without adding background_label. Accepted values are True or False. (default is False).
    • use_full_label_map - allows to use original label map (with 91 object categories) from paper instead public available(80 categories).
    • sort_annotations - allows to save annotations in a specific order: ascending order of image id or ascending order of image size.
    • sort_key - key by which annotations will be sorted(supported keys are image_id and image_size, default is image_id).
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • mscoco_segmentation - converts MS COCO dataset for object instance segmentation task to CocoInstanceSegmentationAnnotation.
    • annotation_file - path to annotation file in json format.
    • has_background - allows convert dataset with/without adding background_label. Accepted values are True or False. (default is False).
    • use_full_label_map - allows to use original label map (with 91 object categories) from paper instead public available(80 categories).
    • sort_annotations - allows to save annotations in a specific order: ascending order of image id or ascending order of image size.
    • sort_key - key by which annotations will be sorted(supported keys are image_id and image_size, default is image_id).
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
    • semantic_only - converts MS COCO dataset annotation to SegmentationAnnotation. (Optional, default value is False)
    • masks_dir - path to store segmentation masks in semantic_only mode
  • mscoco_mask_rcnn - converts MS COCO dataset to ContainerAnnotation with DetectionAnnotation and CocoInstanceSegmentationAnnotation named detection_annotation and segmentation_annotation respectively.
    • annotation_file - path to annotation file in json format.
    • has_background - allows convert dataset with/without adding background_label. Accepted values are True or False. (default is False).
    • use_full_label_map - allows to use original label map (with 91 object categories) from paper instead public available(80 categories).
    • sort_annotations - allows to save annotations in a specific order: ascending order of image id or ascending order of image size.
    • sort_key - key by which annotations will be sorted(supported keys are image_id and image_size, default is image_id).
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • mscoco_keypoints - converts MS COCO dataset for keypoints localization task to PoseEstimationAnnotation.
    • annotation_file - path to annotation file in json format.
    • sort_annotations - allows to save annotations in a specific order: ascending order of image id or ascending order of image size.
    • sort_key - key by which annotations will be sorted(supported keys are image_id and image_size, default is image_id).
  • wider - converts from Wider Face dataset to DetectionAnnotation.
    • annotation_file - path to txt file, which contains ground truth data in WiderFace dataset format.
    • label_start - specifies face label index in label map. Default value is 1. You can provide another value, if you want to use this dataset for separate label validation, in case when your network predicts other class for faces.
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • detection_opencv_storage - converts detection annotation stored in Detection OpenCV storage format to DetectionAnnotation.
    • annotation_file - path to annotation in xml format.
    • image_names_file - path to txt file, which contains image name list for dataset.
    • label_start - specifies label index start in label map. Default value is 1. You can provide another value, if you want to use this dataset for separate label validation.
    • background_label - specifies which index will be used for background label. You can not provide this parameter if your dataset has not background label.
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • cityscapes - converts CityScapes Dataset to SegmentationAnnotation.
    • dataset_root_dir - path to dataset root.
    • images_subfolder - path from dataset root to directory with validation images (Optional, default imgsFine/leftImg8bit/val).
    • masks_subfolder - path from dataset root to directory with ground truth masks (Optional, gtFine/val).
    • masks_suffix - suffix for mask file names (Optional, default _gtFine_labelTrainIds).
    • images_suffix - suffix for image file names (Optional, default _leftImg8bit).
    • use_full_label_map - allows to use full label map with 33 classes instead train label map with 18 classes (Optional, default False).
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • mapillary_20 - converts Mapillary dataset contained 20 classes to SegmentationAnnotation.
    • data_dir - path to dataset root folder. Relative paths to images and masks directory determine as imgs and masks respectively. In way when images and masks are located in non default directories, you can use parameters described below.
    • images_dir - path to images folder.
    • mask_dir - path to ground truth mask folder.
    • images_subfolder - sub-directory for images(Optional, default imgs)
    • mask_subfolder - sub-directory for ground truth mask(Optional, default masks)
  • mapillary_vistas - converts Mapillary Vistas dataset contained 20 classes to SegmentationAnnotation.
    • data_dir - path to dataset root folder. Relative paths to images and masks directory determine as images and labels respectively. In way when images and masks are located in non default directories, you can use parameters described below.
    • images_dir - path to images folder.
    • mask_dir - path to ground truth mask folder.
    • images_subfolder - sub-directory for images(Optional, default images)
    • mask_subfolder - sub-directory for ground truth mask(Optional, default labels)
  • vgg_face - converts VGG Face 2 dataset for facial landmarks regression task to FacialLandmarksAnnotation.
    • landmarks_csv_file - path to csv file with coordinates of landmarks points.
    • bbox_csv_file - path to cvs file which contains bounding box coordinates for faces (optional parameter).
  • lfw - converts Labeled Faces in the Wild dataset for face reidentification to ReidentificationClassificationAnnotation.
    • pairs_file - path to file with annotation positive and negative pairs.
    • train_file - path to file with annotation positive and negative pairs used for network train (optional parameter).
    • landmarks_file - path to file with facial landmarks coordinates for annotation images (optional parameter).
  • face_recognition_bin - converts preprocessed face recognition dataset stored in binary format to ReidentificationClassificationAnnotation.
    • bin_file - file with dataset. Example of datasets can be found here.
    • images_dir - directory for saving converted images (Optional, used only if convert_images enabled, if not provided <dataset_root>/converted_images will be used)
    • convert_images - allows decode and save images.
  • mars - converts MARS person reidentification dataset to ReidentificationAnnotation.
    • data_dir - path to data directory, where gallery (bbox_test) and query subdirectories are located.
  • market1501_reid - converts Market1501 person reidentification dataset to ReidentificationAnnotation.
    • data_dir - path to data directory, where gallery (bounding_box_test) and query subdirectories are located.
  • veri776_reid - converts VeRi776 vehicle reidentification dataset to ReidentificationAnnotation.
    • data_dir - path to data directory, where gallery (image_test) and image_query subdirectories are located.
  • image_processing - converts dataset for common single image processing tasks (e.g. image denoising, style transferring) to ImageProcessingAnnotation. This converter is suitable for tasks where model output produced on specific input image should be compared with target image.
    • data_dir - path to folder, where images in low and high resolution are located.
    • input_suffix - input file name's suffix (default in).
    • target_suffix - target ground truth file name's suffix (default out).
    • annotation_loader - which library will be used for ground truth image reading. Supported: opencv, pillow (Optional. Default value is pillow). Note, color space of image depends on loader (OpenCV uses BGR, Pillow uses RGB for image reading).
  • super_resolution - converts dataset for single image super resolution task to SuperResolutionAnnotation.
    • data_dir - path to folder, where images in low and high resolution are located.
    • lr_dir - path to directory, where images in low resolution are located.
    • hr_dir - path to directory, where images in high resolution are located. Note: inside converted annotation, path to directory is not stored, only file name, please use additional_data_source for providing prefix.
    • upsampled_dir - path to directory, where upsampled images are located, if 2 streams used.
    • ignore_suffixes - matched low resolution, high resolution image located in different directories without usage suffixes, using numeric ids (Optional, default False).
    • lr_suffix - low resolution file name's suffix (default lr).
    • hr_suffix - high resolution file name's suffix (default hr).
    • annotation_loader - which library will be used for ground truth image reading. Supported: opencv, pillow, dicom. (Optional. Default value is pillow). Note, color space of image depends on loader (OpenCV uses BGR, Pillow uses RGB for image reading).
    • two_streams - enable 2 input streams where usually first for original image and second for upsampled image. (Optional, default False).
    • upsample_suffix - upsample images file name's suffix (default upsample).
  • super_resolution_dir_based - converts dataset for single image super resolution task to SuperResolutionAnnotation which have directory based structure (high resolution and low resolution images located on separated directories and matches by name or numeric id). The main difference between this converter and super_resolution in data organization. super_resolution converter should be used if all high and low resolution images located in the same dir and have difference in suffixes.
    • annotation_loader - which library will be used for ground truth image reading. Supported: opencv, pillow, dicom. (Optional. Default value is pillow). Note, color space of image depends on loader (OpenCV uses BGR, Pillow uses RGB for image reading).
    • two_streams - enable 2 input streams where usually first for original image and second for upsampled image. (Optional, default False).
    • images_dir - path to dataset root, where directories with low and high resolutions are located.
    • lr_dir - path to directory, where images in low resolution are located (Optional, default <images_dir>/LR).
    • hr_dir - path to directory, where images in high resolution are located (Optional, default <images_dir>/HR). Note: inside converted annotation, path to directory is not stored, only file name, please use additional_data_source for providing prefix.
    • upsampled_dir - path to directory, where upsampled images are located, if 2 streams used (Optional, default <images_dir>/upsample).
    • relaxed_names - allow to use more relaxed search of high resolution or/and upsampled images matching only numeric ids. Optional, by default full name matching required.
  • multi_frame_super_resolution - converts dataset for super resolution task with multiple input frames usage.
    • data_dir - path to folder, where images in low and high resolution are located.
    • lr_suffix - low resolution file name's suffix (default lr).
    • hr_suffix - high resolution file name's suffix (default hr).
    • annotation_loader - which library will be used for ground truth image reading. Supported: opencv, pillow (Optional. Default value is pillow). Note, color space of image depends on loader (OpenCV uses BGR, Pillow uses RGB for image reading).
    • number_input_frames - the number of input frames per inference.
    • reference_frame - the id of frame in sample frame sequence used for matching with high resolution. You can define number of frame or choose one of predefined: first (first frame used as reference), middle (num_frames / 2), last (last frame in sequence).
  • multi_target_super_resolution - converts dataset for single image super resolution task with multiple target resolutions to ContainerAnnotation with SuperResolutionAnnotation representations for each target resolution.
    • data_dir - path to dataset root, where directories with low and high resolutions are located.
    • lr_path - path to low resolution images directory relative to data_dir.
    • hr_mapping - dictionary which represent mapping between target resolution and directory with images. Keys are also used as keys for ContainerAnnotation. All paths should be relative to data_dir.
  • icdar_detection - converts ICDAR13 and ICDAR15 datasets for text detection challenge to TextDetectionAnnotation.
    • data_dir - path to folder with annotations on txt format.
    • word_spotting - if it is true then transcriptions that have lengths less than 3 symbols or transcriptions containing non-alphanumeric symbols will be marked as difficult.
  • icdar13_recognition - converts ICDAR13 dataset for text recognition task to CharacterRecognitionAnnotation.
    • annotation_file - path to annotation file in txt format.
  • unicode_character_recognition - converts Kondate dataset and Nakayosi for handwritten Japanese text recognition task , and SCUT-EPT for handwritten simplified Chinese text recognition task to CharacterRecognitionAnnotation.
    • annotation_file - path to annotation file in txt format.
    • decoding_char_file - path to decoding_char_file, consisting of all supported characters separated by '\n' in txt format.
  • brats - converts BraTS dataset format to BrainTumorSegmentationAnnotation format.
    • data_dir - dataset root directory, which contain subdirectories with validation data (imagesTr) and ground truth labels (labelsTr). Optionally you can provide relative path for these subdirectories (if they have different location) using image_folder and mask_folder parameters respectively.
    • mask_channels_first - allows read gt mask nifti files and transpose in order where channels first (Optional, default False)
    • labels_file - path to file, which contains labels (optional, if omitted no labels will be shown)
  • movie_lens_converter - converts Movie Lens Datasets format to HitRatioAnnotation format.
    • rating_file - path to file which contains movieId with top score for each userID (for example ml-1m-test-ratings.csv)
    • negative_file - path to file which contains negative examples.
    • users_max_number - the number of users which will be used for validation (Optional, it gives opportunity to cut list of users. If argument is not provided, full list of users will be used.).
  • brats_numpy - converts Brain Tumor Segmentation dataset to BrainTumorSegmentationAnnotation. This converter works with Numpy representation of BraTS dataset.
    • data_dir - path to dataset root directory.
    • ids_file - path to file, which contains names of images in dataset
    • labels_file - path to file, which contains labels (optional, if omitted no labels will be shown)
    • data_suffix - suffix for files with data (default _data_cropped)
    • label_suffix - suffix for files with groundtruth data (default _label_cropped)
    • boxes_file - path to file with brain boxes (optional). Set this option with including postprocessor segmentation-prediction-resample(see Postprocessors).
  • wmt - converts WMT dataset for Machine Translation task to MachineTranslationAnnotation.
    • input_file - path to file which contains input sentences tokens for translation.
    • reference_file - path to file with reference for translation.
  • common_semantic_segmentation - converts general format of datasets for semantic segmentation task to SegmentationAnnotation. The converter expects following dataset structure:
    1. images and GT masks are located in separated directories (e.g. <dataset_root>/images for images and <dataset_root>/masks for masks respectively)
    2. images and GT masks has common part in names and can have difference in prefix and postfix (e.g. image name is image0001.jpeg, mask for it is gt0001.png are acceptable. In this case base_part - 0001, image_prefix - image, image_postfix - .jpeg, mask_prefix - gt, mask_postfix - .png)
    • images_dir - path to directory with images.
    • masks_dir - path to directory with GT masks.
    • image_prefix - prefix part for image file names. (Optional, default is empty).
    • image_postfix - postfix part for image file names (optional, default is .png).
    • mask_prefix - prefix part for mask file names. (Optional, default is empty).
    • image_postfix - postfix part for mask file names (optional, default is .png).
    • mask_loader - the way how GT mask should be loaded. Supported methods: pillow, opencv, nifti, numpy, scipy.
    • dataset_meta_file - path to json file with prepared dataset meta info. It should contains label_map key with dictionary in format class_id: class_name and optionally segmentation_colors (if your dataset uses color encoding). Segmentation colors is a list of channel-wise values for each class. (e.g. if your dataset has 3 classes in BGR colors, segmentation colors for it will looks like: [[255, 0, 0], [0, 255, 0], [0, 0, 255]]). (Optional, you can provide self-created file as dataset_meta in your config). Note: since OpenVINO 2020.4 converter behaviour changed. data_source parameter of dataset should contains directory for images only, if you have segmentation mask in separated location, please use segmentation_masks_source for specifying gt masks location.
  • camvid - converts CamVid dataset format to SegmentationAnnotation.
    • annotation_file - file in txt format which contains list of validation pairs (<path_to_image> <path_to_annotation> separated by space)
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • image_retrieval - converts dataset for image retrieval task to ReidentificationAnnotation. Dataset should have following structure:
    1. the dataset root directory contains 2 subdirectory named gallery and queries for gallery images and query images respectively.
    2. Every of these subdirectories should contains text file with list of pairs: <path_to_image> <image_ID> (image_path and image_ID should be separated by space), where <path_to_image> is path to the image related dataset root, <image_ID> is the number which represent image id in the gallery.
    • data_dir - path to dataset root directory.
    • gallery_annotation_file - file with gallery images and IDs concordance in txt format (Optional, default value is <data_dir>/gallery/list.txt)
    • queries_annotation_file - file with queries images and IDs concordance in txt format (Optional, default value is <data_dir>/queries/list.txt)
  • cvat_object_detection - converts CVAT XML annotation version 1.1 format for images to DetectionAnnotation.
    • annotation_file - path to xml file in appropriate format.
    • has_background - allows prepend original labels with special class represented background and convert dataset for n+1 classes instead n (default value is True).
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
  • cvat_attributes_recognition - converts CVAT XML annotation version 1.1 format for images to ClassificationAnnotation or ContainerAnnotation with ClassificationAnnotation as value type and attribute names as keys (in multiple attributes case). Used bbox attributes as annotation classes.
    • annotation_file - path to xml file in appropriate format.
    • label - the dataset label which will be used for attributes collection (e.g. if your dataset contains 2 labels: face and person and you want recognise attributes for face, you should use face as value for this parameter).
  • cvat_age_gender - converts CVAT XML annotation version 1.1 format for images which represent dataset for age gender recognition to ContainerAnnotation with ClassificationAnnotation for gender recognition, ClassificationAnnotation for age classification and RegeressionAnnotation for age regression. The identifiers for representations following: gender_annotation, age_class_annotation, age_regression_annotation.
    • annotation_file - path to xml file in appropriate format.
  • cvat_facial_landmarks - converts CVAT XML annotation version 1.1 format for images to FacialLandmarksAnnotation.
    • annotation_file - path to xml file in appropriate format.
  • cvat_pose_estimation - converts CVAT XML annotation version 1.1 format for images to PoseEstimationAnnotation.
    • annotation_file - path to xml file in appropriate format.
  • cvat_text_recognition - converts CVAT XML annotation version 1.1 format for images to CharacterRecognitionAnnotation.
    • annotation_file - path to xml file in appropriate format.
  • cvat_binary_multilabel_attributes_recognition - converts CVAT XML annotation version 1.1 format for images to MultiLabelRecognitionAnnotation. Used bbox attributes as annotation classes. Each attribute field should contains T or F values for attribute existence/non-existence on the image respectively.
    • annotation_file - path to xml file in appropriate format.
    • label - the dataset label which will be used for attributes collection (e.g. if your dataset contains 2 labels: face and person and you want recognise attributes for face, you should use face as value for this parameter).
  • cvat_person_detection_action_recognition converts dataset with CVAT XML annotation version 1.1 for person detection and action recognition task to ContainerAnnotation with DetectionAnnotation for person detection quality estimation named person_annotation and ActionDetectionAnnotation for action recognition named action_annotation.
    • annotation_file - path to xml file with ground truth.
    • use_case - use case, which determines the dataset label map. Supported range actions:
      • common_3_actions(seating, standing, raising hand)
      • common_6_actions(seating, writing, raising hand, standing, turned around, lie on the desk)
      • teacher (standing, writing, demonstrating)
      • raising_hand (seating, raising hand)
  • lpr_txt - converts annotation for license plate recognition task in txt format to CharacterRecognitionAnnotation.
    • annotation_file - path to txt annotation.
    • decoding_dictionary - path to file containing dictionary for output decoding.
  • squad_emb - converts the Stanford Question Answering Dataset (SQuAD) to Question Answering Embedding Annotation. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BERT.
    • testing_file - path to testing file.
    • vocab_file - path to model co vocabulary file.
    • max_seq_length - maximum total input sequence length after word-piece tokenization (Optional, default value is 128).
    • max_query_length - maximum number of tokens for the question (Optional, default value is 64).
    • lower_case - allows switching tokens to lower case register. It is useful for working with uncased models (Optional, default value is False)
  • squad - converts the Stanford Question Answering Dataset (SQuAD) to Question Answering Annotation. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BERT.
    • testing_file - path to testing file.
    • vocab_file - path to model co vocabulary file.
    • max_seq_length - maximum total input sequence length after word-piece tokenization (Optional, default value is 128).
    • max_query_length - maximum number of tokens for the question (Optional, default value is 64).
    • doc_stride -stride size between chunks for splitting up long document (Optional, default value is 128).
    • lower_case - allows switching tokens to lower case register. It is useful for working with uncased models (Optional, default value is False)
  • squad_bidaf - converts the Stanford Question Answering Dataset (SQuAD) to QuestionAnsweringBiDAFAnnotation. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BiDAF using nltk.word_tokenize.
    • testing_file - path to testing file.
  • xnli - converts The Cross-lingual Natural Language Inference Corpus (XNLI) to TextClassificationAnnotattion. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BERT.
    • annotation_file - path to dataset annotation file in tsv format.
    • vocab_file - path to model vocabulary file for WordPiece tokinezation (Optional in case, when another tokenization approach used).
    • sentence_piece_model_file - model used for SentencePiece tokenization (Optional in case, when another tokenization approach used).
    • max_seq_length - maximum total input sequence length after word-piece tokenization (Optional, default value is 128).
    • lower_case - allows switching tokens to lower case register. It is useful for working with uncased models (Optional, default value is False).
    • language_filter - comma-separated list of used in annotation language tags for selecting records for specific languages only. (Optional, if not used full annotation will be converted).
  • mnli - converts The Multi-Genre Natural Language Inference Corpus (MNLI) to TextClassificationAnnotattion. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BERT.
    • annotation_file - path to dataset annotation file in tsv format.
    • vocab_file - path to model vocabulary file for WordPiece tokinezation. (Optional, can be not provided in case, when another tokenization approach used.)
    • sentence_piece_model_file - model used for SentencePiece tokenization (Optional in case, when another tokenization approach used).
    • max_seq_length - maximum total input sequence length after tokenization (Optional, default value is 128).
    • lower_case - allows switching tokens to lower case register. It is useful for working with uncased models (Optional, default value is False).
  • mrpc - converts The Microsoft Research Paraphrase Corpus (MRPC) to TextClassificationAnnotattion. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BERT.
    • annotation_file - path to dataset annotation file in tsv format.
    • vocab_file - path to model vocabulary file for WordPiece tokenization. (Optional, can be not provided in case, when another tokenization approach used.)
    • sentence_piece_model_file - model used for SentencePiece tokenization (Optional in case, when another tokenization approach used).
    • max_seq_length - maximum total input sequence length after tokenization (Optional, default value is 128).
    • lower_case - allows switching tokens to lower case register. It is useful for working with uncased models (Optional, default value is False).
  • cola - converts The Corpus of Linguistic Acceptability (CoLA) to TextClassificationAnnotattion. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BERT.
    • annotation_file - path to dataset annotation file in tsv format.
    • vocab_file - path to model vocabulary file for WordPiece tokinezation. (Optional, can be not provided in case, when another tokenization approach used.)
    • sentence_piece_model_file - model used for SentencePiece tokenization (Optional in case, when another tokenization approach used).
    • max_seq_length - maximum total input sequence length after tokenization (Optional, default value is 128).
    • lower_case - allows switching tokens to lower case register. It is useful for working with uncased models (Optional, default value is False).
  • cola - converts The Corpus of Linguistic Acceptability (CoLA) to TextClassificationAnnotattion. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BERT.
    • annotation_file - path to dataset annotation file in tsv format.
    • vocab_file - path to model vocabulary file for WordPiece tokenization. (Optional, can be not provided in case, when another tokenization approach used.)
    • sentence_piece_model_file - model used for SentencePiece tokenization (Optional in case, when another tokenization approach used).
    • max_seq_length - maximum total input sequence length after tokenization (Optional, default value is 128).
    • lower_case - allows switching tokens to lower case register. It is useful for working with uncased models (Optional, default value is False).
  • imdb - converts IMDB sentiment dataset to TextClassificationAnnotattion. Note: This converter not only converts data to metric specific format but also tokenize and encodes input for BERT.
    • annotation_file - path to dataset annotation file in tsv format.
    • vocab_file - path to model vocabulary file for WordPiece tokinezation. (Optional, can be not provided in case, when another tokenization approach used.)
    • sentence_piece_model_file - model used for SentencePiece tokenization (Optional in case, when another tokenization approach used).
    • max_seq_length - maximum total input sequence length after tokenization (Optional, default value is 128).
    • lower_case - allows switching tokens to lower case register. It is useful for working with uncased models (Optional, default value is False).
  • bert_xnli_tf_record - converts The Cross-lingual Natural Language Inference Corpus (XNLI) stored in tf records format. This converter usage requires TensorFlow installation. Please make sure that TensorFlow installed before conversion.
    • annotattion_file - path to annotation file in tf records format.
  • cmu_panoptic_keypoints - converts CMU Panoptic dataset to PoseEstimation3dAnnotation format.
    • data_dir - dataset root directory, which contain subdirectories with validation scenes data.
  • clip_action_recognition - converts annotation video-based action recognition datasets. Before conversion validation set should be preprocessed using approach described here.
    • annotation_file - path to annotation file in json format.
    • data_dir - path to directory with prepared data (e. g. data/kinetics/frames_data).
    • clips_per_video - number of clips per video (Optional, default 3).
    • clip_duration - clip duration (Optional, default 16)
    • temporal_stride - temporal stride for frames selection (Optional, default 2).
    • numpy_input - allows usage numpy files instead images. It can be useful if data required difficult preprocessing steps (e.g. conversion to optical flow) (Optional, default False)
    • subset - dataset split: train, validation or test (Optional, default validation).
    • dataset_meta_file - path path to json file with dataset meta (e.g. label_map, color_encoding).Optional, more details in Customizing dataset meta section.
    • num_samples - select first n samples from dataset (Optional, if not provided full subset of samples will be used).
  • continuous_clip_action_recognition - converts annotation of video-based MS-ASL dataset to ClassificationAnnotation.
    • annotation_file - path to annotation file in txt format.
    • data_dir - dataset root directory, which contains subdirectories with extracted video frames.
    • out_fps - output frame rate of generated video clips.
    • clip_length - number of frames of generated video clips.
  • redweb - converts ReDWeb dataset for monocular relative depth perception to DepthEstimationAnnotation
    • data_dir - the dataset root directory, where imgs - directory with RGB images and RD - directory with relative depth maps are located (Optional, if you want to provide annotation_file)
    • annotation_file- the file in txt format which contains pairs of image and depth map files. (Optional, if not provided full content of data_dir will be considered as dataset.)
  • nyu_depth_v2 - converts NYU Depth Dataset V2 for depth estimation to DepthEstimationAnnotation. This converter accept preprocessed data stored in HDF5 format, which can be downloaded from this page
    • data_dir - directory with HDF5 files. (Optional, can be omitted if you already have converted images and depth maps).
    • images_dir - directory for images. If data_dir provided, the directory will be used for saving converted images, otherwise used for data reading. (Optional, can be not provided in conversion case, default value <data_dir>/converted/images).
    • depth_map_dir - directory for reference depth maps, stored in numpy format. If data_dir provided, the directory will be used for saving converted depth maps, otherwise used for data reading. (Optional, can be not provided in conversion case, default value <data_dir>/converted/depth). Please, note, you need to specify path to directory with depth maps with additional_data_source parameter in your config during evaluation.
  • inpainting - converts images to ImageInpaintingAnnotation.
    • images_dir - path to images directory.
    • masks_dir - path to mask dataset to be used for inpainting (Optional).
  • aflw2000_3d - converts AFLW2000-3D dataset for 3d facial landmarks regression task to FacialLandmarks3DAnnotation.
    • data_dir - directory, where input images and annotation files in MATLAB format stored.
  • style_transfer - converts images to StyleTransferAnnotation.
    • images_dir - path to images directory.
  • ade20k - converts ADE20K dataset to SegmentationAnnotation.
    • images_dir - path to directory with images (e.g. ADEChallengeData2016/images/validation).
    • annotations_dir - path to directory with annotations (e.g. ADEChallengeData2016/annotations/validation).
    • object_categories_file - path to file with labels (e.g. ADEChallengeData2016/objectInfo150.txt).
    • num_classes - number of used classes.
  • criteo_kaggle_dac - converts Criteo datasets to ClassificationAnnotation.
    • testing_file - path to preprocessed Criteo file (e.g. criteo/terabyte/terabyte_preprocessed,npz).
    • batch - batch size expected by model
    • subsample_size - number of batches in test-only dataset, If provided, total number of records is batch * subsample_size
    • validation - if provided, only second half of dataset converted to annotations, according to dataset definition
    • preprocessed_dir - path to store preprocessed batch files (e.g. criteo/terabyte/preprocessed).
    • separator - symbol used to separate feature identifiers from batch data filename.
  • features_regression - converts dataset stored in format of directories with preprocessed input numeric data (features) in text files and reference data in the same format to FeatureRegressionAnnotation. This approach allows comparing output of model from different frameworks (e.g. OpenVINO converted model and source framework realisation).
    • input_dir - directory with input data files.
    • reference_dir - directory with reference data. Note: inside converted annotation, path to directory is not stored, only file name, please use additional_data_source for providing prefix.
    • input_suffix - suffix for input files (usually file extension). Optional, default .txt.
    • reference_suffix - suffix for reference files (usually file extension). Optional, default .txt.
  • multi_feature_regression - converts dataset stored in format of directories with preprocessed input numeric data (features) in dictionary format, where keys are layer names and values - features and reference data in the same format to FeatureRegressionAnnotation. This approach allows comparing output of model from different frameworks (e.g. OpenVINO converted model and source framework realisation). Please note, that input and reference should be stored as dict-like objects in npy files.
    • data_dir - directory with input and reference files.
    • input_suffix - suffix for input files (usually file extension). Optional, default in.npy.
    • reference_suffix - suffix for reference files (usually file extension). Optional, default out.npy.
    • prefix - prefix for input files selection (Optional, ignored if not provided).
  • librispeech - converts librispeech dataset to CharachterRecognitionAnnotation.
    • data_dir - path to dataset directory, which contains converted wav files.
    • annotation_file - path to file which describe the data which should be used in evaluation (audio_filepath, text, duration). Optional, used only for data filtering and sorting audio samples by duration.
    • use_numpy - allows to use preprocessed data stored in npy-files instead of audio (Optional, default False).
    • top_n - numeric value for getting only the n shortest samples Note: applicable only with annotation_file providing.
  • criteo - converts Criteo datasets to ClassificationAnnotation.
    • testing_file - Path to testing file, terabyte_preprocessed.npz (Criteo Terabyte) or day_6_processed.npz (Criteo Kaggle Dac)
    • batch - Model batch.
    • subsample_size - Subsample size in batches
    • validation - Allows to use half of dataset for validation purposes
    • block - Make batch-oriented annotations
    • separator - Separator between input identifier and file identifier
    • preprocessed_dir - Preprocessed dataset location
    • dense_features - Name of model dense features input
    • sparse_features - Name of model sparse features input. For multiple inputs use comma-separated list in form :
    • lso_features - Name of lS_o-like features input
  • im2latex - converts im2latex-like datasets to CharacterRecognitionAnnotation. Example of the dataset
    • images_dir - path to input images (rendered or scanned formulas)
    • formula_file - path to file containing one formula per line
    • split_file - path to file containing img_name and corresponding formula index in formula_file separated by tab per line
    • vocab_file - file containing vocabulary to cast token class indices into human-readable tokens
  • dna_sequence - converts dataset for DNA sequencing to DNASequenceAnnotation.
    • chunks_file - npy file with input chunks.
    • ref_file - npy file with reference sequence.
    • num_chunks - subset size for usage in validation, if not provided the whole dataset will be used.
    • alphabet - alphabet for sequence decoding (Optional, default ["N", "A", "C", "G", "T"]).
  • place_recognition - converts dataset for image based localization task to PlaceRecognitionAnnotation
    • subset_file - matlab file contains info about subset used in validation.
  • mpii - converts MPII Human Pose Estimation dataset to PoseEstimationAnnotation.
    • annotation_file - json-file with annotation.
    • headboxes_file - file with boxes contained head coordinates for each image.
  • cluttered_mnist - converts MNIST dataset from spatial transformer network example to ClassificationAnnotation.
    • data_file - npz file with dataset.
    • split - dataset split: train - for training subset, valid - for train-validation subset, test - for testing subset (Optional, default test).
    • convert_images - allows convert images from raw data stored in npz and save them into provided directory (Optional, default True).
    • images_dir - directory for saving converted images (Optional, if not provided, the images will be saved into converted_images directory in the same location, where data_file is stored)
  • antispoofing - converts dataset for antispoofing classification task to ClassificationAnnotation
    • data_dir - path to root folder of the dataset
    • annotation_file - path to json file containing annotations to the dataset ({index: {path:"...", labels:[...], bbox:[...] (optional), ...})
    • label_id - number of label in the annotation file representing spoof/real labels
    • dataset_meta_file - path to json file with dataset meta (e.g. label_map)
  • sound_classification - converts dataset for sound classification to ClassificationAnnotation. The dataset should be represented by directory with input wav files and annotation in 2 column csv format, where first column is audio file name and second is label id from dataset.
    • annotation_file - csv file with selected subset for evaluation, file structure described above.
    • audio_dir - directory with input data, (optional, required only if you want check file existence during annotation conversion).
  • ade20k_image_translation - converts ADE20K dataset to ImageProcessingAnnotation according to reference_file.
    • annotations_dir - path to directory with annotations (e.g. ADEChallengeData2016/annotations).
    • reference_file - path to file with pairs key (validation): value (train).
  • salient_object_detection - converts dataset for salient object detection to SalientRegionAnnotation. The dataset should have following structure:
    1. images have numeric ids like names and jpg extension (e.g. image/0.jpg, image/1.jpg, image/2.jpg, ...).
    2. salience map located in separated directory, have the same ids like images and png extension (e.g. mask/0.png, mask/1.png, mask/2.png).
    • images_dir - directory with input images.
    • masks_dir - directory with reference salience maps.
    • annotation_file - txt file with selected image ids.
  • wflw - converts WFLW dataset for facial landmarks regression task to FacialLandmarksAnnotation.
    • annotation_file - path to txt file with ground truth data in WFLW dataset format.
    • images_dir - path to dataset images, used only for content existence check (optional parameter).
  • common_object_detection - converts object detection dataset to DetectionAnnotation. Dataset should be stored in following format:
    1. labels_map defined as text file, where defined labels line by line.
    2. annotations for each image stored in separated text file. Box is represented by space separated info: <label_id> <x_min> <y_min> <x_max> <y_max>.
    3. name of annotation file the same like image name (or additional file with file mapping should be defined).
    • annotation_dir - path to directory with annotation files.
    • images_dir - path to directory with images (Optional, used only for content check step).
    • labels_file - path to file with labels.
    • pairs_file - path to file where described image and annotation file pairs (Optional, if not provided list will be created according to annotation_dir content).
    • has_background - flag that background label should be added to label_map (Optional, default False).
    • add_background_to_label_id - flag that label_ids defined in annotation should be shifted if has_background enabled.

Customizing Dataset Meta

There are situations when we need customize some default dataset parameters (e.g. replace original dataset label map with own.) You are able to overload parameters such as label_map, segmentation_colors, background_label using dataset_meta_file argument. dataset meta file is JSON file, which can contains following parameters:

  • label_map is dictionary where <CLASS_ID> is key and <CLASS_NAME> - value.
  • labels is the list of strings, which represent class names (order is matter, the index of class name used as class id). Can be used instead label_map.
  • background_label - id of background label in the dataset.
  • segmentation_colors (if your dataset for semantic segmentation task uses color encoding). Segmentation colors is a list of channel-wise values for each class. (e.g. if your dataset has 3 classes in BGR colors, segmentation colors for it will looks like: [[255, 0, 0], [0, 255, 0], [0, 0, 255]]). Example of dataset_meta.json content:
{
"label_map": {"0": "background", "1": "cat", "2": "dog"},
"background_label": "0",
"segmentation_colors": [[0, 0, 0], [255, 0, 0], [0, 0, 255]]
}