heat.utils.data._utils

This file contains functions which may be useful for certain datatypes, but are not test in the heat framework This file contains standalone utilities for data preparation which may be useful The functions contained within are not tested, nor actively supported

Module Contents

dali_tfrecord2idx(train_dir, train_idx_dir, val_dir, val_idx_dir)

WARNING: This function likely requires adjustments and it is by no means a final product !!! this file contains standalone utilities for data preparation which may be useful this function contained within are not tested, nor actively supported

prepare TFRecords indexes for use with DALI. It will produce indexes for all files in the given train_dir and val_dir directories

merge_files_imagenet_tfrecord(folder_name, output_folder=None)

WARNING: This function likely requires adjustments and it is by no means a final product !!! this file contains standalone utilities for data preparation which may be useful this function contained within are not tested, nor actively supported

merge multiple preprocessed imagenet TFRecord files together, result is one HDF5 file with all of the images stacked in the 0th dimension

Parameters:

  • folder_name (str, optional*) – folder location of the files to join, either filenames or folder_names must not be None

  • output_folder (str, optional) – location to create the output files. Defaults to current directory

Notes

Metadata for both the created files (imagenet_merged.h5 and imagenet_merged_validation.h5):

The datasets are the combination of all of the images in the Image-net 2012 dataset. The data is split into training and validation.

imagenet_merged.h5 -> training imagenet_merged_validation.h5 -> validation

both files have the same internal structure: - file

  • “images”encoded ASCII string of the decoded RGB JPEG image.

    • to decode: torch.as_tensor(bytearray(base64.binascii.a2b_base64(string_repr.encode(‘ascii’))), dtype=torch.uint8)

    • note: the images must be reshaped using: .reshape(file[“metadata”][“image/height”], file[“metadata”][“image/height”], 3)

      (3 is the number of channels, all images are RGB)

  • “metadata”the metadata for each image quotes are the titles for each column

    1. “image/height”

    2. “image/width”

    3. “image/channels”

    4. “image/class/label”

    5. “image/object/bbox/xmin”

    6. “image/object/bbox/xmax”

    7. “image/object/bbox/ymin”

    8. “image/object/bbox/ymax”

    9. “image/object/bbox/label”

  • “file_info”string information related to each image

    1. “image/format”

    2. “image/filename”

    3. “image/class/synset”

    4. “image/class/text”

The dataset was created using the preprocessed data from the script:

https://github.com/tensorflow/models/blob/master/research/inception/inception/data/download_and_preprocess_imagenet.sh