Utility package which helps to ease the manipulation of tfrecords.
tfrmaker helps to ease the manipulation of tfrecords for your next machine learning project with tensorflow. You can now easily create, extract and load image datasets in the form of tfrecords with help of tfrmaker. With the help of the package, large image datasets can be converted into tfrecords and fed directly into tensorflow models for training and testing purposes. Some key feature of the package includes:
- dynamic resizing
- splitting tfrecords into optimal shards
- spliting training, validation, testing of tfrecords
- count no of images in tfrecords
- asynchronous tfrecord creation
TFRecords stores data as a sequence of binary records with help of protocol buffers, a cross-platform, cross-language library. It has many advantages like:
- Efficient storage: TFRecord data can take up less space than the original data; it can also be partitioned into multiple files.
- Fast I/O: TFRecord format can be read with parallel I/O operations, which is useful for TPUs or multiple hosts.
Use the package manager pip to install tfrmaker.
pip install tfrmakerA minimal usage of tfrmaker with image data, organized as directores with name as class labels:
from tfrmaker import images, display
# mapping label names with integer encoding.
LABELS = {"bishop": 0, "knight": 1, "pawn": 2, "queen": 3, "rook": 4}
# specifiying data and output directories.
DATA_DIR = "datasets/chess/"
OUTPUT_DIR = "tfrecords/chess/"
# create tfrecords from the images present in the given data directory.
images.create(DATA_DIR, LABELS, OUTPUT_DIR)
# load one or more tfrecords as an iterator object.
dataset = images.load(["tfrecords/chess/queen.tfrecord","tfrecords/chess/bishop.tfrecords"], batch_size=32, repeat=True)
# iterate one batch and visualize it along with labels.
databatch = next(iter(dataset))
display.batch(databatch, LABELS)Refer examples folder for more advanced usage.
"Your mental support by staring the repo is much appreciated."
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.