DLC Inference Speed Benchmark

This repository stores the benchmark results for DeepLabCut-live for each standard dataset, grouped by operating system, processor, and DLC model. Each configuration is tested on a fixed set of videos.

How to contribute!

1. Install the DeepLabCut-live! SDK
2. git clone the DeepLabCut-live! repo: git clone https://github.com/DeepLabCut/DLC-inferencespeed-benchmark.git and run ./reinstall.sh to be sure it's properly installed.
3. Run our benchmarking script on your system (with our data/model). Within the DeepLabCut-Live directory you will find the following structure:

DeepLabCut-Live
   -Benchmarking
   --> run_dlclive_benchmark.py

Then you can run (with python3, pythonw on MacOS):

python run_dlclive_benchmark.py

This will take some time, depending on your internet connection and hardware. Note that downloading, might take a few minutes, as the multiple models & videos comprise about 2,2 GB. Then 4 models will be run on two videos for various video sizes. To get you a sense, this takes about 90 minutes on a Titan RTX. IF you want to run the benchmark on a CPU or slow hardware, you can also change the number of frames, to 1000 in https://github.com/DeepLabCut/DeepLabCut-live/blob/master/benchmarking/run_dlclive_benchmark.py#L24.

3. Please make a pull request here (i.e., add the resulting file to your forked repo under the _data folder--i.e., no need to hand edit the file, we will automatically convert your files into the correct yaml file format), and create a new pull request!) or email us: [email protected] if you have any trouble!