What is SpSeg?
SpSeg models
Training data
How to Use SpSeg?
Setting up the tools
Camera-Trap image data organization
Running SpSeg
How to use the output to process the images?
Further development
Citing SpSeg and License information
'SpSeg' (short for 'Species Segregator) is a Machine-learning tool for species-level segregation of camera-trap images originating from wildlife census and studies. SpSeg is currently trained specifically for the Central Indian landscape .
SpSeg is part of Microsoft's MegaDetector ecosystem and works as a second-step to species level segregation of camera-trap images. The approach in MegaDetector pipeline is to first classify the images into Animal, Person, Vehicle and Blank, followed by a fine level classification of Animal images into useful taxonomic classes.
📝 Check out this detailed overview of the approach in a Medium article and an explainer by Dan Morris on why this two-step process.
SpSeg model identifies an Indian fox within the bounding box.
We initially trained different CNN architecture models from keras (see the results below). We found two models performing well - SpSeg_x.hdf5 and SpSeg_r152v2.hdf5 - and have tested those on various field datasets. SpSeg_x.hdf5 is an Xception model that achieved 88.9% test accuracy, and SpSeg_r152v2.hdf5 is a ResNet152v2 model that achieved 89.2% test accuracy. Find the models from a publicly shared SpSeg_Models Google Drive folder.
The models in different architectures were trained for 100 ephocs each with the same training and test dataset.
| Architecture | avg top-1 acc | Architecture | avg top-1 acc |
|---|---|---|---|
| Xception | 88.9% | InceptionResNetv2 | 89.2% |
| ResNet50 | 88.5% | ResNet50v2 | 87.5% |
| ResNet101 | 88.8% | ResNet101v2 | 89.1% |
| ResNet152 | 82.0% | ResNet152v2 | 89.2% |
SpSeg models can currently segregate 36 commonly encountered species (or taxonomic class) in camera-trap surveys in the Eastern Vidarbha Landscape, Maharashtra, India. A maximum of 5000 images were randomly selected for each species from our image dataset to train the models. The models were trained with 70% of the selected data, while 15% images were used for validation and remaining 15% were used for testing. Therefore, species with more than 5000 images in our image dataset (table below) have better accuracy, except sexually dimorphic species like Nilgai.
| Species | Scientific name | Image set | Species | Scientific name | Image set |
|---|---|---|---|---|---|
| 00_barking_deer | Muntiacus muntjak | 7920 | 18_langur | Semnopithecus entellus | 12913 |
| 01_birds | Excluding fowls | 2005 | 19_leopard | Panthera pardus | 7449 |
| 02_buffalo | Bubalus bubalis | 7265 | 20_rhesus_macaque | Macaca mulatta | 5086 |
| 03_spotted_deer | Axis axis | 45790 | 21_nilgai | Boselaphus tragocamelus | 6864 |
| 04_four_horned_antelope | Tetracerus quadricornis | 6383 | 22_palm_squirrel | Funambulus palmarum & Funambulus pennantii | 1854 |
| 05_common_palm_civet | Paradoxurus hermaphroditus | 8571 | 23_indian_peafowl | Pavo cristatus | 10534 |
| 06_cow | Bos taurus | 7031 | 24_ratel | Mellivora capensis | 5751 |
| 07_dog | Canis lupus familiaris | 4150 | 25_rodents | Several mouse, rat, gerbil and vole species | 4992 |
| 08_gaur | Bos gaurus | 14646 | 26_mongooses | Urva edwardsii & Urva smithii | 5716 |
| 09_goat | Capra hircus | 3959 | 27_rusty_spotted_cat | Prionailurus rubiginosus | 1649 |
| 10_golden_jackal | Canis aureus | 2189 | 28_sambar | Rusa unicolor | 28040 |
| 11_hare | Lepus nigricollis | 8403 | 29_domestic_sheep | Ovis aries | 2891 |
| 12_striped_hyena | Hyaena hyaena | 2303 | 30_sloth_bear | Melursus ursinus | 6348 |
| 13_indian_fox | Vulpes bengalensis | 379 | 31_small_indian_civet | Viverricula indica | 4187 |
| 14_indian_pangolin | Manis crassicaudata | 1442 | 32_tiger | Panthera tigris | 9111 |
| 15_indian_porcupine | Hystrix indica | 5090 | 33_wild_boar | Sus scrofa | 18871 |
| 16_jungle_cat | Felis chaus | 4376 | 34_wild_dog | Cuon alpinus | 7743 |
| 17_jungle_fowls | Includes Gallus gallus, Gallus sonneratii & Galloperdix spadicea | 4760 | 35_indian_wolf | Canis lupus pallipes | 553 |
The best place to start is the User Guide of SpSeg, which covers the process in details.
SpSeg model (either SpSeg_x.hdf5 or SpSeg_r152v2.hdf5) reads the results of MegaDetector and classifies the animal images into species (or a defined biological taxonomic level). Therefore, both the models (detector and classifier) are required to run simultaneously on the images.
📝 You can find more about how you can run MegaDetector on your images from this official MegaDetecor GitHub page.
These instructions are quite similar to the instruction for MegaDetector installation, where more details could be found. For technical details check the official websites of the mentioned software.
Download and install Anaconda (or Miniconda). Installing Git is optional, but recommended to keep your repo updated. The latest NVIDIA drivers need to be checked if you are using a GPU.
Download MegaDetector model v4.1.0 and SpSeg models and keep at an accessible location. These instructions assume that the models are downloaded to a folder called c:\spseg_models.
There is no need to setup a separate MegaDetector environment, which is incorporated in the codes here. We are giving the instructions for Windows machines only, which is expected to work on Linux machines in a similar way. The environment is not tested on Mac.
Open Anaconda Prompt and make sure you have not opened an Anaconda Powershell Prompt or Windows Command Prompt. It should look something like this with the name of environment in the parenthesis:
Run the following commands one by one to download SpSeg repo from GitHub:
mkdir c:\git
cd c:\git
git clone https://github.com/bhlab/SpSegThe commnads above are suggestions. git command will work at any directory and you can download GitHub repo wherever you find it convinient. Alternatively, you can download the SpSeg repo as zip folder and unzip it to a desired location.
environment_SpSeg.yml file in notepad, change the dependency tensorflow-gpu to tensorflow and save the file.
Next, you need to enter where SpSeg repo is located using cd command:
cd c:\git\spseg
conda env create --file environment_SpSeg.ymlYou are all set to run SpSeg! 🎉
We assume that your camera-trap data was organized in either of the following directory structures:
dataDir/Station
dataDir/Station/Camera
The first structure arrives when a single camera was placed at the station, while the second structure arrives when A and B cameras were placed facing each other. The directory before Station ID (dataDir) could include site ID, range ID and/or block ID.
Next time when you open Anaconda Prompt, first you need to activate SpSeg environment:
conda activate SpSegand then locate the SpSeg tools folder.
cd c:\git\spseg\toolsWhich should look something like this:
The main tool to use SpSeg models is run_spseg.py
Once you are in the right setup (environment and tools folder), the following command calls detector and classifier models and runs over all the images:
python run_spseg.py "C:\spseg_models\md_v4.1.0.pb" "D:\dataDir" --recursive --spseg_model " C:\spseg_models\SpSeg_r152v2.hdf5" "D:\dataDir\data_output.json""D:\dataDir" is the location (local or on network server) where your organized camera-trap images are stored. You can set the path and name of the output file at "D:\dataDir\data_output.json".
The above command returns a .json directory for each image with detector tags for animal, person or vehicle, and species tags for animals (both the tags with a confident value).
💡 The image paths are retuned as absolute, which creates mismatches during further processing of the data specially if processing is carried out on a different system. Therefore, the best practice is to get the image paths as relative and keep the output file within image folder.
Relative paths in the output can be returned by setting the flag for it:
python run_spseg.py "C:\spseg_models\md_v4.1.0.pb" "C:\dataDir" --recursive --spseg_model " C:\spseg_models\SpSeg_ r152v2.hdf5" "C:\dataDir\data_output.json" -- output_relative_filenamesAI-tools are not 100% accurate. Therefore, a workflow is required to process the output and correct for the errors. Currently, there are two approaches to process the output and finalize the identification/segregation of the images for further analyses-
- Using Timelapse to review the images (only for Microsoft Windows systems),
- Using an Image file manager system like Windows File Explorer, ExifPro, IrfanView or FastStone.
Detailed instructions on these workflows are provided in the User Guide of SpSeg.
We have in pipeline- 1. integration with MegaDetector version 5. 2. developement of DigiKam based workflow, 3. Further refinement of the SpSeg models (particularlly for dimorphic ungulate species and species with small dataset), and 4. Addition of more species, e.g. we missed out on the elephant in the current models.
So stay tuned!
Also, if you have any suggestion for additional species or would like to add to training dataset, reach out to us ([email protected]) to Participate!
Cite the use of SpSeg models as:
Shrotriya, Shivam; Guthula, Venkanna Babu; Mondal, Indranil; and Habib, Bilal (2022). SpSeg User's Guide, version 1.0. TR No. 2022/29. Wildlife Institute of India, Dehradun, India.
And, don't forget to cite MegaDetector (without which SpSeg won't even exist):
Beery, Sara; Morris, Dan; and Yang, Siyu (2019). Efficient Pipeline for Camera Trap Image Review. arXiv preprint arXiv:1907.06772
SpSeg models and tools are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license. Find details on the terms of the license here.
Dr Bilal Habib's lab (BHlab) at Wildlife Institute India, Dehradun, India is a collaborator in the development, evaluation and use of MegaDetector model. Development of SpSeg was supported by Microsoft AI for Earth (Grant ID:00138001338). We thank Mahrastra Forest Department for funding long term research in the state of Maharashtra. Kathan Bandopadhyay helped in preparing training dataset. The researchers and volunteers are thanked for their time for reviewing and testing the tools in real workfow.