Mmdet TensorRT support

[enesozi]

• TensorRT model wrapper for both instance seg. and object detector added to mmdet.py

GPU Info: NVIDIA GeForce RTX 3090

Model used: rtmdet_l_8xb32-300e_coco.py

Deployment cfg file used: detection_tensorrt-fp16_static-640x640.py -> you may find examples here

Average inference times(Just the _call_ method of the wrappers used):

• Vanilla model ~ 28 ms
• TensorRT model ~ 14 ms

Average inference times(get_sliced_prediction function):

• image used with a size of (height=1432, width=4089, 3) and slice used with a size of (height=640,width=640)
• Vanilla model ~ 1.4 s
• TensorRT model ~ 1 s

The difference was around 400 ms but it might yield significant speed improvement for edge devices such as jetson.

Disclaimer: This implementation is not perfect as it could be generalized to work for other frameworks(yolo, detectron2, etc.). I needed this for my current project and wanted to share it as a base for anyone interested.

Example usage:

deploy_config_path = None
category_mapping = None
path_detector = "detection.pth"
if trt:
  path_detector = "end2end.engine"
  deploy_config_path = "detection_tensorrt-fp16_static-640x640.py"
  category_mapping = {id:class} #Class mapping needed

model = AutoDetectionModel.from_pretrained(
            model_type="mmdet",
            model_path=path_detector,
            deploy_config_path=deploy_config_path,
            config_path="rtmdet_l_8xb32-300e_coco.py",
            category_mapping=category_mapping,
            device=device,
        )

[fcakyon]

@enesozi, amazing contribution! Can you please fix the tests?

[enesozi]

@fcakyon mmdeploy might be annoying when it comes to installing it. I hope the latest changes fix it.

[enesozi]

Run mim install mmdeploy==1.3.1 mim install mmdeploy-runtime==1.3.1
Looking in links: https://download.openmmlab.com/mmcv/dist/cpu/torch1.13.0/index.html
Collecting mmdeploy==1.3.1
Downloading mmdeploy-1.3.1-py3-none-manylinux2014_x86_64.whl.metadata (19 kB)
Collecting mim
Downloading mim-0.2.43.tar.gz (38 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
error: subprocess-exited-with-error

Weirdly, the workflow is trying to run the command mim install mmdeploy==1.3.1 twice. So, the command looks like this mim install mmdeploy==1.3.1 mim - another library called mim.

[enesozi]

I checked the mmdet_tests locally and confirmed they work. CI pipeline now fails at huggingface tests.

ERROR: test_perform_inference (tests.test_huggingfacemodel.TestHuggingfaceDetectionModel)

Traceback (most recent call last):
File "/home/runner/work/sahi/sahi/tests/test_huggingfacemodel.py", line 75, in test_perform_inference
scores, cat_ids, boxes = huggingface_detection_model.get_valid_predictions(
File "/home/runner/work/sahi/sahi/sahi/models/huggingface.py", line 135, in get_valid_predictions
valid_confidences = torch.where(scores >= self.confidence_threshold, 1, 0)
TypeError: '>=' not supported between instances of 'Tensor' and 'NoneType'