Bounding Box Camera

[AmrElsersy]

🎉 New feature

related to #135

Summary

Bounding boxes camera to extract labeled item's bounding boxes from the scene using OGRE 2.1

Visible Bounding Boxes

• Supports visible bounding boxes which shows a box on the visible part of the item (box dosen't include the occluded part of the object) (datasets ex: COCO & PASCAL VOC datasets)
• Idea: Rendering a map which contains ogre Ids of objects, and by looping on its pixels to get the the objects boundaries

Full Bounding Boxes

• Supports full bbox which shows all the object boundaries even if occluded (datasets ex: KITTI, nuscenes)
• Idea: accessing all item's vertices and projecting them to get the boundaries of the object, and check visibility by rendering a map which contains ogre Ids of objects

Features:

• The user label each item by setting its UserData
• Each unlabeled item is considered as a background (not finished yet)
• Added example in ign-rendering/examples to try it easily
• User can change the mode (Visible / Full) through the rendering API

[adlarkin]

@AmrElsersy here are some ideas for integration testing:

1. Generate a scene with 2 objects: one that is completely to the left of another. Make sure that you end up with 2 bounding boxes, and that all of the coordinates for the left object's bounding box are actually to the left of or right object's bounding box (so, this would probably involve checking that all of the x coordinates for the left object are less than all of the x coordinates for the right object)
2. Test objects that overlap one another:
   1. two objects, with one object completely behind the other. We should only get one bounding box in this case, and the bounding box should be for the object that is not occluded
   2. two objects, with one object partially behind the other. We can then test the visible and full bounding boxes - the partially blocked object's bounding box for the visible bounding box sensor should be smaller than the partially blocked object's bounding box for the full bounding box sensor

Also, a note about checking the validity of bounding box coordinates: you can see if a bounding box's corner pixel coordinates are close enough to a "baseline" pixel coordinate by using EXPECT_NEAR

[chapulina]

The example looks super cool!

Can you comment on what's the intended use case? I see the sensor outputs the bounding box coordinates, as well as an RGB image with the boxes drawn. I'm wondering if we need the RGB image too, or if this could be a simpler sensor that just provides the BBs. That would make the sensor more similar to GpuRays than to a camera.

Another thing that came to mind is that this camera is providing the 2D bounding boxes, but 3D bounding boxes are also widely used in ML datasets. Are there plans to add 3D boxes too?

[AmrElsersy]

I am trying to add the 3D boxes feature but have a problem in the oreintation of the bbox
That is because the OGRE item has a Quaternion that specifies the rotation around the object axes.
and we have the view matrix which specifies the rotation of the camera

I want to add these 2 Quaternions
Does anyone know how to do that ?, I am trying to add the euler angles of both of them , but it is not working,

I think that is because the Item Quaternion is in World coord, so roll, pitch and yaw is the rotation around x, y and z ... but the view matrix Quaternion is in camera coord, so roll, pitch, yaw are rotations around different axes than the world coord

Any ideas on how to do that ?

This photo shows that the 3d position is correct but has a wrong rotation

[mjcarroll]

It sounds like you have two rotations:

1. The rotation from body to world q_b_w
2. The rotation from camera to world. q_c_w

You should be able to multiply these two quaternions to get the rotation from body to camera, but you may need to invert one first.

So q_c_b = q_c_w * inv(q_b_w)

[iche033]

by the way, the ViewMatrix() call does additional conversion from y to z up. Not sure if you need that. Otherwise, you can just call camera->WorldRotation() to get the camera rotation in world frame

[AmrElsersy]

It is working now, thanks :D @iche033 @mjcarroll

[codecov]

Codecov Report

Merging #334 (47b4456) into ign-rendering6 (e7c8b3a) will increase coverage by 0.44%.
The diff coverage is 64.10%.

@@                Coverage Diff                 @@
##           ign-rendering6     #334      +/-   ##
==================================================
+ Coverage           54.51%   54.96%   +0.44%     
==================================================
  Files                 198      202       +4     
  Lines               20270    21069     +799     
==================================================
+ Hits                11051    11581     +530     
- Misses               9219     9488     +269     

Impacted Files	Coverage Δ
include/ignition/rendering/BoundingBoxCamera.hh	100.00% <ø> (ø)
src/PixelFormat.cc	73.33% <ø> (ø)
ogre2/src/Ogre2BoundingBoxCamera.cc	59.67% <59.67%> (ø)
ogre2/src/Ogre2BoundingBoxMaterialSwitcher.cc	84.28% <84.28%> (ø)
src/BoundingBox.cc	92.98% <93.75%> (+61.73%)	⬆️
ogre2/src/Ogre2Scene.cc	76.85% <100.00%> (+0.14%)	⬆️
...e/ignition/rendering/base/BaseBoundingBoxCamera.hh	36.36% <0.00%> (ø)
src/base/BaseScene.cc	75.53% <0.00%> (+1.87%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7c8b3a...47b4456. Read the comment docs.

[chapulina]

This is working for me, except for the 2d_full box type. On the example, it crashes after I orbit with the mouse for a bit. It also happens on gz-sim. I'm not being able to get an informative backtrace though.

[AmrElsersy]

This is working for me, except for the 2d_full box type. On the example, it crashes after I orbit with the mouse for a bit. It also happens on gz-sim. I'm not being able to get an informative backtrace though.

@chapulina I will try to solve that :D

[WilliamLewww]

Fix out of bounds error when creating lines for 2D_full

Applied changes, thanks!

[chapulina]

All 3 camera types are working on the examples, tutorial, and downstream! Thanks to all the folks who got this across the finish line: @AmrElsersy @adlarkin @WilliamLewww ! 🏁