Running on my own camera - how to make a custom cfg-file?

[mikkeljakobsen]

Hi,
I'm trying to run XIVO on a custom sequence of RGB-images + IMU-data recorded by a PicoZense DCAM710. I understand that in order for it to work, I need to update the configuration file.

Based on the phab.json, I've updated the Initial state "X", "imu_calib" and "camera_cfg". I obtained the IMU calibration parameters using the IMU_tk library. However, it does not work properly.

As far as I understand, the Wbc and Tbc is the rotation and translation of the camera-frame wrt the body frame respectively. Is that correct?

I'm most likely doing something wrong, and I'm in general quite confused of the many parameters in the config-file. So I was hoping that someone could give me a short step by step guide of what would needed to be done in order to apply XIVO to a custom camera+IMU.

[stephanietsuei]

Wbc and Tbc transform coordinates from the camera frame (attached to camera pinhole, right-down-out) to the body frame (IMU frame). So for a point Xc written in coordinates of the camera frame, the same point written in the body frame is Xb = Rbc*Xc + Tbc where Rbc is the rotation matrix version of the rotation vector Wbc. You can find this using the Kalibr toolbox if you don't have the numbers already.

I wrote about other things you can look at in this link.

Finally, since you are making your own dataset and using your own camera, you need to make sure that the motion in your dataset is sufficiently exciting (6-DOF motion). A sufficiently exciting initialization sequence at the beginning is fine - the filters just have to converge before the motion stops being sufficiently exciting. Planar motion, or motion with no linear acceleration, is not sufficiently exciting.

[mikkeljakobsen]

Thanks for your answer. I've used IMU_tk to get misalignment matrices, scale factors and biases. Then calibrated camera + IMU using Kalibr to obtain Rbc. Tbc is estimated by Kalibr to be [-0.2, -0.1, -1.7] which can't be correct. I don't know the exact translation of the IMU wrt the camera, but it should be very near so I quess that it's OK to put Tbc=[0,0,0] in the configuration file?

I still have problems with correct initialization of gravity. Tried to use the value from Kalibr, and also tried your suggestions from the link. But when I run the pyxivo.py-script, it does not look correct (the frame suddenly goes towards infinity). What am I doing wrong?

[stephanietsuei]

It is probably not okay to put Tbc = [0, 0, 0]. If you have your camera's spec sheet (probably a ~100 page document), then you can put that value and turn on online calibration. Judging by the value of Tbc that Kalibr computed, I'm guessing that it did not optimize properly. (You can try inflating numbers from the IMU's Allen plot, running the optimization for more iterations. Pay attention to all warnings that it prints out in the process.)

If the frame suddenly goes towards infinity, then my first guess is that gravity is not initialized properly. The IMU's x, y, z axes may not be perfectly aligned with gravity's x, y, z axes (the alignment is the state Wg). To calculate this alignment correctly, start your sequence from rest for a second or two, and set gravity_init_counter to an integer greater than zero. Otherwise, you will need an accurate value of Wg in your initial state X.

[mikkeljakobsen]

I've tried some of your suggestions, made a new calibration with Kalibr with inflated noise parameters and now I have a good approximation of the initial state X. It turned out that the estimated Rbc was wrong, but the value I've got from the new calibration seems more appropriate. Tbc is still way off. According to the camera-specs, Tbc is [-0.0007, -0.00052, 0.0], but it doesn't seem to make much difference if I set it to zero - it was Rbc that was the problem.

I've noticed that if Xivo looses track of the visual features, the frame will accelerate in a random direction and never get "back on track" - it just keeps accelerating in the same direction. Why does this happen? I'm mostly interested in getting correct camera poses within a local time frame. Is it possible to prioritize this somehow?

[stephanietsuei]

Glad to see that you made some progress!

Yes, if Xivo loses track of the visual features, as it is now, the frame will just accelerate in a random direction and it will never regain track as it is now. My guess is that errors in Rg (which is actually part of the state) cause us to subtract gravity in the wrong direction. 9.8m/s^2 in the wrong direction adds up. One thing that sometimes works for me is to carefully inflate the values in Qimu, initial_std _x, initial_std_y, initial_std_z, subfilter, visual_meas_std. This means that the filter is more likely to converge, but will be less accurate if it does.

[mikkeljakobsen]

I've tried to tune different parameters and got a better feeling of how it works. It's a bit more stable than before although the result is much more noisy than if I use the provided datasets, e.g. data9_workbench.

I would like to dump the features in an "online setting" for each image as sparse depth maps in order to use it together with this great work, also from your Lab:
https://github.com/alexklwong/unsupervised-depth-completion-visual-inertial-odometry

I can get the features, their pixel-coordinates and estimated position in the world frame. But I'm not sure how to get the depth. I've looked into the depth calculation scripts in the scripts folder, but I don't understand it completely.

Not sure if this is even the right place for this question, but I hope you can help me in the right direction.

[stephanietsuei]

Please ignore those depth calculation scripts. Those were used to generate data for one figure in the depth completion paper you are looking at, but basically, they are doing coordinate transformations to move each feature from the world frame to the camera frame and taking the Z-value of the position in the camera frame as the depth.

As you've discovered, there is not currently an interface to grab the depth of each feature in Xivo. Of course, Xivo does store depth of each feature, and writing that interface is not going to take long and is a useful interface to have. I'll let you know when it's done.

[stephanietsuei]

Done. See commit 2a12413

Please open a new issue if you've got any other questions or run into any other problems.