System lock up with multiple devices

[DrMaxP]

I am using the latest SDK with multiple cameras connected and powered by USB C only. All cameras have the latest firmware. I see consistent whole level system hangs (all I can do is reboot) when using serval devices at once - e.g. using multiple instances of the Kinect viewer app or 3rd party solutions. There is nothing in the event log to suggest what happens. What else could I do to investigate the cause of the system lock up? This is on Window 11. The lock up occurs consistently, within a few minutes and normally within a free seconds.

I still need to look at the Azure Kinect logs, I've enabled the registry keys but can't locate the logs.

Not sure if this is a USB issue or a low level driver issue, given that the system is hanging and has to be rebooted. Seems to happen irrespective of which USB C ports / controllers I use.

[DrMaxP]

There's nothing obvious in the logs. Individual cameras seem to work fine. The more cameras I start up the more likely the system is to hang. This happens even at relatively low CPU usage. I did think it was a thermal issue but I've seen it happen when cores temps are still not very elevated. Does anyone else see similar behavior with multiple cameras? I have 8 connected, but once I start 4 the likelihood of a system lock up rises rapidly.

[DrMaxP]

RAM, GPU usage, GPU temp all seem fine when the crash happens.

The screenshot shows the system locked up when 7 Kinects were streaming using the Kinect Viewer app.

[DrMaxP]

So the system is more stable now, in that I can connect 8 devices and they'll stream from Azure Kinect viewer for much longer than they used to. I've shifted some of the cameras to a 4 port StarTech card which are now connected via USB-A->USB-C + power rather than all being powered by USB-C. It's certainly not reliable - I still see system crashes, but they are not immediate (i.e. within 1 minute). Yesterday they all streamed for >1hr, which they would never do before.

My conclusion is that the instability was coming from multiple USB stacks for different parts of the system (motherboard controllers, add in cards, TB4 cards, USB-C hub) running at the same time.

I have one Kinect that's somewhat unreliable in terms of connecting / errors.

[DrMaxP]

Upload.from.GitHub.for.iOS.MOV

[moversevlad]

Hello! For clarification, can you please share the model of the USB hub you're using? We also deal with those kind of problems and we seek to find a solution. Also, do you know at what FPS you're getting data? Are there any framedrops?
Thank you

[DrMaxP]

Hi, I have connected it to different USB ports, including those using the Renesas controller recommended by Microsoft. I have 7 other Azure Kinect cameras which do not show the same issues. Sometime the Kinect will not enumerate (appear in the list to select), sometimes it cannot be opened, sometimes it cannot be started, sometimes it will start but show corrupt data, sometimes it will start and then fail after a few seconds / minutes and sometimes it will work fine - e.g. at 30fps. Most of the time it has issues when all other 7 cameras do not. Regards, Max  On Wednesday, 26 April 2023 at 14:24:09 BST, moversevlad ***@***.***> wrote: Hello! For clarification, can you please share the model of the USB hub you're using? We also deal with those kind of problems and we seek to find a solution. Also, do you know at what FPS you're getting data? Are there any framedrops? Thank you — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[DrMaxP]

Hi, just following this up as I haven't heard anything back to my previous reply. On Thursday, 27 April 2023 at 16:30:38 BST, Max Palmer ***@***.***> wrote: Hi, I have connected it to different USB ports, including those using the Renesas controller recommended by Microsoft. I have 7 other Azure Kinect cameras which do not show the same issues. Sometime the Kinect will not enumerate (appear in the list to select), sometimes it cannot be opened, sometimes it cannot be started, sometimes it will start but show corrupt data, sometimes it will start and then fail after a few seconds / minutes and sometimes it will work fine - e.g. at 30fps. Most of the time it has issues when all other 7 cameras do not. Regards, Max  On Wednesday, 26 April 2023 at 14:24:09 BST, moversevlad ***@***.***> wrote: Hello! For clarification, can you please share the model of the USB hub you're using? We also deal with those kind of problems and we seek to find a solution. Also, do you know at what FPS you're getting data? Are there any framedrops? Thank you — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[TomHVR]

I think I am encountering a similar issue #1950

edit: I have a really stupid work around I discovered just for a laugh running the sensors each through a usb 3.0 hub, and then later USB 3.0 active extensions.

It seems being at 3.1 or 3.2 speed, at least on my host controllers (and possibly yours) causes some hardware fault that locks up the system. The active repeaters in the hub/cables forces the data to be at 3.0 speeds, thus not triggering the fault.

Extremely stupid but I guess it works.

[HM281099]

how to solve it, I had a similar problem, thanks

[TomHVR]

What worked for me was to connect the sensors with USB 3.0 extensions or use a 3.0 hub that downgraded the connection to the NUC.

[HM281099]

thanks for your reply , best wishes @TomHVR