How can I improve the behavior of trained model in real-time task?

[biohuangjue]

I have already trained my model and do well in predicting in a recorded video, which I recorded just as I am doing real-time task. It can recognize instances in all of random 20 frames . But in real-time task, it can only recognize approximately 1/6 of the frames , why so? And how can I improve my code? I've checked the latency is about15ms, so it can't be due to latency. Shall I add some thing to my code?I use opencv to grab real-time frame.