Loss goes to NaN during training

[marshallllllll]

After a few hundred global steps, the loss goes to NaN. Any ideas as to why?

[parksunwoo]

Hi, I have a same problem...

8 random batch data selected by DataLoader... Done!
from graph, get pred_conf, pred_loc ... Done!
calculating (positive, negative, true_location) with jaccard overlap by DefaultBoxControl ... calculate_pos_neg_trueloc START
matching_dbboxes_gtboxes_in_batch_i START
get_top_confidences START
get_top_confidences END
matching_dbboxes_gtboxes_in_batch_i END
prepare_pos_neg_trueloc_in_matches START
prepare_pos_neg_trueloc_in_matches END
calculate_pos_neg_trueloc END
Done!
...
GLOBAL STEP : 29003 / LEARNING RATE : 0.001 / LOSS : nan ( 1.9297988414764404 secs)