Loss Word2Vec

[nicocheh]

I am trying to get the loss of word2vec model. After trying in small models, I realized that taking the difference between the loss that the model gives in epochs is a decent measure of what is going on. I know is an issue that will take work to fix the loss logging. With that small solution I was able to track some kind of loss for experimenting.

I am trying it now with a new model and the loss that the model gives is always 134217728, that is 2**27. Probably because of a variable definition for the model as float32 or sth like that. Is there any chance to fix that at least so we can use the difference as a measure?

Does any one had this problem before?
With the current code, what do you recommend to track kind of a loss or sth like that?

Thanks in advance!

[gojomo]

The current loss-tallying & -reporting have a bunch of known problems (#2517); you're specifically hitting previously-reported #2735. The discussion on those issues have some ideas for partial workarounds; in some cases manually resetting the loss-tally to 0.0 between epochs may help – though you'll still hit problems if your corpus is large enough.

Outdated PR #2922 had some small fixes that could remedy some of the precision & multithreading issues; they might not be too hard to port into a local patch, or updated PR.

[gojomo]

Closing as duplicate & known-issue, and workarounds would be better discussed on #2735.