Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training results in NaN (commit 2b03725) #27

Open
bliep opened this issue Jan 29, 2018 · 3 comments
Open

Training results in NaN (commit 2b03725) #27

bliep opened this issue Jan 29, 2018 · 3 comments

Comments

@bliep
Copy link

bliep commented Jan 29, 2018

Running train.py --nr_gpu=1 results in NaN for the loss from bits_per_dim on tensorflow 1.5.0.
With some API changes (keepdims -> keep_dims, axis= -> ) the same NaN occurs on tensorflow 1.4.x. Disabling the data driven initialization of the weight normalization (commenting out sess.run(init_pass, feed_dict)) however does not result in NaN loss.
Any clues?

@yang-song
Copy link

I have the same problem.

@learning-luke
Copy link

I'm also getting an error like this. What did you change axis= into, by the way? I changed it to dim= to get it to work on tf 1.4.

@attaullah
Copy link

same issue here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants