Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

influence of the batch size and the number of GPUs #1

Open
he-y opened this issue Sep 22, 2020 · 3 comments
Open

influence of the batch size and the number of GPUs #1

he-y opened this issue Sep 22, 2020 · 3 comments
Labels
good first issue Good for newcomers question Further information is requested
Projects

Comments

@he-y
Copy link

he-y commented Sep 22, 2020

Thanks for your great work!
Could you please share the influence of the batch size and the number of GPUs?
Also how to choose a suitable learning rate and batch size if the available GPUs is not enough.
Thank you!

@PistonY
Copy link
Owner

PistonY commented Sep 23, 2020

Recently I use distribute train more often. You need to make sure single gpu has same batch size with me, you should get same result but may take more time if you have less gpu.

@he-y
Copy link
Author

he-y commented Sep 23, 2020

Thanks for your reply.
I understand that a single gpu should has the same batch size (128) as yours.
I have a question about the learning rate. Does the learning rate need to be changed?

Based on the Linear Scaling Rule in the paper(Accurate, Large Minibatch SGD:Training ImageNet in 1 Hour), the learning rate should be changed according to the batch size.

Linear Scaling Rule: When the minibatch size is multiplied by k, multiply the learning rate by k.

Thank you very much!

@PistonY
Copy link
Owner

PistonY commented Sep 23, 2020

No need to change I think. This paper should mean batch size on one device, normally batch size in paper just mean on device hold, take care of the difference between DistributedDataParallel and DataParallel.

@PistonY PistonY added good first issue Good for newcomers question Further information is requested labels Sep 24, 2020
@PistonY PistonY added this to Done in Work Plan Dec 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers question Further information is requested
Projects
Development

No branches or pull requests

2 participants