Intro to optimization in deep learning: Gradient Descent

[ngun7]

TL;DR

This article gives a basic but detailed understanding of optimization strategy in neural networks with beautiful visualizations.

Article Link

https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/

Author

Ayoosh Kathuria

Key Takeaways

• Loss function
• high-level overview of local & global minima
• Gradient descent & its challenges
• Different types of gradient descent functions

[khuyentran1401]

I like the visualization! From the pictures, it is easier to see the disadvantage of gradient descent such as the risk of stucking in local minima and how Stochastic Gradient Descent can fix the problem by introducing randomness