Transformers-from-Scratch

Table of Contents

• Transformers-from-Scratch
  • Table of Contents
  • About The Project
  • Demo
  • Documentation
  • File Structure
  • Contributors
  • References
  • License

About

PyTorch-based custom Transformer model from scratch for English-to-Spanish translation, inspired by the "Attention is All You Need" paper.

Demo

Documentation

Please refer /documentation or click here for complete documentation of the project

File Structure

👨‍💻Transformers-from-Scratch
 ┣ 📂assets                            // Contains all the reference gifs, images
 ┣ 📂documentation                     // Contains documentation and my notes on transformers
 ┃ ┣ 📄README.md
 ┣ 📄model.py                          // Transformer Architecture
 ┣ 📄train.py                          // Training loop
 ┣ 📄dataset.py                        // Loading & Preprocessing Dataset  
 ┣ 📄config.py 
 ┣ 📂visualization                     // Contains other visualizations
 ┃ ┣ 📄embedding.py
 ┃ ┣ 📄README.md
 ┣ 📄README.md

Contributors

• Prit Kanadiya

References

• YouTube Video by Umar Jamil on developing transformers from scratch.
• Link to Attention is all you need paper explaining transformer architecture
• opus_books dataset by huggingface
• Amirhossein Kazemnejad's Blog on Positional Encodings

License

MIT License