Pix2Pix multiclass structure for generating Lego figures from real images.
Access the website here: https://pikeras72.github.io/LegofyNet/
This project stems from inspiration found in the Generative Methods subject at the Polytechnic University of Madrid in 2023, where we initially developed the first version using a standard Pix2Pix implementation. This repository represents the continuation of the project, implementing a new Pix2Pix multiclass structure allowing users to select their desired class for generation. I have been experimenting with this new Pix2Pix approach for some time, and it yields more accuarate and realistic images compared to a standard Pix2Pix implementation for generating Lego figures from real-world images (focused on the Star Wars Universe). This is the first project I have encountered with this type of structure.
The dataset was created through web scraping and subsequently modified and cleaned. All training images were 128x64px and in .png format.
The images were categorized into different classes based on their type. The training set comprised 2328 images.
For example, given this image and selecting the class Calamari:
| Selected Image: | Generated Image: |
|---|---|
 |
 |
Another example, given this image and selecting the class Yoda:
| Selected Image: | Generated Image: |
|---|---|
 |
 |
Version 0.1 is out! This version includes the following:
- Model trained with
128x64px images. - Very simple interface just to try the functionality of the model.
- Deployment of the web page on Netlify.
- Allows users to upload an image, select a class and generate their image.
Weak points:
- The interface is very poor as it is only a prototype.
- Users should not have to choose the class every time if they want to generate multiple images from the same category.
- The image shown on the web page is very small, making it hard to appreciate.
Future Improvements:
- Improve interface (nav bar, favicon.ico, multiple HTMLs for each class, credits, help section, etc.). (#11)
- Upload a new and better model that generates images with higher quality. (#9)
- Increase the size of the image shown on the web page. (#4)
Version 0.2 now available! New changes added:
- New HTMLs pages for each class.
- LegofyNet logo and favicon.ico added to the web page.
- Nav bar included.
- If you want to generate an image you just have to click on the corresponding class name and you will be redirected to a page where you can generate images of the selected class.
Weak points:
- The nav bar is empty, no options are available.
- The interface is better but still very poor. Needs more animations and eye-catching elements.
- There are some js problems in production when trying to generate the first image, related to the model 'predict' method. (Not always occurs).
Future Improvements:
- Correct the 'model.predict' error. (#1)
- Add the version of the project in the webpage. (#13)
- Make the web responsive. (#12)
Version 0.3 is here! New changes added:
- Dark and Light mode available.
- Version number added to the webpage.
- 'model.predict' error detected.
Weak points:
- The 'model.predict' error is due to the loadModel() function if the user try to generate an image too quick.
- The webpage initialize in Dark mode, but the button shows as it was Light mode.
- The model is loaded in all of the webpages of the different classes, making the user experience slower.
Future Improvements:
- Since I'm a student, I'm still learning new things and I recently learned the importance of creating new branches for each feature and not always deploying to the main branch. (How had I not realized this before...) (ΓοΉΓ)
- Complete the fix for the 'model.predict' error. (#1)
- Initialize the Dark/Light mode button to Dark. (#7)
- Pass the version number to a global variable. (#8)
- Generate the images with and appropriate decoration and surroundings. (#5)
- Add details to the squares of each class at the initial page to better identify the class. (#6)
- Complete the Nav Bar sections (About, Credits, The Model, etc.). (#10)
Version 0.4 just landed! New changes added:
- Improved Dark/Light button and themes.
- Model loading errors fixed.
- The generated images are a bit bigger.
- Version number now is much easier to change.
- Drag&Drop system for uploading files.
Weak points:
- When Draging a file to the Drag&Drop section, it automatically start generating the Lego image, instead, it might be good to show the name of the selected file and a small image of it so the user knows what file they have selected.
- After having uploaded a file, a 'Generate Image' button should appear in order to generate a new image.
- Sometimes the user faces the situation where the model is not completely loaded yet. Until it's loaded, they can not select a file, buit they do not know that it is due to the model still loading. There should be a loading bar or a notification for them to know what is happening.
- The code have some variables or comments is Spanish that should be changed to English.
- The Dark/Light mode selection should be saved at the LocalSession so we could maintain the selected option through all the pages.
- There should be a text that indicates the name of the class you are generating in each page.
Future Improvements:
- Show the name of the file and it's image once uploaded. (#18)
- Show a 'Generate button' after having uploaded an image. (#19)
- Notify the user that the model is still loading until it's completely loaded. (#20)
- Translate the whole project to English. (#21)
- Improve the black/light mode. (#22)
- Indicate with a text the class that corresponds to each page. (#23)
Version 0.5 has arrived! New changes added:
- Preview of the name and image of the file uploaded by the user.
- The button to generate an image from the uploaded file is now available.
- Each web page displays the class that the user has selected.
- The generated images now have decorations around them.
- Until the model is not completely loaded, the user is not allowed to upload a file.
Weak points:
- It could be easier for the users if they could download the generated image by clicking on a button.
Future Improvements:
- Add a 'Download' button in case the user wants to save the generated image. (#31)
TensorFlow, TensorFlow.Keras and TensorFlow.js for the model training, the model structure and implementing the model in the web page.
HTML, CSS and JavaScript for the web page design and functionality.
Github Pages for hosting the web page.
ChatGPT-3.5 for the initial structure (HTML, CSS and JS) of the web page, the implementation of the model via TensorFlow.js in the web page and questions during the development of the project.
Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2016). Image-to-Image Translation with Conditional Adversarial Networks. arXiv.
pix2pix: Image-to-image translation with a conditional GAN. (n.d.). TensorFlow.
Brownlee, J. (2021, April 29). How to Implement Pix2Pix GAN Models From Scratch With Keras. MachineLearningMastery.com.