-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to solve the slow inference problem with multiple categories? #20
Comments
Hi - if you only want to use the model for inference (rather than fine-tuning, further training, or something similar), have you tried using pybioclip? It might be easier to install/deploy/use. That said, pybioclip (as the name suggests) doesn't really do anything wildly different from the model inference code here. If I understand you correctly, you want to supply 6000 categories of your own and have it predict from among those? The part that takes substantially more time than the default "open-ended classification" (which uses the classes we trained on, for which the embeddings are already computed) is, in my understanding, embedding the custom classes, especially if you don't have a GPU. There's no way around that. However, this step should have to be done only once for the batch. I'm not sure whether pybioclip actually does this indeed only once (cc @johnbradley), or how the code here behaves. |
@hlapp pybioclip does not cache the category embeddings, it has the same logic as https://huggingface.co/spaces/imageomics/bioclip-demo. It would be simple to start caching them. One other thing to note is that bioclip generates 80 embeddings for each category. So for 6000 categories you end up with 480000 text embeddings. bioclip has templates that it applies to each category: |
@johnbradley am I right that by not caching you mean the category embeddings would be recomputed between one invocation of the program ( |
Text embeddings are calculated for each image image processed. This is really inefficient, but pre-existing logic. |
I created Imageomics/pybioclip#15 to address the caching problem in pybioclip. |
@ccc524 pybioclip now has changes to cache the text embeddings. The changes apply to both the command line tool and the python API. Given you are using 6000 categories the python API may be easier to work with. In my testing using a GPU sped up creating the text embeddings. So assuming you have a cuda graphics card you could run something similar to this:
Please see the updated documentation: |
Thanks @johnbradley. Perhaps we can close this issue now? @samuelstevens @thompsonmj any votes? |
Yes, I believe this addresses this issue adequately. To be clear for @ccc524, we expect with I might suggest as a further improvement for a new issue: the text embeddings cache to be optionally saved to disk and an option to load these from a specified filepath rather than recomputing if the category list is reused. |
The demo code is expected to be demo code only, and I had originally expected users to read and modify the code as necessary for particular use cases. But it seems to be a common issue that no-one is modifying the demo code, simply using as is. Saving embeddings to disk is an example of writing application-specific code, and while I am happy to make an example of this, I don't think this should be the default demo code. |
Thanks @samuelstevens, seems like you agree this is out of scope here, especially now that it's available in pybioclip. Thanks @ccc524 for bringing this to our attention and thereby prompting the change in pybioclip! |
Thank you very much for open-sourcing your model for us to learn from. I am facing a small issue and would appreciate your help. I want to infer a specific category name from a batch of images (there are about 6000 categories), and I only need the most confident one. However, the process is too slow because it seems to compute all 6000 categories. How can I resolve this?
![2](https://private-user-images.githubusercontent.com/171657490/344923792-d2d39cee-5761-435e-a74f-e161b2a556ff.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE0NDA2MjIsIm5iZiI6MTcyMTQ0MDMyMiwicGF0aCI6Ii8xNzE2NTc0OTAvMzQ0OTIzNzkyLWQyZDM5Y2VlLTU3NjEtNDM1ZS1hNzRmLWUxNjFiMmE1NTZmZi5qcGc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyMFQwMTUyMDJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xZDlkMTNkMjBhODc3NDU4M2I3MzE3NzhjNDFlNzVmYzk3MDQ3OWM0ZDBjNGY2MDYzZGNlYjdhMGEwYjViYmEwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.tujNwuANDSN7_IuhEn_LBo2efVGbCVzR8z8Qjq4FpTg)
Thank you!
The text was updated successfully, but these errors were encountered: