Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multimodal support for llamacpp_hf #6089

Draft
wants to merge 60 commits into
base: dev
Choose a base branch
from
Draft

Multimodal support for llamacpp_hf #6089

wants to merge 60 commits into from
+1,095 −15

Conversation

Touch-Night
Copy link
Contributor

@Touch-Night Touch-Night commented Jun 3, 2024 •
edited
Loading

Checklist:

I intergrated multimodal extensions, and am trying to make gguf multimodal works.
Now it can load and you can send image through ui, but it's super slow and the result is wrong too.
I don't know how to make it further, I need some help.

@oobabooga
Merge pull request oobabooga#4927 from oobabooga/dev
c3e0fcf 
Merge dev branch
@oobabooga
Merge pull request oobabooga#4937 from oobabooga/dev
443be39 
Merge dev branch
@oobabooga
Merge pull request oobabooga#4961 from oobabooga/dev
7be0983 
Merge dev branch
@oobabooga
Merge pull request oobabooga#4980 from oobabooga/dev
b28020a 
Merge dev branch
@oobabooga
Merge pull request oobabooga#4988 from oobabooga/dev
781367b 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5002 from oobabooga/dev
71eb744 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5005 from oobabooga/dev
5b791ca 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5011 from oobabooga/dev
c1f78db 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5012 from oobabooga/dev
489f4a2 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5022 from oobabooga/dev
11288d1 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5039 from oobabooga/dev
4b25acf 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5073 from oobabooga/dev
af87609 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5078 from oobabooga/dev
19d1374 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5100 from oobabooga/dev
3fd7073 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5132 from oobabooga/dev
3e3a66e 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5152 from oobabooga/dev
3f28925 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5163 from oobabooga/dev
c54d1da 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5181 from oobabooga/dev
8ea3f31 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5195 from oobabooga/dev
e169993 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5199 from oobabooga/dev
ad1ff53 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5220 from oobabooga/dev
2dc8db8 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5253 from oobabooga/dev
61e4bfe 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5266 from oobabooga/dev
d8c3a5b 
Merge dev branch (oobabooga#5257)
@oobabooga
Merge pull request oobabooga#5347 from oobabooga/dev
1343aa3 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5348 from oobabooga/dev
837bd88 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5379 from oobabooga/dev
e7a760e 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5404 from oobabooga/dev
4f3fdf1 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5452 from oobabooga/dev
a329db0 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5453 from oobabooga/dev
0f134bf 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5496 from oobabooga/dev
dc6adef 
Merge dev branch
oobabooga and others added 29 commits February 17, 2024 14:02
@oobabooga
Merge pull request oobabooga#5530 from oobabooga/dev
dd46229 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5534 from oobabooga/dev
7838075 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5549 from oobabooga/dev
d6bb6e7 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5574 from oobabooga/dev
ba85271 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5617 from oobabooga/dev
60f3d87 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5641 from oobabooga/dev
992affe 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5655 from oobabooga/dev
aa0da07 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5680 from oobabooga/dev
1934cb6 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5716 from oobabooga/dev
7cf1402 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5772 from oobabooga/dev
1a7c027 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5810 from oobabooga/dev
5b91dbb 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5822 from oobabooga/dev
65099dc 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5823 from oobabooga/dev
91a7370 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5848 from oobabooga/dev
26d822f 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5887 from oobabooga/dev
a4b732c 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5927 from oobabooga/dev
ad12236 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5959 from oobabooga/dev
81f603d 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5970 from oobabooga/dev
8f12fb0 
Merge dev branch
@oobabooga
Merge pull request oobabooga#5996 from oobabooga/dev
9ac5287 
Merge dev branch
@oobabooga
Merge pull request oobabooga#6027 from oobabooga/dev
abe5ddc 
Merge dev branch
@Touch-Night
Intergrated multimodal functionality
fcdd9d0
@Touch-Night
Remove instructblip-pipeline from the repo.
e8d714d
@Touch-Night
Add pipelines folder to .gitignore
904c53d
@Touch-Night
Now you can load multimodal models in llama.cpp
9b1da75
@Touch-Night
Also, you can use llamacpp_hf to load multimodal
248d9cc 
model, however it's not working now, you can only
load it but can't use it to do anything.
@Touch-Night
A small fix
7ce96dc
@Touch-Night
Multimodal support for llamacpp_hf, not working
d8caa3c 
embed_tokens() is not working, need to fix it
@Touch-Night
Add files via upload
605487a
@Touch-Night
Send your images to llamacpp_hf, wrong and slow
9372fb3
@Touch-Night Touch-Night mentioned this pull request Jul 12, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Touch-Night @oobabooga