[Alpha-VLLM Team] Add Lumina-T2X to diffusers #8652
Conversation
PommesPeter
changed the title
Co-authored-by: YiYi Xu <[email protected]>
Co-authored-by: YiYi Xu <[email protected]>
Co-authored-by: YiYi Xu <[email protected]>
Co-authored-by: YiYi Xu <[email protected]>
- Removed extra input parameter - Fixed typo on Feed_Forward Co-authored-by: YiYi Xu <[email protected]>
Co-authored-by: YiYi Xu <[email protected]>
Co-authored-by: YiYi Xu <[email protected]>
Co-authored-by: YiYi Xu <[email protected]>
Yes, this is called Grouped Query Attention proposed in this paper, which can optimize training and inference efficiency. |
There was a problem hiding this comment.
thanks! I think we can merge this soon
I have one question here need your input #8652 (comment) - can you look into to refactor with get_1d_rotary_pos_embed, or create a method similar to it for Lumina
the rest of changes are just to make sure the code structure and naming conventions are consistent with other models - we can certainly help with these to finish it up!
|
|
||
| return Transformer2DModelOutput(sample=output) | ||
|
|
||
| def precompute_freqs_cis( |
There was a problem hiding this comment.
I think we can use this instead?
diffusers/src/diffusers/models/embeddings.py
Line 277 in 0bae6e4
There was a problem hiding this comment.
Yes! I have refactored this by adding a new function called get_2d_rotary_pos_embed_lumina using get_1d_rotary_pos_embed inside. I added some lines of code in get_1d_rotary_pos_embed to enable context extrapolation proposed in our paper. (Note that this is a universal method as long as the model use RoPE, such as lumina and hunyuan. Besides, the added argument are disabled in default so they will not influence existing pipelines.)
| scale_msa, gate_msa, scale_mlp, gate_mlp = self.adaLN_modulation(adaln_input).chunk(4, dim=1) | ||
|
|
||
| # Self-attention | ||
| hidden_states = modulate(self.attn_norm1(hidden_states), scale_msa) |
There was a problem hiding this comment.
can we make a LuminaLayerNormZero similar to AdaLayerNormZero and keep it in this file since it's specific to Luminn?
Co-authored-by: YiYi Xu <[email protected]>
What does this PR do?
Add Lumina-T2X to diffusers
Fixes #8652
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.