NameError: name 'MHA' is not defined

[byrneml]

Trying to load the model from HuggingFace across many environments (Jupyter Notebook, local on MacOS, all python 3.10 and latest version of transformers) yields the error:

self.inner_mha_cls = MHA(
NameError: name 'MHA' is not defined

The code I'm using is the sample code from your HuggingFace example!

[Zymrael]

Did you install flash attention?

[byrneml]

Yep - realized it’s likely an issue with the GPU I am using. Trying on an A100

…

On Wed, Feb 28, 2024 at 2:58 PM Michael Poli ***@***.***> wrote: Did you install flash attention? — Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AP7TXHX46RLZYOPOPS727XLYV6D5PAVCNFSM6AAAAABD4Y65MKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRZG42TCOJYGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Zymrael]

Was this resolved?

[OdedKBio]

I'm getting the same error on Apple M3, any solutions?

[amoskalev]

I have the following issue with A100 NameError: name 'swap_mha_rope' is not defined. Have anyone faced the same issue?

[xmz111]

Was this resolved?

I encounter this issue as well, it seems that flash_attn has some problems.
When I use pip install flash_attn, I get the version 2.5.6, but this version has some importing issues.
When I choose to install low version of flash_attn, it fails since cuda and pytorch version issues.
(3090)

[xmz111]

Was this resolved?

I can't solve this problem, using RTX3090.

[exnx]

It sounds like these are all flash attention compatibility issues with specific gpus

[xmz111]

It sounds like these are all flash attention compatibility issues with specific gpus

Hi! After experiment, a conclusion can be get that there is nothing to do with GPU, at least, RTX3090 doesn't cause this problem.
It is some compiling error in flash_attn library, and I guess it is caused by version conflict.
Class MHA can not be imported because this lib isn't be compiled correctly. By run the code in model.py in huggingface repo:
from flash_attn.modules.mha import MHA print(MHA), there goes a problem: File "/miniconda3/envs/for_evo/lib/python3.8/site-packages/flash_attn/flash_attn_interface.py", line 10, in <module> import flash_attn_2_cuda as flash_attn_cuda ImportError: libtorch_cuda_cpp.so: cannot open shared object file: No such file or directory
This problem occurs when I try to install flash_attn by using ./install.py from their repo. However, I try pip install flash_attn and install the version 2.5.6, no error is raised. Nevertheless, the problem still exists but there is no alert. When I import class in flash_attn, the compiling problem hinder importing, so this problem occurs.

[xmz111]

Dear all, this problem finally solved by changing flash_attn version.
First, this issue isn't caused by GPU. In flash_attn repo, it indicates A100 and RTX3090 is supported (See their repo), and I run the code successfully using 3090.
Second, according to README.md, pytorch should be installed first, however, version should be compatible between torch, cuda and related libs. For me, my version is below: torch 2.0.0+cu118 torchaudio 2.0.1+cu118 torchvision 0.15.1+cu118. Using nvcc -V, my version is Cuda compilation tools, release 11.8, V11.8.89. You may need install cudatoolkit and related libs using pip. For instance, conda install cudatoolkit==11.6 -c nvidia, conda install cuda-nvcc.
Third, this problem is caused by flash_attn lib and importing issue should be a global event, in other word, if you can import flash_attn correctly in your code, MHA can be imported correctly in model.py from Huggingface repo. So you should test if this lib can be used correctly first. For example, from flash_attn.modules.mha import MHA print(MHA), and a class will be returned.
The most important thing is that flash_attn has a lot of versions, so following to README.md, using pip install flash_attn , you may not get the version you need. What's worse, this lib won't report errors, even though compiling isn't complete, that's why I recommend you test whether this lib can be used. Given versions of cuda, torch, python, you can select a lib fits you in https://github.com/Dao-AILab/flash-attention/releases. Take myself as an example, pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.6/flash_attn-2.5.6+cu118torch2.0cxx11abiFALSE-cp38-cp38-linux_x86_64.whl.
I waste 3 days to figure out what happened to EVO. Hope that people can be reminded in README and everyone can enjoy EVO successfully.

[salehsereshkiucr]

Same issue for me using cuda_11.7.r11.7

[salehsereshkiucr]

The problem for me was the triton package was not installed. This solved the problem for me.

pip install triton

Also, I checked here https://github.com/Dao-AILab/flash-attention/releases and downloaded the right .whl file compatible with the cuda and torch version running on my system. and ran it by

pip install flash_attn-2.3.0+cu117torch1.13cxx11abiTRUE-cp38-cp38-linux_x86_64.whl

[hforoughmand]

I have similar problem on a V100. I installed several versions of pytorch and flash_attn and triton packages. But I receive the following error:

RuntimeError: Internal Triton PTX codegen error: 
ptxas /tmp/compile-ptx-src-b231af, line 147; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 147; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher              ptxas /tmp/compile-ptx-src-b231af, line 149; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 149; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 160; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 160; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 162; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 162; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 181; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 181; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 183; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 183; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 197; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 197; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 199; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 199; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 233; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 233; error   : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 235; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 235; error   : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 243; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 243; error   : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 245; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-b231af, line 245; error   : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas fatal   : Ptx assembly aborted due to errors

Any suggestion how to fix this problem?

[salehsereshkiucr]

Sorry, I have no idea what is going on. For me just installing triton have solved the problem

…

On Tue, Apr 30, 2024 at 11:51 AM Hadi ***@***.***> wrote: I have similar problem on a V100. I installed several versions of pytorch and flash_attn and triton packages. But I receive the following error: RuntimeError: Internal Triton PTX codegen error: ptxas /tmp/compile-ptx-src-b231af, line 147; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 147; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 149; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 149; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 160; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 160; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 162; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 162; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 181; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 181; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 183; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 183; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 197; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 197; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 199; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 199; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 233; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 233; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 235; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 235; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 243; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 243; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 245; error : Feature '.bf16' requires .target sm_80 or higher ptxas /tmp/compile-ptx-src-b231af, line 245; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher ptxas fatal : Ptx assembly aborted due to errors Any suggestion how to fix this problem? — Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYCKKOLLMGRAKECNTNUEXQ3Y77RZ5AVCNFSM6AAAAABD4Y65MKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBWGU3TCMJUG4> . You are receiving this because you commented.Message ID: ***@***.***>

[cclough]

I have the following issue with A100 NameError: name 'swap_mha_rope' is not defined. Have anyone faced the same issue?

I'm getting this too. did you find a solution?

[hforoughmand]

One suggestion from flash attention team was to use nightly build of everything. That solves some problems with the flash attention. But I still cannot use evo as I receive similar errors:

RuntimeError: Internal Triton PTX codegen error: 
ptxas /tmp/tmp8b6rqkxa.ptx, line 157; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 157; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 161; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 161; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 178; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 178; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 182; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 182; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 207; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 207; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 211; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 211; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 231; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 231; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 235; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 235; error   : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 271; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 271; error   : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 275; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 275; error   : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 289; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 289; error   : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 293; error   : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/tmp8b6rqkxa.ptx, line 293; error   : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas fatal   : Ptx assembly aborted due to errors

Any idea how can I fix it?

[JinwnK]

I have the following issue with A100 NameError: name 'swap_mha_rope' is not defined. Have anyone faced the same issue?

I'm getting this too. did you find a solution?

I have the same problem, any solutions?

[cx0]

I am running inference on A100 without any issue. Here are the specific versions for the (most) relevant libraries. Hope this helps.