Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnCreate(handle) #164

Open
Alexandrsv opened this issue Jan 9, 2023 · 2 comments

Comments

@Alexandrsv
Copy link

python generate.py -s 512 512 -p "A painting of an apple in a fruit bowl"

It worked correctly before. Other neural networks work correctly. Maybe the reason is the video card update 1080 -> 4090
How to fix the error?

/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/pytorch_lightning/metrics/__init__.py:43: LightningDeprecationWarning: `pytorch_lightning.metrics.*` module has been renamed to `torchmetrics.*` and split off to its own package (https://github.com/PyTorchLightning/metrics) since v1.3 and will be removed in v1.5
  rank_zero_deprecation(
/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/augmentation/augmentation.py:1830: DeprecationWarning: GaussianBlur is no longer maintained and will be removed from the future versions. Please use RandomGaussianBlur instead.
  warnings.warn(
Working with z of shape (1, 256, 16, 16) = 65536 dimensions.
loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
VQLPIPSWithDiscriminator running with hinge loss.
Restored from checkpoints/vqgan_imagenet_f16_16384.ckpt
Using device: cuda:0
Optimising using: Adam
Using text prompts: ['A painting of an apple in a fruit bowl']
Using seed: 162421434451382733
0it [00:00, ?it/s]
Traceback (most recent call last):
  File "/run/media/alexandr/12T/Pytochnaya/VQGAN-CLIP/generate.py", line 862, in <module>
    train(i)
  File "/run/media/alexandr/12T/Pytochnaya/VQGAN-CLIP/generate.py", line 743, in train
    lossAll = ascend_txt()
  File "/run/media/alexandr/12T/Pytochnaya/VQGAN-CLIP/generate.py", line 722, in ascend_txt
    iii = perceptor.encode_image(normalize(make_cutouts(out))).float()
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/run/media/alexandr/12T/Pytochnaya/VQGAN-CLIP/generate.py", line 355, in forward
    batch = self.augs(torch.cat(cutouts, dim=0))
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/augmentation/base.py", line 247, in forward
    output = self.apply_func(in_tensor, in_transform, self._params, return_transform)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/augmentation/base.py", line 212, in apply_func
    output[to_apply] = self.apply_transform(in_tensor[to_apply], params, trans_matrix[to_apply])
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/augmentation/augmentation.py", line 685, in apply_transform
    return warp_affine(
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/geometry/transform/imgwarp.py", line 192, in warp_affine
    dst_norm_trans_src_norm: torch.Tensor = normalize_homography(M_3x3, (H, W), dsize)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/geometry/transform/homography_warper.py", line 396, in normalize_homography
    src_pix_trans_src_norm = _torch_inverse_cast(src_norm_trans_src_pix)
  File "/home/alexandr/.anaconda3/envs/vqgan/lib/python3.9/site-packages/kornia/utils/helpers.py", line 48, in _torch_inverse_cast
    return torch.inverse(input.to(dtype)).to(input.dtype)
RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnCreate(handle)
@ZJChe
Copy link

ZJChe commented Aug 6, 2023

Hi, I just ran into the same issue with 4090 video card. I finally managed to make it work with following modifications to the setup in README

pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 -f https://download.pytorch.org/whl/torch_stable.html

And:

pip install kornia==0.5.10

Don't know whether you still need it, just take a note in case someone else come across the same problem 😄

@lyl7650
Copy link

lyl7650 commented Jan 9, 2024

Hi, I just ran into the same issue with 4090 video card. I finally managed to make it work with following modifications to the setup in README

pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 -f https://download.pytorch.org/whl/torch_stable.html

And:

pip install kornia==0.5.10

Don't know whether you still need it, just take a note in case someone else come across the same problem 😄

it works to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants