dbt crashes: library libcrypto could not be found

[GuillaumeDesforges]

Describe the bug

dbt command fails with The library libcrypto could not be found:

Running with dbt=0.19.1
Found 69 models, 0 tests, 0 snapshots, 0 analyses, 317 macros, 0 operations, 0 seed files, 0 sources, 0 exposures

Encountered an error:
The library libcrypto could not be found

Steps To Reproduce

python -m venv .venv
pip install dbt
dbt run

Expected behavior

dbt should not crash

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.19.1
   latest version: 0.19.1

Up to date!

Plugins:
  - postgres: 0.19.1
  - snowflake: 0.19.1
  - redshift: 0.19.1
  - bigquery: 0.19.1

The operating system you're using:
NixOS

The output of python --version:

$ python --version
Python 3.8.7

Additional context

Note that I have in my environment:

$ echo $LD_LIBRARY_PATH 
/nix/store/sipmc4wnbcws4vahqlf5i06zz7xgnp23-gcc-10.2.0-lib/lib:/nix/store/wmhdjm0x0n4ffqh908gkhah3zr1d29fd-openssl-1.1.1i/lib
$ ls /nix/store/wmhdjm0x0n4ffqh908gkhah3zr1d29fd-openssl-1.1.1i/lib
engines-1.1  libcrypto.so  libcrypto.so.1.1  libssl.so  libssl.so.1.1

which did the trick for the same kind of issue when I tried to use singer snowflake target.

[jtcohen6]

Hey @GuillaumeDesforges, I found a StackOverflow post from four months ago that looks relevant:

https://stackoverflow.com/questions/65691479/the-library-libcrypto-could-not-be-found

The person posting there was trying to install snowflake-connector-python==2.3.7, and dbt-snowflake==0.19.1 requires snowflake-connector-python==2.3.6, so it's very likely the same issue:

This error can happen when loading the oscrypto (libcrypto) if the memory usage is too high. The OOM state cascades upward.

In effect, this likely isn't an issue with dbt per se, as much as an issue installing and importing snowflake-connector-python in your environment.

One way to test this hypothesis: you could try running pip install snowflake-connector-python[secure-local-storage]==2.3.6 in the same virtual environment, see if it's able to complete successfully, import it, and try building a connection with it in the same way dbt does:

import snowflake.connector
handle = snowflake.connector.connect(
    account="...",
    user="...",
    database="...",
    schema="...",
    warehouse="...",
    role="...",
    autocommit=False,
    client_session_keep_alive=False,
    password="..." # or authenticator="..."
)
result = handle.cursor().execute("select 1 as id")
result.fetchall()

I'm going to close this issue, since I don't think there are any code changes we can make in dbt to address this installation issue. We've already bumped the snowflake-connector-python requirement for the next minor version of dbt (v0.20).

[GuillaumeDesforges]

Thanks @jtcohen6 , indeed the code snippet above raises the error.

[GuillaumeDesforges]

To whomever might encounter this issue.

Turns out my system did not have objdump, which is required by ctypes, which otherwise fails silently. (??!)

On NixOs, installing binutils solves the issue.

[jtcohen6]

@GuillaumeDesforges Glad you were able to get to the bottom of it!

[FRidh]

The solution is to patch the calling code to hardcode the path to the library that needs to be loaded.

[parshvms]

hello - i;m getting oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto when i try to run the snowflake-sqlalchemy python connection in a docker image. Is there a solution to this problem? I'm using a docker desktop on windows machine.

[hhcs9527]

same here

[matwalk]

Hi! I've got the same error: oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto.
Any news about this?

[parshvms]

Apparently when i switched from conda to the python venv, this issue was gone. Try it out. I don't know the resolution.

[uberj]

I was getting 'error: oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto'

This was my issue: wbond/oscrypto#78