Unable to install SNOPT

[sreekar2858]

Hello

I am running the latest DaFoam image (v3.0.4) from Docker on Windows.
I have a student license for SNOPT (fortran/C/C++) with the following files.

Following your instructions in #94. I created a folder in repos with pyoptsparse and copied all these SNOPT files into pySNOPT and ran pip install .

I get the output that the pyoptsparse is reinstalled in miniconda but the SNOPT is not installed (I don't see any compiled SNOPT file in pySNOPT in /home/dafoamuser/dafoam/packages/miniconda3/lib/python3.8/site-packages/pyoptsparse/pySNOPT/)

> dafoamuser@309d91951951:~/dafoam/repos/pyoptsparse-main$ pip install .
> Processing /home/dafoamuser/dafoam/repos/pyoptsparse-main
>   Installing build dependencies ... done
>   Getting requirements to build wheel ... done
>   Installing backend dependencies ... done
>   Preparing metadata (pyproject.toml) ... done
> Requirement already satisfied: sqlitedict>=1.6 in /home/dafoamuser/dafoam/packages/miniconda3/lib/python3.8/site-packages (from pyoptsparse==2.9.3) (2.1.0)
> Requirement already satisfied: scipy>1.2 in /home/dafoamuser/dafoam/packages/miniconda3/lib/python3.8/site-packages (from pyoptsparse==2.9.3) (1.7.1)
> Requirement already satisfied: mdolab-baseclasses>=1.3.1 in /home/dafoamuser/dafoam/packages/miniconda3/lib/python3.8/site-packages (from pyoptsparse==2.9.3) (1.6.1)
> Requirement already satisfied: numpy>=1.16 in /home/dafoamuser/dafoam/packages/miniconda3/lib/python3.8/site-packages (from pyoptsparse==2.9.3) (1.21.2)
> Building wheels for collected packages: pyoptsparse
>   Building wheel for pyoptsparse (pyproject.toml) ... done
>   Created wheel for pyoptsparse: filename=pyoptsparse-2.9.3-py3-none-any.whl size=893462 sha256=560b233375a713d69591447367e1af5299cfa8279cbd9b444ae4c9abcbbc372f
>   Stored in directory: /home/dafoamuser/.cache/pip/wheels/61/49/08/b144b1a7a3a1f948fb6afef67161c7f117b8290a80d5ad6446
> Successfully built pyoptsparse
> Installing collected packages: pyoptsparse
>   Attempting uninstall: pyoptsparse
>     Found existing installation: pyoptsparse 2.8.1
>     Uninstalling pyoptsparse-2.8.1:
>       Successfully uninstalled pyoptsparse-2.8.1
> Successfully installed pyoptsparse-2.9.3
> 
> [notice] A new release of pip is available: 23.0.1 -> 23.1
> [notice] To update, run: pip install --upgrade pip

I followed dafoam documentation to install pySNOPT and also the official installation guide, but couldn't get it installed.
I would like to run the Pitzdaily topology optimization tutorial.

Hope you can help me on this.

Best Regards,
Sreekar

edit: added pip install output

[friedenhe]

The DAFoam installation assumes you have the SNOPT source code. I am not 100% sure if you can use pySNOPT with a pre-compiled lib. I suggest you post this question/issue to the pyOptSparse repo: https://github.com/mdolab/pyoptsparse

[uttamcadambi07]

Hi, did you find out if you can use a pre-compiled library for SNOPT?

[sreekar2858]

Yes, If you have pre-compiled SNOPT files you can use that. It is working for me.

[friedenhe]

@uttamcadambi07 I want to know it too, please let us know if you manage to do that!

[friedenhe]

@sreekar2858 can you share some detailed steps with us on how you made it work with the pre-compiled lib?

[sreekar2858]

So, I copied a precompiled snopt.cpython-38-x86_64-linux-gnu.so file to the pySNOPT location (python library location from DAFoam documentation) on Docker.

[uttamcadambi07]

Okay I am still facing issues with this and I will list the steps that I followed hoping someone can reproduce it

1. I opened a thread on this link asking the question - SNOPT trial version usage pyoptsparse#363 (comment)
2. I got a response asking me to consult this - Problem Building pyOptSparse from Source on macOS (Apple Silicon) pyoptsparse#335 (comment)
3. So i followed it and wrote out a command like this

python3 -m numpy.f2py -m snopt -c /home/uttamcadambi/dafoam_v2/repos/pyoptsparse-2.8.1/pyoptsparse/pySNOPT/source/f2py/snopt.pyf -L/home/uttamcadambi/dafoam_v2/repos/pyoptsparse-2.8.1/pyoptsparse/pySNOPT/source -lsnopt7

this was as per the recommendations of the pyoptsparse folks. The *.so file @sreekar2858 is talking about is generated this way.
4. I set all the required paths as follows

PYOPTSPARSE_IMPORT_SNOPT_FROM=/home/uttamcadambi/dafoam_v2/repos/pyoptsparse-2.10.1/pyoptsparse/pySNOPT/source/
export SNOPT_LICENSE=/home/uttamcadambi/dafoam_v2/repos/pyoptsparse-2.10.1/pyoptsparse/pySNOPT/licenses/snopt7.lic
export LD_LIBRARY_PATH=/home/uttamcadambi/dafoam_v2/repos/pyoptsparse-2.10.1/pyoptsparse/pySNOPT/source:$LD_LIBRARY_PATH

5. Now when I try to run a script with snopt as the optimizer, I am met with this error:

+------------------------------------------------------------------------------+
| pyOptSparse Error: There was an error importing the compiled snopt module    |
+------------------------------------------------------------------------------+


+------------------------------------------------------------------------------+
| pyOptSparse Error: There was an error importing the compiled snopt module    |
+------------------------------------------------------------------------------+


+------------------------------------------------------------------------------+
| pyOptSparse Error: There was an error importing the compiled snopt module    |
+------------------------------------------------------------------------------+

Traceback (most recent call last):
  File "runScript.py", line 189, in <module>
    opt = OPT(args.opt, options=optOptions)
  File "/home/uttamcadambi/dafoam_v2/packages/miniconda3/lib/python3.8/site-packages/pyoptsparse/pyOpt_optimizer.py", line 985, in OPT
    return opt(*args, **kwargs)
  File "/home/uttamcadambi/dafoam_v2/packages/miniconda3/lib/python3.8/site-packages/pyoptsparse/pySNOPT/pySNOPT.py", line 92, in __init__
    raise Error("There was an error importing the compiled snopt module")
pyoptsparse.pyOpt_error.Error: There was an error importing the compiled snopt module

6. Funnily enough when I try to do python3 and import snopt, I get this error

Python 3.8.10 (default, Jun  4 2021, 15:09:15) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import snopt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /home/uttamcadambi/dafoam_v2/repos/pyoptsparse-2.10.1/pyoptsparse/pySNOPT/source/snopt.cpython-38-x86_64-linux-gnu.so: undefined symbol: openunit_

I will be extremely grateful to anyone who can help me with this. I feel like I am almost there.

[uttamcadambi07]

Update: I fixed this issue

Python 3.8.10 (default, Jun  4 2021, 15:09:15) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import snopt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /home/uttamcadambi/dafoam_v2/repos/pyoptsparse-2.10.1/pyoptsparse/pySNOPT/source/snopt.cpython-38-x86_64-linux-gnu.so: undefined symbol: openunit_

Here is an updated list of commands for those interested:

python3 -m numpy.f2py -m snopt -c ~/pyoptsparse-2.10.1/pyoptsparse/pySNOPT/source/f2py/snopt.pyf ~/pyoptsparse-2.10.1/pyoptsparse/pySNOPT/*.f -L~/pyoptsparse-2.10.1/pyoptsparse/pySNOPT/source -lsnopt7

replace ~ with your own paths.

This generates a binding file that goes like snopt.cpython-38-x86_64-linux-gnu.so

Then you have to set the paths as I have done in the previous post and when you do import snopt, you will not get an error.

But this still has not resolved my issue. I checked the source code and found this in this file ~/packages/miniconda3/lib/python3.8/site-packages/pyoptsparse/pySNOPT

def _import_snopt_from_path(path):
    """Attempt to import snopt from a specific path. Return the loaded module, or `None` if snopt cannot be imported."""
    path = os.path.abspath(os.path.expandvars(os.path.expanduser(path)))
    orig_path = sys.path
    sys.path = [path]
    try:
        import snopt  # isort: skip
    except ImportError:
        warnings.warn(
            f"`snopt` module could not be imported from {path}.",
            ImportWarning,
            stacklevel=2,
        )
        snopt = None
    finally:
        sys.path = orig_path
    return snopt


# Compiled module
_IMPORT_SNOPT_FROM = os.environ.get("PYOPTSPARSE_IMPORT_SNOPT_FROM", None)
if _IMPORT_SNOPT_FROM is not None:
    # if a specific import path is specified, attempt to load SNOPT from it
    snopt = _import_snopt_from_path(_IMPORT_SNOPT_FROM)
else:
    # otherwise, load it relative to this file
    try:
        from . import snopt  # isort: skip
    except ImportError:
        snopt = None

You see in this, if PYOPTSPARSE_IMPORT_SNOPT_FROM returns a non zero value (which it should because i set it in loadDAFoam.sh), then import snopt should work (as I demonstarted earlier). But it is not and this is worrying me a little. @friedenhe , am I making sense or am i missing something here?

[uttamcadambi07]

I fixed this issue! Refer to #520 (comment) for the solution!

[friedenhe]

It is great to see that you got it working!