Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ImportError: numpy.core.multiarray failed to import #54

Open
yuyingyeh opened this issue Feb 12, 2020 · 10 comments
Open

ImportError: numpy.core.multiarray failed to import #54

yuyingyeh opened this issue Feb 12, 2020 · 10 comments

Comments

@yuyingyeh
Copy link

I still cannot run ./build_toolbox.sh to build the environment after checking #40.

#40 suggests to check PYTHONPATH, however, in this post it said that Anaconda doesn't use PYTHONPATH.

I have tested the code on two different machines with Ubuntu 18.04 using Anaconda, but they came into same error messages. Could you help me solve this problem? Thanks.
@skamano
Copy link

skamano commented Feb 17, 2020 •
edited
Loading

I'm also getting the same issue. It seems to be related to install_trimesh.sh, I tried running the commands in the script without updating numpy and rtree and it's able to install trimesh and run build_toolbox.sh successfully but I still get some dependency issues when trying to test one of the models. My guess is that there are some conflicts between what environment.yml and the other build scripts specify for required packages.

@ztzhang
Copy link
Collaborator

ztzhang commented Feb 20, 2020

Would it be possible for you to share the NumPy version and the trimesh version in your environment?
I suspect this is because Trimesh has gone through a significant update and some of its dependencies are updated.
I'll try to see if I could reproduce this.

@skamano
Copy link

skamano commented Feb 20, 2020 •
edited
Loading

All I did to setup the environment was clone this repo and follow the instructions on the readme. This is a portion of the response after creating a new environment using environment.yml and running ./install_trimesh.sh:

  added / updated specs:
    - numpy
    - pyembree
    - rtree
    - scipy
    - shapely


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    certifi-2019.11.28         |           py36_0         149 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         149 KB

The following NEW packages will be INSTALLED:

  embree             conda-forge/linux-64::embree-2.14.0-0
  geos               conda-forge/linux-64::geos-3.8.0-he1b5a44_0
  libblas            conda-forge/linux-64::libblas-3.8.0-14_mkl
  libcblas           conda-forge/linux-64::libcblas-3.8.0-14_mkl
  liblapack          conda-forge/linux-64::liblapack-3.8.0-14_mkl
  pyembree           conda-forge/linux-64::pyembree-0.1.6-py36hb3f55d8_0
  shapely            conda-forge/linux-64::shapely-1.7.0-py36h5d51c17_0

The following packages will be UPDATED:

  numpy               anaconda::numpy-1.15.4-py36h7e9f1db_0 --> conda-forge::numpy-1.18.1-py36h95a1406_0
  rtree                        anaconda::rtree-0.8.3-py36_0 --> conda-forge::rtree-0.9.4-py36h7b0cdae_0

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates      anaconda::ca-certificates-2020.1.1-0 --> conda-forge::ca-certificates-2019.11.28-hecc5488_0
  certifi                                          anaconda --> conda-forge
  openssl                anaconda::openssl-1.1.1-h7b6447c_0 --> conda-forge::openssl-1.1.1d-h516909a_0
  scipy                anaconda::scipy-1.4.1-py36h0b6359f_0 --> conda-forge::scipy-1.4.1-py36h921218d_0


Proceed ([y]/n)? y

So this script updates numpy from 1.15.4 to 1.18 and rtree to 0.9.4. Skimage gets updated from 0.14 to 0.16.2 as well before trimesh is installed via pip. When I open up an interpreter and look at the PYTHONPATH, I see that it looks for packages in ~/.conda/envs/shaperecon/lib/python3.6/site-packages. However, updating numpy to version 1.18 from version 1.15.4 completely wipes out every module and subpackage within numpy. Numpy doesn't even have a __version__ attribute if everything is run exactly as the readme instructs and it seems to me that it's a mostly empty package. Trimesh is version 2.35.47.

After all this, I get exactly the same issue as @yuyingyeh when attempting to run build_toolbox.sh. If I reinstall numpy using conda install numpy=1.15.4, build_toolbox.sh runs to completion with no errors. However, attempting to run ./test_genre.sh results in:

Testing Pipeline
==> Parsing arguments
Traceback (most recent call last):
  File "test.py", line 19, in <module>
    opt = options_test.parse()
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/options/options_test.py", line 44, in parse
    parser, _ = get_model(net_name, test=True).add_arguments(parser)
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/__init__.py", line 5, in get_model
    module = importlib.import_module('models.' + alias)
  File "/datasets/home/73/773/skamano/.conda/envs/shaperecon/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/genre_full_model.py", line 5, in <module>
    from models.depth_pred_with_sph_inpaint import Net as Depth_inpaint_net
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/depth_pred_with_sph_inpaint.py", line 4, in <module>
    from models.marrnet1 import Model as DepthModel
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/marrnet1.py", line 6, in <module>
    from networks.networks import ViewAsLinear
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/networks/networks.py", line 2, in <module>
    from .revresnet import resnet18
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/networks/revresnet.py", line 6, in <module>
    from torchvision.models import resnet18
  File "/datasets/home/73/773/skamano/.conda/envs/shaperecon/lib/python3.6/site-packages/torchvision/__init__.py", line 2, in <module>
    from torchvision import datasets
  File "/datasets/home/73/773/skamano/.conda/envs/shaperecon/lib/python3.6/site-packages/torchvision/datasets/__init__.py", line 9, in <module>
    from .fakedata import FakeData
  File "/datasets/home/73/773/skamano/.conda/envs/shaperecon/lib/python3.6/site-packages/torchvision/datasets/fakedata.py", line 3, in <module>
    from .. import transforms
  File "/datasets/home/73/773/skamano/.conda/envs/shaperecon/lib/python3.6/site-packages/torchvision/transforms/__init__.py", line 1, in <module>
    from .transforms import *
  File "/datasets/home/73/773/skamano/.conda/envs/shaperecon/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 16, in <module>
    from . import functional as F
  File "/datasets/home/73/773/skamano/.conda/envs/shaperecon/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 5, in <module>
    from PIL import Image, ImageOps, ImageEnhance, PILLOW_VERSION
ImportError: cannot import name 'PILLOW_VERSION'

yet another dependency issue. Seems to me that Trimesh and its evolved dependencies are indeed causing the issue.

@ztzhang
Copy link
Collaborator

ztzhang commented Feb 20, 2020 •
edited
Loading

I just updated the install_trimesh.sh script and tested it with a new environment. It should be working now.
There are several issues:

  1. Trimesh updated significantly and requires a newer version of numpy. I added the version specs for Trimesh.
  2. torchvision only works with pillow<7, which is also added in the script. This is the reason why it complains not able to import PILLOW_VERSION.

Also please make sure you have CUDA<10 due to the pytorch version we are using.

@skamano
Copy link

skamano commented Feb 21, 2020 •
edited
Loading

Hi again, all the build scripts and environment setup steps seem to be ok, but I still get the following issue when trying to test the whole reconstruction pipeline using ./scripts/test_genre.sh 1 (I'm not really sure what to supply as an argument for the GPU):

Testing Pipeline
==> Parsing arguments
Traceback (most recent call last):
  File "test.py", line 19, in <module>
    opt = options_test.parse()
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/options/options_test.py", line 44, in parse
    parser, _ = get_model(net_name, test=True).add_arguments(parser)
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/__init__.py", line 5, in get_model
    module = importlib.import_module('models.' + alias)
  File "/datasets/home/73/773/skamano/.conda/envs/shaperecon/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/genre_full_model.py", line 5, in <module>
    from models.depth_pred_with_sph_inpaint import Net as Depth_inpaint_net
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/depth_pred_with_sph_inpaint.py", line 4, in <module>
    from models.marrnet1 import Model as DepthModel
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/marrnet1.py", line 8, in <module>
    from .marrnetbase import MarrnetBaseModel
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/marrnetbase.py", line 7, in <module>
    from toolbox.cam_bp.cam_bp.functions import CameraBackProjection
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/toolbox/cam_bp/cam_bp/functions/__init__.py", line 1, in <module>
    from .cam_back_projection import CameraBackProjection
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/toolbox/cam_bp/cam_bp/functions/cam_back_projection.py", line 4, in <module>
    from .._ext import cam_bp_lib
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/toolbox/cam_bp/cam_bp/_ext/cam_bp_lib/__init__.py", line 3, in <module>
    from ._cam_bp_lib import lib as _lib, ffi as _ffi
ImportError: /datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/toolbox/cam_bp/cam_bp/_ext/cam_bp_lib/_cam_bp_lib.so: undefined symbol: __cudaPopCallConfiguration

I'm kind of at a loss as to what's going on. I have CUDA 9.0 as well.

@ztzhang
Copy link
Collaborator

ztzhang commented Feb 21, 2020 via email

@skamano
Copy link

skamano commented Feb 21, 2020 •
edited
Loading

Yeah I just tried it right now, unfortunately with the same results. The issue may be with the docker image I've been using. The machine comes with CUDA 10 installed, but the conda environment uses CUDA 9, this case is pretty similar. I'll try and get a new image running Ubuntu 16 + Python 3.6 + CUDA 9.0 when I can either get access to a machine with a GPU or permission to use a custom image on the server I've been using, I haven't been able to change the container I've been allowed to use.

@hugovk
Copy link

hugovk commented Feb 24, 2020

2. torchvision only works with pillow<7, which is also added in the script. This is the reason why it complains not able to import PILLOW_VERSION.

torchvision 0.5.0 works with Pillow 7:

@ztzhang
Copy link
Collaborator

ztzhang commented Feb 24, 2020

@hugovk sure, but we never tested with this version. At the time of this project, we used 0.2.1 as stated in the environment.yml file.

@skamano
Copy link

skamano commented Feb 25, 2020

I just tried running the code, and I'm able to get the test script to finally start running so everything seems good build-wise, but the machine running this code must have CUDA 9.0 installed. It won't work if the machine has CUDA 10 since it seems like the dependencies leak out of the conda environment. I also ran into another issue while attempting to test the general reconstruction pipeline:

Namespace(adam_beta1=0.5, adam_beta2=0.9, batch_size=1, classes='chair', dataset=None, epoch=0, epoch_batches=None, eval_at_start=False, eval_batches=None, expr_id=0, full_logdir=None, gpu='GPU-54858a76-6cfa-0192-6ab1-ea7a91a92d56', inpaint_path=None, input_mask='./downloads/data/test/genre/*_silhouette.*', input_rgb='./downloads/data/test/genre/*_rgb.*', joint_train=False, load_offline=False, log_batch=False, log_time=False, logdir=None, lr=0.0001, manual_seed=None, net='genre_full_model', net1_path=None, net_file='./downloads/models/full_model.pt', optim='adam', output_dir='./output/test', overwrite=True, padding_margin=16, pred_depth_minmax=True, resume=0, save_net=1, save_net_opt=False, sgd_dampening=0, sgd_momentum=0.9, suffix='{net}', surface_weight=1.0, tensorboard=False, vis_batches_train=10, vis_batches_vali=10, vis_every_train=1, vis_every_vali=1, vis_param_f=None, vis_workers=4, wdecay=0.0, workers=0)
==> Setting device
[Verbose] All designated GPU(s) free to use. 
==> Setting up output directory
==> Setting up loggers
==> Setting up models
[Warning] Model loaded without optimizer states. 
Testing GenRe
# model parameters: 100,204,619
==> Setting up data loaders
[Verbose] Time spent in data IO initialization: 0.00s
[Verbose] # test points: 4
[Verbose] # test batches: 4
==> Testing
  0%|                                                                                                                                                                                 | 0/4 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "test.py", line 95, in <module>
    model.test_on_batch(i, batch)
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/genre_full_model.py", line 182, in test_on_batch
    pred = self.forward_with_trimesh(batch)
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/models/genre_full_model.py", line 214, in forward_with_trimesh
    rendered_sph = util_sph.render_spherical(pack, silhou)[None, None, ...]
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/util/util_sph.py", line 45, in render_spherical
    tdf = depth_to_mesh_df(depth_im, th, False, 1.0, 2.2)
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/util/util_img.py", line 54, in depth_to_mesh_df
    from util.util_camera import tsdf_renderer
  File "/datasets/home/home-02/73/773/skamano/GenRe-ShapeHD/util/util_camera.py", line 2, in <module>
    from scipy.misc import imresize
ImportError: cannot import name 'imresize'

The update scripts seem to mistakenly update scipy as well, but the above error was easily fixed by installing scipy 1.2.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hugovk @ztzhang @yuyingyeh @skamano