Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include new renderer based on Panda3D #1388

Merged
merged 67 commits into from
Jun 11, 2024
Merged

Conversation

SamFlt
Copy link
Contributor

@SamFlt SamFlt commented Apr 19, 2024

In the context of providing a new Render-based tracker, this PR introduces a new, easy to use renderer based on Panda3D.

This renderer can output:

  • A color image, with support for textures and lighting
  • Depth image (For now, this is a buffer between 0 and 1)
  • Normal maps
    • In world space
    • In camera space

It only supports camera models with no distortion.

It is also possible to compute camera clipping values, depending on the pose of an object in the camera frame. This ensures that the depth buffer is as accurate as possible when considering this object.

here is a set of renders for a textured object

Screenshot from 2024-04-19 18-11-58

and for a metallic object

Screenshot from 2024-04-19 18-28-40

Multi-output rendering is performed via the vpPanda3DRendererSet class, which duplicates the scene across multiple renders and synchronizes changes to objects and the camera. Each Sub renderer implements a specific type of render: geometric (vpPanda3DGeometryRenderer) or color-based (vpPanda3DRGBRenderer) etc. They all inherit from vpPanda3DBaseRenderer, which implements basic functions for a panda renderer.

Before this PR is complete, multiple things must be done:

  • Improve CMake support. As of now, Cmake will not work for windows, and probably not for Mac. Same goes for conda support;
  • Write the doc linked to the tutorial. Improve the tutorial to take path to the model to display as an argument;
  • Implement postprocessing filters. For instance, Running canny edge computation for depth maps via shaders is planned;
    • Base class vpPanda3DPostProcessFilter: which can take any fragment shader and compute an output image
    • grayscale conversion
    • Gaussian blur
    • Canny on grayscale image
  • Support for resizing the buffers;
  • Support for removing and adding sub renderers on the fly;
  • Add an interface to add lights to rendering in a clean way;
  • Use ViSP frames in normal output and object/camera position.

Other, potentially interesting, future tracks include:

  • Segmentation mapping
  • Light mapping: In the context of tracking textureless objects with high specularity or "metallicness", this information can be used to filter the features to use to compute pose update.

Copy link

codecov bot commented Apr 19, 2024

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 45.59%. Comparing base (f95b137) to head (a6ff73a).
Report is 20 commits behind head on master.

Current head a6ff73a differs from pull request most recent head 06791ed

Please upload reports for the commit 06791ed to get more accurate results.

Files Patch % Lines
tutorial/ar/tutorial-panda3d-renderer.cpp 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1388      +/-   ##
==========================================
+ Coverage   45.55%   45.59%   +0.03%     
==========================================
  Files        1082     1083       +1     
  Lines      109238   109216      -22     
  Branches    18658    18758     +100     
==========================================
+ Hits        49767    49792      +25     
+ Misses      59471    59424      -47     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@fspindle
Copy link
Contributor

  • How to build Panda3D from source on Ubuntu 22.04

    $ mkdir -p $VISP_WS/3rdparty/panda3d
    $ cd $VISP_WS/3rdparty/panda3d
    $ git clone https://github.com/panda3d/panda3d
    $ cd panda3d
    $ python3 makepanda/makepanda.py --everything --installer --no-egl --no-gles --no-gles2 --no-opencv
    

    At this point you can either

    1. install the produced Debian package (recommended) with
      $ sudo dpkg -i panda3d1.11_1.11.0_amd64.deb
      
    2. use the Panda3D libraries located in the built folder without installing the Debian package panda3d1.11_1.11.0_amd64.deb, but in that case you need to set LD_LIBRARY_PATH environment var:
      $ export LD_LIBRARY_PATH=$VISP_WS/3rdparty/panda3d/panda3d/built/lib:$LD_LIBRARY_PATH
      
      Without setting LD_LIBRARY_PATH you may experience the following error when running a binary that uses Panda3D capabilities:
      $ ./tutorial-panda3d-renderer 
      ./tutorial-panda3d-renderer: error while loading shared libraries: libp3dtoolconfig.so.1.11: cannot open shared object file: No such file or directory
      
  • How to build ViSP with Panda3D support when Debian package panda3d1.11_1.11.0_amd64.deb is installed

    There is nothing specific to do, just run cmake as usual

    $ cd $VISP_WS/visp-build
    $ cmake ../visp
    $ make -j$(nproc)
    
  • How to build ViSP with Panda3D support without installing Debian package panda3d1.11_1.11.0_amd64.deb

    • By setting Panda3D_DIR cmake var to the Panda3D cloned folder
      $ cd $VISP_WS/visp-build
      $ cmake ../visp -DPanda3D_DIR=$VISP_WS/3rdparty/panda3d/panda3d
      $ make -j$(nproc)
      
    • By setting Panda3D_DIR environment variable
      $ export Panda3D_DIR=$VISP_WS/3rdparty/panda3d/panda3d
      $ cd $VISP_WS/visp-build
      $ cmake ../visp
      $ make -j$(nproc)
      

@fspindle
Copy link
Contributor

fspindle commented Apr 23, 2024

  • How to build Panda3D from source on macOS

    • On macOS, you will need to download a set of precompiled third-party packages in order to compile Panda3D. Navigate to PandaED download page, select the lastest SDK (in our case SDK 1.10.14), and under </> Source Code section, download Thirdparty tools for macOS (in our case panda3d-1.10.14-tools-mac.tar.gz).
    • Extract third-party tools for macOS from downloaded archive
      $ cd ~/Downloads
      $ tar xvzf panda3d-1.10.14-tools-mac.tar.gz 
      
    • Once done clone Panda3D:
      $ mkdir -p $VISP_WS/3rdparty/panda3d
      $ cd $VISP_WS/3rdparty/panda3d
      $ git clone https://github.com/panda3d/panda3d
      $ cd panda3d
      
    • Move the downloaded third-party tools in Panda3D source code folder
      $ mv ~/Downloads/panda3d-1.10.14/thirdparty .
      
    • Build Panda3D from source
      $ python3 makepanda/makepanda.py --everything --installer --no-egl --no-gles --no-gles2 --no-opencv --no-python --threads $(sysctl -n hw.logicalcpu)
      
  • At this point you can either

    1. install the produced Panda3D-1.11.0-py3.9.dmg file (recommended) just by double clicking on it. In the installer window, don't forget to enable the C++ Header Files check box before pressing the installation button. After that you have to set DYLIB_LIBRARY_PATH environment var:
      % export DYLD_LIBRARY_PATH=/Library/Developer/Panda3D/lib:$DYLD_LIBRARY_PATH
      
    2. or use the Panda3D libraries located in the built folder without installing .dmg file, but in that case you need to set DYLIB_LIBRARY_PATH environment var:
      $ export DYLD_LIBRARY_PATH=$VISP_WS/3rdparty/panda3d/panda3d/built/lib:$DYLD_LIBRARY_PATH
      
      Without setting DYLD_LIBRARY_PATH you may experience the following error when running a binary that uses Panda3D capabilities:
      % ./tutorial-panda3d-renderer
      dyld[257]: Library not loaded: @loader_path/../lib/libpanda.1.11.dylib
      
  • How to build ViSP with Panda3D support when .dmg file Panda3D-1.11.0-py3.9.dmg is installed

    There is nothing specific to do, just run cmake as usual

    $ cd $VISP_WS/visp-build
    $ cmake ../visp -DUSE_PCL=OFF
    $ make -j$(sysctl -n hw.logicalcpu)
    
  • How to build ViSP with Panda3D support without installing the .dmg file

    • By setting Panda3D_DIR cmake var to the Panda3D cloned folder
      $ cd $VISP_WS/visp-build
      $ cmake ../visp -DUSE_PCL=OFF -DPanda3D_DIR=$VISP_WS/3rdparty/panda3d/panda3d
      $ make -j$(sysctl -n hw.logicalcpu)
      
    • Or by setting Panda3D_DIR environment variable
      $ export Panda3D_DIR=$VISP_WS/3rdparty/panda3d/panda3d
      $ cd $VISP_WS/visp-build
      $ cmake ../visp -DUSE_PCL=OFF
      $ make -j$(sysctl -n hw.logicalcpu)
      
  • How to install Panda3D latest SDK from existing packages

    • Installer are available for Ubuntu, macOS and Windows browsing the download page.
    • Note: For the latest Panda3D 1.10.14 SDK there is an Installer for macOS X 10.9+ that is only compatible with architecture x86_64. If you are using a Mac M1 or M2, there is no Panda3D SDK available yet for arm64 architecture. The solution is to build Panda3D from source.
  • Known issue on macOS

    • Segfault: :framework(error): Unable to create window
      % ./tutorial-panda3d-renderer
      Initializing Panda3D rendering framework
      Known pipe types:
        CocoaGLGraphicsPipe
      (all display modules loaded.)
      :framework(error): Unable to create window.
      zsh: segmentation fault  ./tutorial-panda3d-renderer
      
      This issue is probably due to EIGEN_MAX_ALIGN_BYTES and HAVE_PNG macro redefinition that occurs when building ViSP with Panda3D support:
      $ cd visp-build
      $ make
      ...
      [100%] Building CXX object tutorial/ar/CMakeFiles/tutorial-panda3d-renderer.dir/tutorial-panda3d-renderer.cpp.o
      In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
      In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
      In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
      In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
      In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
      In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:22:
      $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtool_config.h:40:9: warning: 'HAVE_PNG' macro redefined [-Wmacro-redefined]
      #define HAVE_PNG 1
              ^
      /opt/homebrew/include/pcl-1.14/pcl/pcl_config.h:53:9: note: previous definition is here
      #define HAVE_PNG
          ^
      In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
      In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
      In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
      In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
      In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
      In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:22:
      $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtool_config.h:64:9: warning: 'HAVE_ZLIB' macro redefined [-Wmacro-redefined]
      #define HAVE_ZLIB 1
              ^
      /opt/homebrew/include/pcl-1.14/pcl/pcl_config.h:55:9: note: previous definition is here
      #define HAVE_ZLIB
              ^
      In file included from $VISP_WS/visp/tutorial/ar/tutorial-panda3d-renderer.cpp:17:
      In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h:39:
      In file included from $VISP_WS/visp/modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h:42:
      In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandaFramework.h:17:
      In file included from $VISP_WS/3rdparty/panda3d/panda3d/built/include/pandabase.h:21:
      $VISP_WS/3rdparty/panda3d/panda3d/built/include/dtoolbase.h:432:9: warning: 'EIGEN_MAX_ALIGN_BYTES' macro redefined [-Wmacro-redefined]
      #define EIGEN_MAX_ALIGN_BYTES MEMORY_HOOK_ALIGNMENT
              ^
      /opt/homebrew/include/eigen3/Eigen/src/Core/util/ConfigureVectorization.h:175:11: note: previous definition is here
        #define EIGEN_MAX_ALIGN_BYTES EIGEN_IDEAL_MAX_ALIGN_BYTES
                ^
      3 warnings generated.
      [100%] Linking CXX executable tutorial-panda3d-renderer
      [100%] Built target tutorial-panda3d-renderer
      
      The work around consists in disabling PCL usage during ViSP configuration
      $ cd $VISP_WS/visp-build
      $ cmake ../visp -DUSE_PCL=OFF
      $ make -j$(sysctl -n hw.logicalcpu)
      

SamFlt and others added 27 commits April 24, 2024 17:36
- build fix on macOS
- Update copyright header
- Make header protection macro upper case
- Protect pragma omp with _OPENMP macro
- Remove empty lines
- Fix doxygen warnings
@fspindle fspindle merged commit f217431 into lagadic:master Jun 11, 2024
72 of 73 checks passed
@fspindle fspindle changed the title [WIP] Include new Panda3D Include new rendere based on Panda3D Jun 11, 2024
@fspindle fspindle changed the title Include new rendere based on Panda3D Include new renderer based on Panda3D Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants