3D MNIST Classification Using PointNet, 2D-CNN, 3D-CNN, and some other ML methods

Introduction

In this work I used Pointnet, 2D-CNN, 3D-CNN, and some other ML methods to classify 3d-mnist point clouds. You can find the dataset here.

Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. You can use 3D-CNN_classifier.ipynb for applying 3D-CNN on 3D-voxel grids.

This, however, renders data unnecessarily voluminous and causes issues. In pointnet paper, they design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Their network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective.

In this repository, we used the code they published in their github and did some changes to use it in our work and 3d mnist data for training a PointNet classification network on point clouds.

You can use vis_data.py file to visualize point cloud data. I used plotly library for this.

Installation

Install TensorFlow. I used python 3.5 with TensorFlow 1.5. You may also need to install h5py.

To install h5py for Python:

sudo apt-get install libhdf5-dev
sudo pip install h5py

Usage

To train a pointnet model to classify point clouds sampled from 3D shapes use main_pointnet_3dmnist.ipynb file. Put the 3d mnist data to data folder.

You can also use other .ipynb's to use 2D-CNN, 3D-CNN, and some other ML methods for point cloud classification.

Selected Projects that Use PointNet

• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space by Qi et al. (NIPS 2017) A hierarchical feature learning framework on point clouds. The PointNet++ architecture applies PointNet recursively on a nested partitioning of the input point set. It also proposes novel layers for point clouds with non-uniform densities.
• Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds by Engelmann et al. (ICCV 2017 workshop). This work extends PointNet for large-scale scene segmentation.
• PCPNET: Learning Local Shape Properties from Raw Point Clouds by Guerrero et al. (arXiv). The work adapts PointNet for local geometric properties (e.g. normal and curvature) estimation in noisy point clouds.
• VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection by Zhou et al. from Apple (arXiv) This work studies 3D object detection using LiDAR point clouds. It splits space into voxels, use PointNet to learn local voxel features and then use 3D CNN for region proposal, object classification and 3D bounding box estimation.
• Frustum PointNets for 3D Object Detection from RGB-D Data by Qi et al. (arXiv) A novel framework for 3D object detection with RGB-D data. The method proposed has achieved first place on KITTI 3D object detection benchmark on all categories (last checked on 11/30/2017).