BAAF-Net - Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Overview

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

PWC
PWC
PWC
PWC

This repository is for BAAF-Net introduced in the following paper:

"Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion"
Shi Qiu, Saeed Anwar, Nick Barnes
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021)

Paper and Citation

The paper can be downloaded from here (CVF) or here (arXiv).
If you find our paper/codes/results are useful, please cite:

@inproceedings{qiu2021semantic,
  title={Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion},
  author={Qiu, Shi and Anwar, Saeed and Barnes, Nick},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={1757-1767},
  year={2021}
}

Updates

  • 04/05/2021 Results for S3DIS dataset (mIoU: 72.2%, OA: 88.9%, mAcc: 83.1%) are available now.
  • 04/05/2021 Test results (sequence 11-21: mIoU: 59.9%, OA: 89.8%) for SemanticKITTI dataset are available now.
  • 04/05/2021 Validation results (sequence 08: mIoU: 58.7%, OA: 91.3%) for SemanticKITTI are available now.
  • 28/05/2021 Pretrained models can be downloaded on all 6 areas of S3DIS dataset are available at google drive.
  • 28/05/2021 codes released!

Settings

  • The project is tested on Python 3.6, Tensorflow 1.13.1 and cuda 10.0
  • Then install the dependencies: pip install -r helper_requirements.txt
  • And compile the cuda-based operators: sh compile_op.sh
    (Note: may change the cuda root directory CUDA_ROOT in ./util/sampling/compile_ops.sh)

Dataset

  • Download S3DIS dataset from here.
  • Unzip and move the folder Stanford3dDataset_v1.2_Aligned_Version to ./data.
  • Run: python utils/data_prepare_s3dis.py
    (Note: may specify other directory as dataset_path in ./util/data_prepare_s3dis.py)

Training/Test

  • Training:
python -B main_S3DIS.py --gpu 0 --mode train --test_area 5

(Note: specify the --test_area from 1~6)

  • Test:
python -B main_S3DIS.py --gpu 0 --mode test --test_area 5 --model_path 'pretrained/Area5/snap-32251'

(Note: specify the --test_area index and the trained model path --model_path)

6-fold Cross Validation

  • Conduct training and test on each area.
  • Extract all test results, Area_1_conferenceRoom_1.ply ... Area_6_pantry_1.ply (272 .ply files in total), to the folder ./data/results
  • Run: python utils/6_fold_cv.py
    (Note: may change the target folder original_data_dir and the test results base_dir in ./util/6_fold_cv.py)

Pretrained Models and Results on S3DIS Dataset

  • BAAF-Net pretrained models on all 6 areas can be downloaded from google drive.
  • Download our results (ply files) via google drive for visualizations/comparisons.
  • More Functions about loading/writing/etc. ply files can be found from here.

Results on SemanticKITTI Dataset

  • Online test results (sequence 11-21): mIoU: 59.9%, OA: 89.8%
  • Download our test results (sequence 11-21 label files) via google drive for visualizations/comparisons.

  • Validation results (sequence 08): mIoU: 58.7%, OA: 91.3%
  • Download our validation results (sequence 08 label files) via google drive for visualizations/comparisons.
  • Visualization tools can be found from semantic-kitti-api.

Acknowledgment

The code is built on RandLA-Net. We thank the authors for sharing the codes.

Issues
  • Bilateral Context Module ends with downsampling?

    Bilateral Context Module ends with downsampling?

    Maybe it's not too significant,

    but it seems like your Bilateral Context Module ends with DownSampling, whereas the figure 2. in your paper describes BCM to end with Bilateral context block. Is it intended or is there something I'm missing?

    opened by deepshwang 5
  • The problem about aug_loss

    The problem about aug_loss

    Hi, it is a good job.I am interested in it! But I am confused with aug_loss. The code is followed:

    aug_loss_weights = tf.constant([0.1, 0.1, 0.3, 0.5, 0.5]) aug_loss = 0 # new_xyz_list=(B,N,16,3)...xyz_list=(B,N,3)... for i in range(self.config.num_layers): centroids = tf.reduce_mean(self.new_xyz[i], axis=2) # (B,N,3) relative_dis = tf.sqrt(tf.reduce_sum(tf.square(centroids-self.xyz[i]), axis=-1) + 1e-12) # (B,N,1) aug_loss = aug_loss + aug_loss_weights[i] * tf.reduce_mean(tf.reduce_mean(relative_dis, axis=-1), axis=-1) #weight*B

    Q1: why is the aug_loss_weight assigned like this? Q2: the output of code" tf.reduce_mean(tf.reduce_mean(relative_dis, axis=-1), axis=-1)" is B? Is my remarks correct? I hope to get your reply.

    opened by yangpanquan 4
  • The shifted neighbors and shifted neighbor feature in the BCM

    The shifted neighbors and shifted neighbor feature in the BCM

    Dear Sir: Hi , I am confused with the shifted problem in the bilateral context module.According to my memory, the RandLA-Net is not have the shifted problem.And your code is built on RandLA-Net.Is this a basic challenge in the point cloud semantic segmentation? what caused you to consider to solve it?I would like to take this opportunity to learn your way of thinking from you.Moreover, can you share your opinions about the problems exited in point cloud semantic segmentation.

    opened by yangpanquan 3
  • How to use multi-gpu for training

    How to use multi-gpu for training

    Dear sir: Have you try to use multi-gpu for your training? I have changed the code os.environ['CUDA_VISIBLE_DEVICES'] = "str(FLAGS.gpu)" to os.environ['CUDA_VISIBLE_DEVICES'] = "0, 1" When I type the message "nvidia-smi" in the command line , and I found that gpu1 is still not working in fact. It still have the problem of "OOM".How should I fix this?

    opened by yangpanquan 3
  • Mild request on code review written in pytorch

    Mild request on code review written in pytorch

    Hello,

    I have been doing projects on implementing your code on PyTorch, and the link is here

    https://github.com/deepshwang/BAAF-pytorch

    However, the model I coustructed is not converging, and is in process of debugging it...

    I know it's a tedious and time-consuming to review other's code, but wish you may review the model if you have some free time.. :)

    Many thanks to your outstanding work!

    opened by deepshwang 2
  • The error in the code running

    The error in the code running

    It is a good job, but I encountered some problems. After processing the data, there is no error, and the .ply file is also generated, but when I run BAAF-Net.py and main_s3dis.py , both files report the same error: 'process finished with exit code 139 (interrupted by signal 11: SIGSEGA)'.Besides, It can't Debug in the Pycharm. This troubles me. What should I do in this situation?

    opened by yangpanquan 2
  • Source Code

    Source Code

    Hi, @ShiQiu0419 I am very interesting in you work, so i want to know when the source code of this project will be published? Thank you very much!

    Best Regards.

    opened by xiaoyuamw 2
  • model.summary() /  model parameters calculation issue

    model.summary() / model parameters calculation issue

    I want to Print model summary, but when I call model.summary() in s3dis.main after model = Network(dataset, cfg) attribute error occurs. If you know any other way to calculate the number of model parameters that will be also helpful.

    thankyou

    Error: AttributeError: 'Network' object has no attribute 'summary'

    opened by raoumairwaheed 1
  • The Input need to be fixed in network?

    The Input need to be fixed in network?

    Hi,thanks for your @ShiQiu0419 Due to I saw the network have a parameter "num_points" in network to do Fathest Point Sampling,so why not use point.shape[0] as a dynamic n_points? Because the number of point in semantic KITTI is not all the same but in a range of 120000~130000.So whether the performance will be influenced if I change the way of setting the number of point?

    I have a question about how the network be used on semantic KITTI that if the input is 64*2^10 with batch_size 1, the gpu memory useage is about 18G+ like below:

    Sat Dec 11 21:38:27 2021       
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 465.31       Driver Version: 465.31       CUDA Version: 11.3     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
    | 36%   47C    P2   114W / 370W |  18763MiB / 24265MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |    0   N/A  N/A      1005      G   /usr/lib/xorg/Xorg                596MiB |
    |    0   N/A  N/A    723501      C   ...conda3/envs/pc/bin/python    18163MiB |
    +-----------------------------------------------------------------------------+
    
    opened by LeopoldACC 1
  • How to train Semantic3D dataset ?

    How to train Semantic3D dataset ?

    Thanks for your excellent work about point cloud semantic segmentaton。 I have tested the S3DIS dataset with the code you shared。But my own dataset is same as semantic3D dataset,so I want to know if you have any plans to share the code for training the Semantic3D dataset recently。

    opened by Zhaoguanhua 1
  • The training parameters of S3DIS

    The training parameters of S3DIS

    Hello, I have followed your tips, and trained the model. On Area 5, S3DIS dataset, the model I trained only achieved 61.078 mIOU. The mIoU in your paper is 65.4. Could your share your training parameters on S3DIS dataset? Looking forward to your reply.

    opened by M-leng 3
  • About the cuda error.

    About the cuda error.

    Hello, thank you for your good work. when I try to train the network with S3DIS, I get this error report: tensorflow.python.framework.errors_impl.NotFoundError: /home/logic/Desktop/2021Project/BAAF-Net/utils/sampling/tf_sampling_so.so: undefined symbol: __cudaPushCallConfiguration Do you have any suggestions?

    opened by WangZhouTao 8
Owner
PhD student of ANU affiliated with Data61-CSIRO
null
Visual-inertial-wheel fusion odometry, better performance in scenes with drastic changes in light

VIW-Fusion An visual-inertial-wheel fusion odometry VIW-Fusion is an optimization-based viusla-inertial-wheel fusion odometry, which is developed as a

庄庭达 170 Jun 22, 2022
A Multi-sensor Fusion Odometry via Smoothing and Mapping.

LVIO-SAM A multi-sensor fusion odometry, LVIO-SAM, which fuses LiDAR, stereo camera and inertial measurement unit (IMU) via smoothing and mapping. The

Xinliang Zhong 130 Jun 24, 2022
This repository is used for automatic calibration between high resolution LiDAR and camera in targetless scenes.

livox_camera_calib livox_camera_calib is a robust, high accuracy extrinsic calibration tool between high resolution LiDAR (e.g. Livox) and camera in t

HKU-Mars-Lab 365 Jun 20, 2022
This repository uses a ROS node to subscribe to camera (hikvision) and lidar (livox) data. After the node merges the data, it publishes the colored point cloud and displays it in rviz.

fusion-lidar-camera-ROS 一、介绍 本仓库是一个ROS工作空间,其中ws_fusion_camera/src有一个工具包color_pc ws_fusion_camera │ README.md │ └───src │ └───package: c

hongyu wang 14 Jun 15, 2022
Cloud Native Data Plane (CNDP) is a collection of user space libraries to accelerate packet processing for cloud applications.

CNDP - Cloud Native Data Plane Overview Cloud Native Data Plane (CNDP) is a collection of userspace libraries for accelerating packet processing for c

Cloud Native Data Plane 17 Jun 18, 2022
This code converts a point cloud obtained by a Velodyne VLP16 3D-Lidar sensor into a depth image mono16.

pc2image This code converts a point cloud obtained by a Velodyne VLP16 3D-Lidar sensor into a depth image mono16. Requisites ROS Kinetic or Melodic Ve

Edison Velasco Sánchez 6 May 18, 2022
This project is used for lidar point cloud undistortion.

livox_cloud_undistortion This project is used for lidar point cloud undistortion. During the recording process, the lidar point cloud has naturally th

livox 42 Jun 16, 2022
A simple localization framework that can re-localize in one point-cloud map.

Livox-Localization This repository implements a point-cloud map based localization framework. The odometry information is published with FAST-LIO. And

Siyuan Huang 71 Jun 20, 2022
Point Cloud Library (PCL)

Point Cloud Library Website The new website is now online at https://pointclouds.org and is open to contributions ??️ . If you really need access to t

Point Cloud Library (PCL) 7.5k Jun 20, 2022
GROR:A New Outlier Removal Strategy Based on Reliability of Correspondence Graph for Fast Point Cloud Registration

GROR GROR:A New Outlier Removal Strategy Based on Reliability of Correspondence Graph for Fast Point Cloud Registration About (a): correspondences gen

Pengcheng Wei 22 Jun 15, 2022
around — adaptive rounding operation

around — adaptive rounding operation Attempts to perform nice rounding of a floating point number, like a human would do. Usage: around.h #include "ar

Jan Ringoš 2 Jan 24, 2022
Adaptive Runtime AUTOSAR Linux Simulator

Adaptive-AUTOSAR Adaptive AUTOSAR is a simulated Adaptive Platform enviroment over Linux defined by AUTOSAR. The goal of this project is to implement

Armin Kassemi Langroodi 70 Jun 18, 2022
A composable container for Adaptive ROS 2 Node computations. Select between FPGA, CPU or GPU at run-time.

adaptive_component A composable stateless container for Adaptive ROS 2 Node computations. Select between FPGA, CPU or GPU at run-time. Nodes using har

ROS 2 Hardware Acceleration Working Group 6 Apr 25, 2022
Hands-On example code for Sensor Fusion and Autonomous Driving Stack based on Autoware

Autoware "Hands-On" Stanford Lecture AA274 / Graz University of Technology M. Schratter, J. Zubaca, K. Mautner-Lassnig, T. Renzler, M. Kirchengast, S.

Virtual Vehicle 23 Mar 25, 2022
Mars_lib - MaRS: A Modular and Robust Sensor-Fusion Framework

Introduction The Modular and Robust State-Estimation Framework, or short, MaRS, is a recursive filtering framework that allows for truly modular multi

Control of Networked Systems - University of Klagenfurt 124 Jun 19, 2022
VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation

VID-Fusion VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation Authors: Ziming Ding , Tiankai Yang, Kunyi Zhan

ZJU FAST Lab 71 Jun 9, 2022
Fisheye version of VINS-Fusion

VINS-Fisheye This repository is a Fisheye version of VINS-Fusion with GPU and Visionworks acceleration. It can run on Nvidia TX2 in real-time, also pr

HKUST Aerial Robotics Group 26 Jun 19, 2022
Port of Adafruit / NXP Sensor Fusion filter

AHRS Fusion Port of Adafruit NXP sensor fusion algorithms based on Kalman filters for rust. Resources https://github.com/adafruit/Adafruit_AHRS https:

Gaute Hope 4 May 14, 2022
EKF-based late fusion

深蓝学院多传感器融合感知课程 项目实现了Lidar与Camera的后融合感知算法,融合的算法基于扩展卡尔曼滤波(Extended Kalman Filter,EKF)。输入数据为Lidar检测结果以及Camera检测结果,检测算法与Apollo 6.0一致,Lidar检测算法为PointPillar

深蓝学院 4 Jun 8, 2022