实现SLAM与OpenMVS对接

Overview

详细介绍如何实现ORB-SLAM2与OpenMVS的对接过程,打通上下游实现三位重建,二话不说先干图

对于学SLAM的同学对ORB-SLAM2可能并不陌生,系统框架清晰明了,代码风格清新脱俗,几乎都快成为入门SLAM的必修课了。三维重建结构分为SFM和MVS这两个部分,常见的三维重建方法如图所示

对于学过SLAM的同学看到这可能就觉得不舒服了,对于SLAM和三位重建这俩同宗师兄弟竟然都不给介绍介绍,这那说的过去,确实对与SLAM,我们所理解的实时定位与建图其实他同样可以充当Sturt form Motion的角色,为MVS提供初始的相机位姿与稀疏点云。

本文以ORB-SLAM2与OpenMVS为例,详细介绍对接过程: ORB-SLAM2位姿导出 为与OpenMVS进行对接本次进对ORB-SLAM2进行部分修改,使之可以为OpenMVS提供稀疏点云、关键帧的位姿、内参,以及稀疏点云在各个View中的可见性。

主要更改如下

  • 在Map文件下增添如下函数
public:    
    void Save(const string &filename,const cv::MatSize image_size);
    void SaveMapPoint(ofstream &f, MapPoint* mp);
    void SaveKeyFrame(ofstream &f, KeyFrame* kf);
protected:
std::vector<int> KeyId;
  • 在System下增加:
void System::SaveMap(const string &filename,const cv::MatSize image_size);
  • 在mono_tum.cc或者orbslam的其他Examples中对System::SaveMap(const string &filename,const cv::MatSize image_size)这个函数进行调用即可。
SLAM.SaveMap("../Examples/output/sfm.txt",im.size);

通过ORB-SLAM2 我们最终生成了sfm.txt这个保存相机位姿以及稀疏点云的文档,他的文件结构如下图所示

数据集地址: https://vision.in.tum.de/data/datasets/rgbd-dataset/download#freiburg1_plant 以下是我们运行数据集的效果

sfm.txt的文件内容为:

ORB-SLAM2导出位姿验证

在与OpenMVS 进行对接之前,一定确保自己导出的信息是准确的,可以将相机三位空间坐标点以及相机在空间中的位置保存成ply、obj等三维格式,在meshlab中进行查看,或者如果你用的rgb-d相机的话,同样可以将深度图、rgb图一同投影下来,在meshlab下进行查看

下面则是我们根据得到的数据开始对OpenMVS进行初始化 OpenMVS初始化 为与SLAM进行对接,我们加入了read_pose.cpp、read_pose.h这两个c++文件,目的是对SLAM导出的位姿和稀疏点云进行读取,并对OpenMVS进行初始化。 主要核心函数有

bool load_scene(string file,Scene &scene);
bool read_mvs_pose(string file,MVSPOSE &mvs_pose);
bool save_pointcloud_obj(string name, vector<POINT3F> points,int num_keyframes,RGB color)

我们只需要在DensifyPointCloud.cpp下 对加入函数如下

// load and estimate a dense point-cloud
#define use_custom_pose
#ifdef use_custom_pose
    if(!load_scene(string(MAKE_PATH_SAFE(OPT::strInputFileName)),scene))
		return EXIT_FAILURE;
#else
	if (!scene.Load(MAKE_PATH_SAFE(OPT::strInputFileName)))
		return EXIT_FAILURE;
#endif

即可实现我们的初始化操作 对接结果 一下分别为稠密重建后的深度图融合、mesh重构、mesh优化以及问题贴图效果

InterfaceDensifyPointCloud接口理解 Scene.image中包含 尺度 分辨率 name 以及相关连的相机(这个platforms里面放置的是相机的内参和位姿) 在platforms中包含相机的内参 位姿 Camera类中包含相机ID 分辨率 相机内参,通过Read函数对stream进行读取自身参数 Image类这里包含图片ID,外参,相关连的相机,图片name以及稀疏点在当前图片的投影点

IMAGE_ID, QW, QX, QY, QZ, TX, TY, TZ, CAMERA_ID, NAME

Point类中 点的ID,位置,颜色,以及当前点的可见图像id 首先明确一点,我们SLAM的内容要如实的传入到Scene这个类中 首先将数据传入imgaes中 然后将数据传入platforms中 以及poses

对于想对其进行了解的同学可以到Bilibili观看我在计算机视觉life上的分享https://www.bilibili.com/video/BV1yX4y1A7qR?t=1

OpenMVS对SLAM位姿初始化部分已经合并到github: https://github.com/electech6/openMVS_comments

对于想要单独下载更改的文件的可以访问github: https://github.com/WangWen-Believer/ORB_SLAM2-to-OpenMVS

You might also like...
Comments
  • error: reference image 174 has not enough images in view

    error: reference image 174 has not enough images in view

    你好,使用orbslam2跑出sfm.txt后,放入openmvs中运行, 出现了这个问题。

    17:19:34 [App     ] CPU: Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz (6 cores)
    17:19:34 [App     ] RAM: 15.51GB Physical Memory 0B Virtual Memory
    17:19:34 [App     ] OS: Linux 5.10.0-8-generic (x86_64)
    17:19:34 [App     ] Disk: 23.68GB (78.24GB) space
    17:19:34 [App     ] SSE & AVX compatible CPU & OS detected
    17:19:34 [App     ] Command line: -w /home/frans/code/data/npu_image -i sfm.txt -o dense.mvs
    We are using Keyframe from SLAM to achieve 3D reconstruction
    ===========read_mvs_pose========
    ===========save_pointcloud_obj========
    ===========save_pointcloud_obj========
    load mvs ok 
    numViews 204
    deal with feature points
    =====================load_scene==================end======
    17:19:35 [App     ] Preparing images for dense reconstruction completed: 204 images (572ms)
    17:19:35 [App     ] error: reference image  56 has not enough images in view
    ···
    17:19:37 [App     ] error: reference image 146 has not enough images in view
    17:19:37 [App     ] error: reference image  67 has not enough images in view
    17:19:37 [App     ] error: reference image 139 has not enough images in view
    ···
    17:19:38 [App     ] error: reference image 120 has not enough images in view
    17:19:38 [App     ] error: reference image  39 has not enough images in view
    17:19:38 [App     ] error: reference image 174 has not enough images in view
    17:19:38 [App     ] Selecting images for dense reconstruction completed: 0 images (3s156ms)
    Fused depth-maps 0 (100%, 0ms)
    17:19:38 [App     ] Depth-maps fused and filtered: 0 depth-maps, 0 depths, 0 points (-2147483648%%) (0ms)
    17:19:38 [App     ] Densifying point-cloud completed: 0 points (3s812ms)
    17:19:38 [App     ] Scene saved (1ms):
    	204 images (204 calibrated)
    	0 points, 0 vertices, 0 faces
    17:19:38 [App     ] MEMORYINFO: {
    17:19:38 [App     ] 	VmPeak:	 1368416 kB
    17:19:38 [App     ] 	VmSize:	 1302880 kB
    17:19:38 [App     ] } ENDINFO
    

    这个问题是点云太少了吗?谢谢解答。

    opened by fanshixiong 6
  • 采用TUM 数据集,导入openmvs 程序crash

    采用TUM 数据集,导入openmvs 程序crash

    /libs/MVS/../Common/List.h:369: TYPE& SEACAVE::cList<TYPE, ARG_TYPE, useConstruct, grow, IDX_TYPE>::operator[](SEACAVE::cList<TYPE, ARG_TYPE, useConstruct, grow, IDX_TYPE>::IDX) [with TYPE = MVS::Image; ARG_TYPE = const MVS::Image&; int useConstruct = 2; int grow = 16; IDX_TYPE = unsigned int; SEACAVE::cList<TYPE, ARG_TYPE, useConstruct, grow, IDX_TYPE>::IDX = unsigned int]: Assertion `index < _size' failed.

    opened by liqingshengli 0
  • DensifyPointCloud运行失败

    DensifyPointCloud运行失败

    [email protected]:~/Desktop/work/test$ DensifyPointCloud -i sfm.txt 16:01:04 [App ] Build date: Jul 19 2021, 14:56:48 16:01:04 [App ] CPU: Intel(R) Xeon(R) E-2244G CPU @ 3.80GHz (8 cores) 16:01:04 [App ] RAM: 31.20GB Physical Memory 2.00GB Virtual Memory 16:01:04 [App ] OS: Linux 5.4.0-72-generic (x86_64) 16:01:04 [App ] SSE & AVX compatible CPU & OS detected 16:01:04 [App ] Command line: -i sfm.txt 16:01:04 [App ] error: invalid project

    工作空间目录结构: test ----images(包含orbslam2运行时读入的所有image) ----sfm.txt

    请问如何运行DensifyPointCloud指令呢

    opened by HuangTY96 8
Owner
wangWen
wangWen