Check out the work required to continue publishing under the BSD license.

https://github.com/OpenVSLAM-Community/openvslam/pull/35#issuecomment-786322269

I think it would be good to clarify that the similarity between OpenVSLAM and ORB_SLAM2 is based on the ORB_SLAM2 paper

Tasks:

• [ ] (moved) #239
• Clarify the scope of contribution for each paper and treat it appropriately in README.md (for moral issue)
  • [x] ORB_SLAM/ORB_SLAM2
  • [x] ProSLAM
  • [x] UcoSLAM

Recently, I pulled the code with the latest version (openvslam commit code: f42bfcfabbaf3e4fd0021ac3e283bb89a4312db9 openvslam_ros commit code: a1e575aa4b811639a545644b438f5bf83cd17684 )

The updated codes are compiled successfully and installed in the dynamic link library When I run the executable run_slam in ROS2, it works fine temporarily. However, run time errors occur at any random time without outputting any error messages.

Using gdb prefix, it shows me the following backtrace just before the process dies.

#0  0x00007ffff7b24b1d in openvslam::data::landmark::erase_observation(std::shared_ptr<openvslam::data::keyframe> const&) () at /usr/local/lib/libopenvslam.so
#1  0x00007ffff7bc437e in openvslam::optimize::local_bundle_adjuster::optimize(std::shared_ptr<openvslam::data::keyframe> const&, bool*) const () at /usr/local/lib/libopenvslam.so
#2  0x00007ffff7ae3dde in openvslam::mapping_module::mapping_with_new_keyframe() ()
    at /usr/local/lib/libopenvslam.so
#3  0x00007ffff7ae4904 in openvslam::mapping_module::run() () at /usr/local/lib/libopenvslam.so
#4  0x00007ffff6b62de4 in  () at /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff7243609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007ffff6850293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

I've experimented on [intel NUC core i7 10gen with 16GB RAM], [RYZEN 7 5800H with 32GB RAM] and they both show the same issue. Is there anyone who solved this problem?

Continued from https://github.com/OpenVSLAM-Community/openvslam/issues/37.

Extracted similar modules between ORB_SLAM2 and OpenVSLAM.

• data modules
  • Frame - data::frame
  • KeyFrame - data::keyframe
  • MapPoint - data::landmark
  • Map - data::map_database
  • KeyFrameDatabase - data::bow_database
• visualization modules
  • Viewer, MapDrawer, FrameDrawer - pangolin_viewer, publish::*
• other modules
  • System - system, io::trajectory_io
  • Initializer - solve::*, initialize::*
  • Tracking - tracking_module, module::frame_tracker, module::keyframe_inserter, module::local_map_updater, module::relocalizer, module::initializer
  • LocalMapping - mapping_module, module::local_map_cleaner, module::two_view_triangulator
  • LoopClosing - global_optimization_module, module::loop_bundle_adjuster, module::loop_detector
  • Optimizer - optimize::*
  • ORBextractor - feature::*
  • ORBmatcher - match::*
  • PnPsolver - solve::pnp_solver
  • Sim3Solver - solve::sim3_solver

Based on these correspondences, similarity should be assessed by making comparisons at a appropriately abstract level. It is important to note that their creative expression may be limited by ideas. (Please refer Merger doctrine.)

In summary, under efficiency constraints, if different expressions are considered, they should be removed and rewritten from scratch.

Hi, thanks for maintaining the openVSLAM.

1. Build instruction doesn't build executable

I try to fresh install using the OpenVSLAM-Community documentation on Installation section, however, I found that it does not build the executable like run_video_slam, etc. This makes it hard to simply follow the tutorial here.

Shouldn't we update the build instruction to

cd /path/to/openvslam
mkdir build && cd build
cmake \
    -DBUILD_WITH_MARCH_NATIVE=ON \
    -DUSE_PANGOLIN_VIEWER=OFF \
    -DUSE_SOCKET_PUBLISHER=ON \
    -DUSE_STACK_TRACE_LOGGER=ON \
    -DBOW_FRAMEWORK=DBoW2 \
    -DBUILD_TESTS=ON \
    -DBUILD_EXAMPLES=ON \

    ..
make -j4

to also build the examples.

2. ROS example instruction

When I followed the build instruction for ROS. It will fail with

Base path: /home/jy/openvslam_ws
Source space: /home/jy/openvslam_ws/src
Build space: /home/jy/openvslam_ws/build
Devel space: /home/jy/openvslam_ws/devel
Install space: /home/jy/openvslam_ws/install
####
#### Running command: "cmake /home/jy/openvslam_ws/src -DUSE_PANGOLIN_VIEWER=ON -DUSE_SOCKET_PUBLISHER=OFF -DCATKIN_DEVEL_PREFIX=/home/jy/openvslam_ws/devel -DCMAKE_INSTALL_PREFIX=/home/jy/openvslam_ws/install -G Unix Makefiles" in "/home/jy/openvslam_ws/build"
####
-- Using CATKIN_DEVEL_PREFIX: /home/jy/openvslam_ws/devel
-- Using CMAKE_PREFIX_PATH: /opt/ros/melodic
-- This workspace overlays: /opt/ros/melodic
-- Found PythonInterp: /usr/bin/python2 (found suitable version "2.7.17", minimum required is "2") 
-- Using PYTHON_EXECUTABLE: /usr/bin/python2
-- Using Debian Python package layout
-- Using empy: /usr/bin/empy
-- Using CATKIN_ENABLE_TESTING: ON
-- Call enable_testing()
-- Using CATKIN_TEST_RESULTS_DIR: /home/jy/openvslam_ws/build/test_results
-- Forcing gtest/gmock from source, though one was otherwise available.
-- Found gtest sources under '/usr/src/googletest': gtests will be built
-- Found gmock sources under '/usr/src/googletest': gmock will be built
-- Found PythonInterp: /usr/bin/python2 (found version "2.7.17") 
-- Using Python nosetests: /usr/bin/nosetests-2.7
-- catkin 0.7.29
-- BUILD_SHARED_LIBS is on
-- BUILD_SHARED_LIBS is on
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- ~~  traversing 2 packages in topological order:
-- ~~  - cv_bridge
-- ~~  - openvslam_ros
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- +++ processing catkin package: 'cv_bridge'
-- ==> add_subdirectory(cv_bridge)
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found version "2.7.17") 
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   python
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.17", minimum required is "2.7") 
-- +++ processing catkin package: 'openvslam_ros'
-- ==> add_subdirectory(openvslam_ros)
-- Build type: Release
-- Address/Memory sanitizer: DISABLED
-- Stack trace logger: DISABLED
-- Google Perftools: DISABLED
-- Architecture-aware optimization (-march=native): DISABLED
-- C++11 support: OK (-std=c++17)
CMake Error at openvslam_ros/CMakeLists.txt:124 (find_package):
  By not providing "Findopenvslam.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "openvslam", but CMake did not find one.

  Could not find a package configuration file provided by "openvslam" with
  any of the following names:

    openvslamConfig.cmake
    openvslam-config.cmake

  Add the installation prefix of "openvslam" to CMAKE_PREFIX_PATH or set
  "openvslam_DIR" to a directory containing one of the above files.  If
  "openvslam" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!
See also "/home/jy/openvslam_ws/build/CMakeFiles/CMakeOutput.log".
See also "/home/jy/openvslam_ws/build/CMakeFiles/CMakeError.log".
Invoking "cmake" failed

I think it should be updated such that beginners/ practitioners could seamlessly use it. But since you refactor out the ros package of openVSLAM. I am not sure how to do it.

Describe the bug

When I load a map database with SLAM.load_map_database(map_db_path);, I get the following:

load a orb_params "default ORB feature extraction setting" from JSON

This seems to happen as part of load_map_database(), as the SLAM system then proceeds to parse keyframes and landmarks etc prior to the line [info] startup SLAM system. The parameters appear to be parsed correctly from the .yaml as the expected values are printed during system(). My current thinking is that the ORB settings used for creating the db end up being included as part of the file.

To Reproduce

Steps to reproduce the behavior:

1. Edit your config file to not have default name and/or values for ORB feature extraction
2. Observe how your desired values are successfully parsed during the ctor for system(const std::shared_ptr& cfg, const std::string& vocab_file_path)
3. call system::load_map_database(map_db_path)
4. Observe the message load a orb_params "default ORB feature extraction setting" from JSON, where "default ORB feature extraction setting" was the name of the Feature config used to build the file specified by map_db_path

Partial debug log is as follows:

Feature:
  name: Altopack Interior Test
  scale_factor: 1.2
  num_levels: 8
  ini_fast_threshold: 15
  min_fast_threshold: 10
Mapping:
  baseline_dist_thr_ratio: 0.02
  redundant_obs_ratio_thr: 0.9
Initializer:
  num_min_triangulated_pts: 100
PangolinViewer:
  keyframe_size: 1.2
  keyframe_line_width: 1
  graph_line_width: 2
  point_size: 2
  camera_size: 0.8
  camera_line_width: 3
  viewpoint_x: 0.0
  viewpoint_y: -300
  viewpoint_z: -0.1
  viewpoint_f: 2800

[2022-07-27 16:05:53.921] [info] loading ORB vocabulary: /orb_vocab.fbow
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: data::camera_database
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: data::map_database
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: data::bow_database
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: publish::frame_publisher
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: publish::map_publisher
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: module::initializer
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: module::relocalizer
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: tracking_module
[2022-07-27 16:05:53.939] [debug] CONSTRUCT: mapping_module
[2022-07-27 16:05:53.939] [debug] load mapping parameters
[2022-07-27 16:05:53.939] [debug] load monocular mappping parameters
[2022-07-27 16:05:53.939] [debug] Use baseline_dist_thr_ratio: 0.02
[2022-07-27 16:05:53.940] [debug] CONSTRUCT: loop_detector
[2022-07-27 16:05:53.940] [debug] CONSTRUCT: global_optimization_module
[2022-07-27 16:05:53.940] [debug] load depthmap factor
[2022-07-27 16:05:53.940] [debug] CONSTRUCT: data::orb_params_database
[2022-07-27 16:05:53.947] [info] clear map database
[2022-07-27 16:05:53.947] [info] clear BoW database
[2022-07-27 16:05:53.947] [info] load the MessagePack file of database from /maps/altopack2.msg
[2022-07-27 16:05:55.363] [info] decoding 1 camera(s) to load
[2022-07-27 16:05:55.363] [info] skip the tracking camera "ZED2"
[2022-07-27 16:05:55.363] [info] decoding 1 orb_params to load
[2022-07-27 16:05:55.363] [info] load a orb_params "default ORB feature extraction setting" from JSON
[2022-07-27 16:05:55.502] [info] decoding 104 keyframes to load
[2022-07-27 16:05:56.416] [info] decoding 40378 landmarks to load
[2022-07-27 16:05:56.440] [info] registering essential graph
[2022-07-27 16:05:56.646] [info] registering keyframe-landmark association
[2022-07-27 16:05:56.931] [info] updating covisibility graph
[2022-07-27 16:05:57.244] [info] updating landmark geometry
[2022-07-27 16:05:57.679] [info] startup SLAM system

config file:

# ZED2

#==============#
# Camera Model #
#==============#

Camera:
  name: "ZED2"
  setup: "stereo"
  model: "perspective"

  fx: 526.2252197265625
  fy: 526.2252197265625
  cx: 634.2841186523438
  cy: 348.4753112792969

  k1: 0.0
  k2: 0.0
  p1: 0.0
  p2: 0.0
  k3: 0.0

  fps: 15.0
  cols: 1280
  rows: 720
  focal_x_baseline: 63.147 # fx * 0.12
  depth_threshold: 40

  color_order: "Gray"

#=====================#
# Tracking Parameters #
#=====================#

Preprocessing:
  max_num_keypoints: 2000
  ini_max_num_keypoints: 4000

#================#
# ORB Parameters #
#================#

Feature:
  name: "Altopack Interior Test"
  scale_factor: 1.2
  num_levels: 8
  ini_fast_threshold: 15
  min_fast_threshold: 10

#====================#
# Mapping Parameters #
#====================#

Mapping:
  baseline_dist_thr_ratio: 0.02
  redundant_obs_ratio_thr: 0.9

#========================#
# Initializer Parameters #
#========================#

Initializer:
  num_min_triangulated_pts: 100

#===========================#
# PangolinViewer Parameters #
#===========================#

PangolinViewer:
  keyframe_size: 1.2
  keyframe_line_width: 1
  graph_line_width: 2
  point_size: 2
  camera_size: 0.8
  camera_line_width: 3
  viewpoint_x: 0.0
  viewpoint_y: -300
  viewpoint_z: -0.1
  viewpoint_f: 2800

Expected behavior

.yaml parameters are not over-written during the map loading process.

Environment

• Commit id: 584cff0c84c68d0cd7546d5158a82123fb04d4de
• Install procedure: docker

This PR fixes the stereo rectification configuration loading, which is reading the configuration from the wrong YAML node (one level up from where it should).

Adding new system.h/system.cc API call for updating camera position by known pose. This is a part required for https://github.com/OpenVSLAM-Community/openvslam_ros/issues/11 ticket to add the support of the /initialpose intentionally set by developer.

Thanks to all who contributed to this project. Also thanks for the community fork.

Is there a timeline for integrating the IMU functionality? Or a WIP branch? There are a number of cameras now equipped with the IMU(intel RealSense, StereoLab's Zed).

Much thanks Weiwu

Describe the bug

You have two videos from the same building. The only differences between the videos is the start and end are from different rooms inside this building. However, the middle part of both videos are from the exact same rooms in the building.

In one run a map is created and save from one of the videos .

Then in another run with the other video, the map is loaded into the system. The system is able to initialize as normally but right after the initialization, the system looses the feature tracking. When the video gets to the middle part, the system can recognize features from the map and feature tracking is regained. From this point the tracking and mapping module works perfectly so even when the video gets to the different end-part it keeps performing feature tracking and creating new key frames.

The value of fixing this bug

Imagine a robot that often have to move around in the same building - if it could use a previously recorded map its navigation would be much more stable. E.g. if the light in one of the usually corridors is out, the SLAM system would normally lose the tracking if the robot went trough this corridor. However, if the SLAM system had access to a previously map it could recover its location perfectly on the other side of the dark corridor.

To Reproduce

In the first run with one of the videos use "SLAM.save_map_database("map_of_stella.msg")" to save the map.

In the second run with the other video use "SLAM.load_map_database("map_of_stella.msg");" to load the map. (IMPORTANT: do NOT disable mapping module with "SLAM.disable_mapping_module();") You will then see the following error messages "[info] tracking lost: frame xx".

If you keep letting it run it will continue with "[info] tracking lost within 5 sec after initialization" and the whole system will reset. This resetting will delete the map you just loaded. You can deactivate the resetting but it does not help to solve the problem.

Suggested solution

I noticed a part of the problem is in the tracking_module.cc with the following lines:

  // pass all of the keyframes to the mapping module
    assert(!is_stopped_keyframe_insertion_);
    const auto keyfrms = map_db_->get_all_keyframes();
    for (const auto& keyfrm : keyfrms) {
        mapper_->queue_keyframe(keyfrm);
    }

This code loads all the key-frames from the previously map in a randomized order.

Therefore, I suggest the following code which makes sure that the key frames are loaded in the correct order:

    // pass all of the keyframes to the mapping module
    assert(!is_stopped_keyframe_insertion_);
    auto keyfrms = map_db_->get_all_keyframes();

    std::sort(keyfrms.begin(), keyfrms.end(),
              [&](std::shared_ptr<stella_vslam::data::keyframe>& keyfrm_1,
                  std::shared_ptr<stella_vslam::data::keyframe>& keyfrm_2) { return *keyfrm_1 > *keyfrm_2; });

However, this solution is not enough to solve the problem.

Environment

Hardware: [PC] CPU: [AMD Ryzen 7 5800X 8-Core Processor] OS: [Ubuntu 22.04] In my case it is not necessary to processes the video in real time

Hi. This is regarding a "tracking" status verification which can be useful for the Openvslam_ros's package in order to decide when publishing the pose info. Please, refer to this discussion for further details.

Hi,

I glanced through that spectrum and no one's answered anything as far as I could scroll. Maybe we should just put that to rest and remove links to it now that the official maintainers have given up on the project entirely.

Describe the bug

When loading a large size map, stella_vslam disconnects from socket_viewer. This problem was avoided by setting a limit on the data size to be sent, but it did not work properly in SLAM mode.

To Reproduce

Load a large map in Localization mode.

Expected behavior

• Map is displayed in socket_viewer in Localization mode when a large map is loaded
• Map is rendered successfully even in SLAM mode

Environment

• SocketViewer is running on Windows10
• Install procedure: docker

What issue is the feature request related to?

It would be useful to be able to disable viewer at runtime.

Describe the solution you'd like

Add a command line argument to start the viewer only when it is true.

Describe the bug

Following variables are unused and can be removed.

frm_obs in lambda thread_right https://github.com/stella-cv/stella_vslam/blob/a404f57c1999d8e29475b490e5fe26c38469df53/src/stella_vslam/system.cc#L326

variable ini_extractor_left_ https://github.com/stella-cv/stella_vslam/blob/a404f57c1999d8e29475b490e5fe26c38469df53/src/stella_vslam/system.h#L242

variable bow_db_ https://github.com/stella-cv/stella_vslam/blob/a404f57c1999d8e29475b490e5fe26c38469df53/src/stella_vslam/mapping_module.h#L225

https://github.com/stella-cv/stella_vslam/blob/a404f57c1999d8e29475b490e5fe26c38469df53/src/stella_vslam/module/initializer.h#L64

variable bow_vocab_ https://github.com/stella-cv/stella_vslam/blob/a404f57c1999d8e29475b490e5fe26c38469df53/src/stella_vslam/module/loop_detector.h#L126

variable fix_scale_in_Sim3_estimation_ https://github.com/stella-cv/stella_vslam/blob/a404f57c1999d8e29475b490e5fe26c38469df53/src/stella_vslam/module/loop_detector.h#L137

variable this https://github.com/stella-cv/stella_vslam/blob/a404f57c1999d8e29475b490e5fe26c38469df53/src/stella_vslam/module/local_map_updater.cc#L101

variable frame_hash_ https://github.com/stella-cv/stella_vslam/blob/a404f57c1999d8e29475b490e5fe26c38469df53/src/socket_publisher/data_serializer.h#L51

To Reproduce

The unused variable can be checked by searching them in the vscode or any other editor.

Expected behavior

Not Applicable

Screenshots or videos

Not Applicable

Environment

• Hardware: PC
• CPU: i7-12700
• OS: Ubuntu 22.04
• Commit id: a404f57c1999d8e29475b490e5fe26c38469df53

Update

I couldn't reopen the original issue, so I made this new issue. I have tried both disabling the reset and also setting the appropriate time stamp, but it still fails. I have created a link with two video, theirs logs, and the used map of the first video that is loaded into the second video, so you can see the exact error yourself. I have implemented the code lines from "suggested solution" into your code. The link is: https://mab.to/ogkuUgu5z.

Let me know if you need more from me.

Thank you so much for responding to these issues - it really makes it so much more fun to work with this repository compare to other SLAM repository :)

Describe the bug

You have two videos from the same building. The only differences between the videos is the start and end are from different rooms inside this building. However, the middle part of both videos are from the exact same rooms in the building.

In one run a map is created and save from one of the videos .

Then in another run with the other video, the map is loaded into the system. The system is able to initialize as normally but right after the initialization, the system looses the feature tracking. When the video gets to the middle part, the system can recognize features from the map and feature tracking is regained. From this point the tracking and mapping module works perfectly so even when the video gets to the different end-part it keeps performing feature tracking and creating new key frames.

The value of fixing this bug

Imagine a robot that often have to move around in the same building - if it could use a previously recorded map its navigation would be much more stable. E.g. if the light in one of the usually corridors is out, the SLAM system would normally lose the tracking if the robot went trough this corridor. However, if the SLAM system had access to a previously map it could recover its location perfectly on the other side of the dark corridor.

To Reproduce

In the first run with one of the videos use "SLAM.save_map_database("map_of_stella.msg")" to save the map.

In the second run with the other video use "SLAM.load_map_database("map_of_stella.msg");" to load the map. (IMPORTANT: do NOT disable mapping module with "SLAM.disable_mapping_module();") You will then see the following error messages "[info] tracking lost: frame xx".

If you keep letting it run it will continue with "[info] tracking lost within 5 sec after initialization" and the whole system will reset. This resetting will delete the map you just loaded. You can deactivate the resetting but it does not help to solve the problem.

Suggested solution

I noticed a part of the problem is in the tracking_module.cc with the following lines:

  // pass all of the keyframes to the mapping module
    assert(!is_stopped_keyframe_insertion_);
    const auto keyfrms = map_db_->get_all_keyframes();
    for (const auto& keyfrm : keyfrms) {
        mapper_->queue_keyframe(keyfrm);
    }

This code loads all the key-frames from the previously map in a randomized order.

Therefore, I suggest the following code which makes sure that the key frames are loaded in the correct order:

    // pass all of the keyframes to the mapping module
    assert(!is_stopped_keyframe_insertion_);
    auto keyfrms = map_db_->get_all_keyframes();

    std::sort(keyfrms.begin(), keyfrms.end(),
              [&](std::shared_ptr<stella_vslam::data::keyframe>& keyfrm_1,
                  std::shared_ptr<stella_vslam::data::keyframe>& keyfrm_2) { return *keyfrm_1 > *keyfrm_2; });

However, this solution is not enough to solve the problem.

Environment

Hardware: [PC] CPU: [AMD Ryzen 7 5800X 8-Core Processor] OS: [Ubuntu 22.04] In my case it is not necessary to processes the video in real time

Describe the bug

Sometimes the system cannot initialize from the beginning of a video if there are only few features in the images in the first part of the video. Therefore, the system initialized sometimes later - let's say at frame 50. Consequently, the trajectory between frame 0-50 is missing.

Suggested solution

This missing start trajectory can often be recovered by using exploiting the fact that the feature criteria for initialization is stricter than for the subsequently tracking. The recovery progress in this case is done by, at some point after the initialization, the video is set to frame 49 and running backwards with frame 48, 47, 46...0.

This works but there is a twist: the system is not able to handle frames with timestamps from the past. In my case, I use the function feed_monocular_frame(img, timestamp, mask). Instead of using the correct timestamps, when I feed the previous frames (49, 48, 47...0), I have to input timestamps that keep being ahead in time/in the future,

The problem of using timestamps that are in the past occurs in at least two places in the code:

• in file tracking_module.cc: if (curr_frm_.timestamp_ < last_reloc_frm_timestamp_ + 1.0) { return false; }

• keyframe_inserter.cc: bool min_interval_elapsed = false; if (min_interval_ > 0.0) { min_interval_elapsed = last_inserted_keyfrm && last_inserted_keyfrm->timestamp_ + min_interval_ <= curr_frm.timestamp_; min_interval_elapsed = true; }

Environment

• Hardware: [PC]
• CPU: [AMD Ryzen 7 5800X 8-Core Processor]
• OS: [Ubuntu 22.04]
• In my case it is not necessary to processes the video in real time