This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection

Last update: Dec 02, 2022

Related tags

Overview

Semantic SLAM

This package can perform optimization of pose estimated from VO/VIO methods which tend to drift over time. It uses planar surfaces extracted from object detections in order to create a sparse semantic map of the environment, thus optimizing the drift of the VO/VIO algorithms.

In order to run this package you will need two additional modules

A VO/VIO algorithm: ROVIO, VINS_MONO, OKVIS etc.
An Object Detector: Yolo, Shape Color Detector

Currently it can extract planar surfaces and create a semantic map from from the following objects:

chair
tvmonitor
book
keyboard
laptop
bucket
car

Related Paper:

@ARTICLE{9045978,
  author={Bavle, Hriday and De La Puente, Paloma and How, Jonathan P. and Campoy, Pascual},
  journal={IEEE Access}, 
  title={VPS-SLAM: Visual Planar Semantic SLAM for Aerial Robotic Systems}, 
  year={2020},
  volume={8},
  number={},
  pages={60704-60718},
  doi={10.1109/ACCESS.2020.2983121}}

Video

How do I set it up?

First install g2o following these instructions (Tested on Kinetic and Melodic Distributions):

- sudo apt-get install ros-$ROS_DISTRO-libg2o
- sudo cp -r /opt/ros/$ROS_DISTRO/lib/libg2o_* /usr/local/lib
- sudo cp -r /opt/ros/$ROS_DISTRO/include/g2o /usr/local/include

Install OctopMap server for map generation capabilities:

- sudo apt install ros-$ROS_DISTRO-octomap*

Try a simple example with pre-recorded VIO pose and a blue bucket detector:

Create a ros workspace and clone the following packages:

Download the rosbag:

    wget -P ~/Downloads/ https://www.dropbox.com/s/jnywuvcn2m9ubu2/entire_lab_3_rounds.bag

Create a workspace, clone the repo and compile:

    mkdir -p workspace/ros/semantic_slam_ws/src/ && cd workspace/ros/semantic_slam_ws/src/    
    git clone https://github.com/hridaybavle/semantic_slam && git clone https://bitbucket.org/hridaybavle/bucket_detector.git   
    cd .. && catkin build --cmake-args -DCMAKE_BUILD_TYPE=Release

Launch and visualize

    source devel/setup.bash
    roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

Using Docker Image

If the code is giving problems with you local machine, you can try the docker image created with the repo and the required settings.

Download Docker from: Docker

Follow the commands to run the algorithm with the docker

  docker pull hridaybavle/semantic_slam:v1 	
  docker run --rm -it --net="host" -p 11311:11311 hridaybavle/semantic_slam:v1 bash
  cd ~/workspace/ros/semantic_slam_ws/
  source devel/setup.bash
  roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=false

Open a new terminal and rviz in local machine

  cd ~/Downloads/ && wget https://raw.githubusercontent.com/hridaybavle/semantic_slam/master/rviz/graph_semantic_slam.rviz
  rviz -d graph_semantic_slam.rviz

Subsribed Topics

/SQ04/snap_vislam/vislam/pose (geometry_msgs/PoseStamped)
The default snapdragon VIO pose published in NED in frame. This message can be remapped remapped to any other VO pose message publishing in NED frame. (See frame conventions)
/rovio/odometry (geometry_msgs/PoseStamped)
The VIO odometry published in ENU frame. Can be remapped to the desired topic name in the launcher.
/depth_registered/points (sensor_msgs/PointCloud2) The point cloud required for planar surface extraction.
/darknet_ros/bounding_boxes(darknet_msgs_ros/BoundingBoxes)
The detection bounding boxes published by yolo if using the yolo detector ros package.
/image_processed/bounding_boxes(ShapeColor_ObjectDetection/DetectedObjects) The detection bounding boxes if using the bucket detector. It can be downloaded from the link above.

Published Topics

robot_pose(geometry_msgs/PoseStamped) The pose of the robot estimated by the algo.
robot_path(nav_msgs/Path)
The path of the robot psoe for visualization.
keyframe_poses(geometry_msgs/PoseArray)
The poses of the keyframe being added to the g2o graph.
mapped_landmarks(visualization_msgs/MarkerArray)
The mapped semantic planar surfaces.
detected_landmarks(visualization_msgs/MarkerArray)
The detected landmarks in the current frame.

The configurations of the algorithms can be found inside the cfg folder in order to be changed accordingly.

Published TFs

map to odom transform: The transform published between the map frame and the odom frame after the corrections from the semantic SLAM.
base_link to odom transform: The transform published between the base_link (on the robot) frame and the odom frame as estimated by the VO/VIO algorithm.

You might also like...

Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

HV-plane reconstruction from a single 360 image Code for our paper in CVPR 2021: Indoor Panorama Planar 3D Reconstruction via Divide and Conquer (pape

36 Jan 3, 2023

[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Planar Surface Reconstruction From Sparse Views Linyi Jin, Shengyi Qian, Andrew Owens, David F. Fouhey University of Michigan ICCV 2021 (Oral) This re

89 Jan 5, 2023

PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking

Homography Decomposition Networks for Planar Object Tracking This project is the offical PyTorch implementation of HDN(Homography Decomposition Networ

48 Dec 15, 2022

Pytorch implementation of paper: "NeurMiPs: Neural Mixture of Planar Experts for View Synthesis"

NeurMips: Neural Mixture of Planar Experts for View Synthesis This is the official repo for PyTorch implementation of paper "NeurMips: Neural Mixture

101 Dec 13, 2022

Sequence lineage information extracted from RKI sequence data repo

Pango lineage information for German SARS-CoV-2 sequences This repository contains a join of the metadata and pango lineage tables of all German SARS-

24 Oct 26, 2022

Official page of Struct-MDC (RA-L'22 with IROS'22 option); Depth completion from Visual-SLAM using point & line features

Struct-MDC (click the above buttons for redirection!) Official page of "Struct-MDC: Mesh-Refined Unsupervised Depth Completion Leveraging Structural R

37 Dec 22, 2022

Code for "Diffusion is All You Need for Learning on Surfaces"

Source code for "Diffusion is All You Need for Learning on Surfaces", by Nicholas Sharp Souhaib Attaiki Keenan Crane Maks Ovsjanikov NOTE: the linked

247 Dec 28, 2022

Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

Implementation for Iso-Points (CVPR 2021) Official code for paper Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations paper |

66 Nov 8, 2022

Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021)

Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021) This repository contains the code

149 Dec 15, 2022

Comments

errors at last step

Hi, I have finished all the steps following the instructions and nothing goes wrong. But when I run

roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

I get errors like this and it stucks for a while.

# Using CSparse poseDim -1 landMarkDim -1 blockordering 0
done
keyframe_delta_trans 0.5
keyframe_delta_angle 0.5
keyframe_delta_time 1
use_const_inf_matrix: 1
const_stddev_x: 0.00667
const_stddev_q: 1e-05
Initialized mapping thread 
camera angle in radians: 0.59219
update keyframe every detection: 1
add first landmark: 0
[semantic_graph_slam_node-9] process has died [pid 23067, exit code -11, cmd /home/nrc/workspace/ros/semantic_slam_ws/devel/lib/semantic_SLAM/semantic_graph_SLAM_node __name:=semantic_graph_slam_node __log:=/home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/semantic_graph_slam_node-9.log].
log file: /home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/semantic_graph_slam_node-9*.log

then I get this. It seems that the visualization program doesn't go right.

[rosbag-2] process has finished cleanly
log file: /home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/rosbag-2*.log

Is there something I have missed? Thank you!

opened by ZhengXinyue 8

[semantic_graph_slam_node-9] process has died

Hi, I have finished all the steps following the instructions and nothing goes wrong. But when I run

roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

I get errors like this.

done
keyframe_delta_trans 0.5
keyframe_delta_angle 0.5
keyframe_delta_time 1
use_const_inf_matrix: 1
const_stddev_x: 0.00667
const_stddev_q: 1e-05
camera angle in radians: 0.59219
update keyframe every detection: 1
add first landmark: 0
[ INFO] [1591944956.099907360, 1661396775.076756992]: waitForService: Service [/depth_rectifier_manager/load_nodelet] is now available.
[ INFO] [1591944956.100243666, 1661396775.076756992]: waitForService: Service [/depth_manager/load_nodelet] is now available.
[ INFO] [1591944956.545617511, 1661396775.518832629]: Stereo is NOT SUPPORTED
[ INFO] [1591944956.545842654, 1661396775.518832629]: OpenGl version: 4.5 (GLSL 4.5).
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000614, using 1248 valid points
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000748, using 1444 valid points
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.001710, using 2303 valid points
[semantic_graph_slam_node-9] process has died [pid 27314, exit code -9, cmd /home/nrc/hd/workspace/ros/semantic_slam_ws/devel/lib/semantic_SLAM/semantic_graph_SLAM_node __name:=semantic_graph_slam_node __log:=/home/nrc/.ros/log/c2c4ddd8-ac79-11ea-96ed-8ca982ff1833/semantic_graph_slam_node-9.log].
log file: /home/nrc/.ros/log/c2c4ddd8-ac79-11ea-96ed-8ca982ff1833/semantic_graph_slam_node-9*.log

When it occurs

[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000614, using 1248 valid points

the program is still mapping , so I think the problem is not caused by 'pcl'.

I tried to run the launchfile seperately :

ROS_NAMESPACE=camera/color rosrun image_proc image_proc 
roslaunch semantic_SLAM shape.launch  
rosrun semantic_SLAM  semantic_graph_SLAM_node

But at the last step i got 'Segmentation fault :

add first landmark: 0
Segmentation fault (core dumped)

Do you have any idea about it? Thanks a lot !!!

opened by He-Rong 6

Dataset download failure problem

Hello, when I run the sample code, I always encounter network interruptions or unknown errors at the last moment when downloading the dataset entire_lab_3_rounds.bag. Can you provide a new way to download the bag?

opened by kycwx 2
Problemas de incompatibilidad de opencv en el bucket detector

Hola, he conseguido que ambos paquetes en conjunto (semantic slam y bucket detector) funciones bien en una distro de ubuntu virgen con ROS melodic, sin embargo, cuando migro al pc donde trabajo habitualmente y que tiene ya instaladas dependencias anteriores y demás me encuentro con estos errores referentes a opencv: Imagino que se deben a incompatibilidades entre versiones de opencv, podrías confirmarme esto último? Sería posible trabajar con una versión de opencv diferente? Gracias, un saludo!

opened by iandresolares 2

Releases(2.0.0)

2.0.0(Aug 10, 2021)

This release contains all the tf's published from map-->odom-->base_link
Source code(tar.gz)
Source code(zip)

Owner

Hriday Bavle

Postdoctoral Researcher at the University of Luxembourg. My research interests are VO/VIO, SLAM, Perception and Planning applied to Mobile Robots.

GitHub Repository

PyTorch reimplementation of REALM and ORQA

17 Aug 20, 2022

Implementation for "Domain-Specific Bias Filtering for Single Labeled Domain Generalization"

DSBF Introduction This repository contains the implementation code for paper: Domain-Specific Bias Filtering for Single Labeled Domain Generalization

7 Jan 05, 2023

A small library for doing fluid simulation with neural networks.

Neural Fluid Fields This is a small library for doing fluid simulation with neural fields. Check out our review paper, Neural Fields in Visual Computi

23 Jun 23, 2022

Convolutional Neural Network for Text Classification in Tensorflow

This code belongs to the "Implementing a CNN for Text Classification in Tensorflow" blog post. It is slightly simplified implementation of Kim's Convo

5.5k Jan 02, 2023

DAN: Unfolding the Alternating Optimization for Blind Super Resolution

DAN-Basd-on-Openmmlab DAN: Unfolding the Alternating Optimization for Blind Super Resolution We reproduce DAN via mmediting based on open-sourced code

72 Dec 13, 2022

Code for the paper Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations (AKBC 2021).

Relation Prediction as an Auxiliary Training Objective for Knowledge Base Completion This repo provides the code for the paper Relation Prediction as

85 Jan 02, 2023

PyTorch implementation of NIPS 2017 paper Dynamic Routing Between Capsules

Dynamic Routing Between Capsules - PyTorch implementation PyTorch implementation of NIPS 2017 paper Dynamic Routing Between Capsules from Sara Sabour,

475 Dec 24, 2022

Implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021).

[PDF] | [Slides] The official implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021 Long talk) Installation Inst

117 Dec 09, 2022

Graph Convolutional Neural Networks with Data-driven Graph Filter (GCNN-DDGF)

Graph Convolutional Gated Recurrent Neural Network (GCGRNN) Improved from Graph Convolutional Neural Networks with Data-driven Graph Filter (GCNN-DDGF

21 Dec 18, 2022

Probabilistic-Monocular-3D-Human-Pose-Estimation-with-Normalizing-Flows

Probabilistic-Monocular-3D-Human-Pose-Estimation-with-Normalizing-Flows This is the official implementation of the ICCV 2021 Paper "Probabilistic Mono

62 Nov 23, 2022

Automatic Attendance marker for LMS Practice School Division, BITS Pilani

LMS Attendance Marker Automatic script for lazy people to mark attendance on LMS for Practice School 1. Setup Add your LMS credentials and time slot t

3 Jun 12, 2021

This is the code of NeurIPS'21 paper "Towards Enabling Meta-Learning from Target Models".

ST This is the code of NeurIPS 2021 paper "Towards Enabling Meta-Learning from Target Models". If you use any content of this repo for your work, plea

7 Dec 06, 2022

SAT Project - The first project I had done at General Assembly, performed EDA, data cleaning and created data visualizations

Project 1: Standardized Test Analysis by Adam Klesc Overview This project covers: Basic statistics and probability Many Python programming concepts Pr

1 Jan 03, 2022

This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021.

GMPQ: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation This is the pytorch implementation for the paper: Generalizable Mix

18 Sep 02, 2022