Here is @barbalberto's post from issue #683 :
Ok, I see there is there is good will for improvement and I like it :+1: so here is the 'long answer', I hope you guys can read it all and provide feedback.
My interest in the OpenNI device came from issue #643 I'm in charge to work on.
I examined all drivers related to kinect-like devices (in IIT we need one in a new robot) and I recently implemented the same in the Gazebo simulator (https://github.com/robotology/gazebo-yarp-plugins/issues/100) so we can work on simulator and then go on the real robot.
What I found out is that there is no real 'standard' in yarp for this kind of devices, and the situation is even worse for pointclouds. So I'd like to make some order in this.
I found out there are 4 kinect-like yarp drivers but all of them lack of some key features or have some dis-advantage.
Please think of the following points as ways to improve the yarp-ish software related to this awesome depth cameras devices.
My observations are:
A) Driver in yarp are meant to be splitted into 2 different part (modules/class/object):
- Proper driver: this part reads the data from hardware (no yarp port!)
- Wrapper: this part uses an interface to read data from the former one and it send the data on the network through yarp ports.
The reason is to re-use the same wrapper with different hw-devices, i.e. we will have the same ports and same data, same rpc calls regardless of using kinect, xtion or other devices (ideally also a streovision sistem with 2 cameras and disparity) through abstraction layer. Obviously abstraction layer has to be comprehensive to cover small differencies that can arise between different hw producers.
This is what is done for motioncontrol, so we use the same controlBoardWrapper2 over CAN devices, ETH devices, simulators (iCub, gazebo, robotran), ethercat device and so on. This allows code to work on different robots by abstracting hw level.
All kinect-like yarp drivers I saw, do the 2 things (driver and wrappper) together in a single module and cannot be separated. This means, for example, there was no 'pure wrapper' I could use with the simulators, therefore I had to write a yet another one.
B) As follow from lack of an unified wrapper, each device opens it own ports and send data using is own format:
- OpenNI2 uses one port for
rgb
and one port for imageOf<PixelMono16>
which is unsigned int
- robotology/kinect-wrapper uses one port for
rgb
and one port for imageOf<PixelMono16>
which is unsigned
- yarp/modules/kinect/OpenNi(1) uses
ImageOf<PixelInt>
which is int32
with sign
- yarp/modules/kinect/freenect uses
PortablePair<yarp::sig::ImageOf<yarp::sig::PixelMono16>
this is a pair, so different type
This make the devices incompatible between each other even if they basically provide the same data, therefore software made to work with one driver cannot work with another one. The usage of an unified wrapper/interface will make possible to test the same software with different sensors or change the sensor for example from kinect to xtion effortlessly.
C) Each device is a bit customized in the way it works, for example both OpenNi2 and kinect-wrapper has a port for skeleton, but their definition of skeleton is not the same while the other 2 devices do not provide skeleton.
This again goes against the re-usage of the code.
Following comment from @kt10aan
The philosophy behind the driver is exactly that, of a driver: exposing the functionality and providing an interface to what is available on the physical device and just streaming the data
to which I agree, I think that since skeleton or pointclouds are not data provided by the sensor, but a software elaboration on top, this feature should be extracted and implemented on a separated module. Again, using this approach will allow user to test the same algorithm (library) for skeleton extraction on different hw or to use different skeleton extraction algorithms (libraries) on the same hw. Those modules could be run on a remote machine more powerful. If I have (and I will) have the sensor installed on board of the robot, maybe compiting skeleton extraction or point cloud generation on board could be too heavy.
Another advantage is to remove dependencies for compilation. I'm not able to install Nite2 on my machine and, at the end of the day, I don't need it either but I got compilation issues because it is required.
So, to cut a long story short, as discussed internally with @lornat75, @drdanz and others, the idea in the long run is to have a 'infrastructure' similar to the one we have currently for motor control with:
- 1 wrapper/client for all RGBD devices
- many driver as required: kinect1 - kinect2 - OpenNI(1) - OpenNi2 - ... sending data
rgb & depth
- One (or more) module doing skeleton extraction from images (hopefully with a compatible skeleton definition)
- One (or more) module generating pointclouds from images
What have been done so far:
- Interface for depth sensors: https://github.com/robotology/yarp/blob/master/src/libYARP_dev/include/yarp/dev/IDepthSensor.h
- Interface for RGBD sensor: https://github.com/robotology/yarp/blob/master/src/libYARP_dev/include/yarp/dev/IRGBDSensor.h
This one basically inherits from FrameGrabber and Depth sensor adding a method to get data in synch
- RGBD client/wrapper pair: https://github.com/robotology/yarp/tree/master/src/libYARP_dev/src/modules/RGBDSensorClient
https://github.com/robotology/yarp/tree/master/src/libYARP_dev/src/modules/RGBDSensorWrapper
Those devices are meant to work together to send/receive data and set/get configuration thorugh RPC calls
- Gazebo plugin for depth camera sensor
- Module converting images into ROS pointcloud. It'll create a yarp point cloud too, as soon as the yarp-ish point cloud type is finalized and implemented. ROS compatibility will still be granted as well. (This is not in yarp repo yet.)
The definition of the interface is tentative, so features can be changed or added. For example mirroring is not present in the interface right now. We could add it so a sw can remotely know how is the mirroring set and comply accordingly.
Anyway, right now the client/server just exchange the image streams, the RPC calls are not yet implemented.
Probably first step will be to create a new OpenNi2 compatible device to be used with the RGBDWrapper and then help migration toward it.
What do you guys think?
Since it is a piece of work, con you spare some time to help in this process?
I do believe the result will make things more usable clear, improve the reusage of code and share of result and modules.
Component: Library - YARP_dev Component: Devices Issue Type: Feat/Enh Req