Web-based Interactive Markers and Realtime 3D Point Cloud Streaming

Submitted by Anonymous on Wed, 05/29/2013 - 17:57

At Willow Garage, we believe in the power of the web enabling new applications in robotics. The web browser, taking advantage of emerging HTML5 web standards such as WebGL, websockets, and unified video streaming, can be a powerful and versatile frontend for accessing, operating and gathering information from robots. Today we are announcing a set of new Open Source libraries for 3D visualization and interaction, promoting the development of new web-based frontends for ROS systems.

Together, these libraries enable a web-based 3D teleoperation interface for the PR2, as shown in the accompanying video. This interface integrates an interactive robot model with real-time point cloud streaming. It is built using clients for Interactive Markers, TF and point cloud streaming that are part of the ros3d.js Javascript library. The connection between the web browser and ROS is established over websockets using rosbridge and ros.js.

On the robot side, dedicated ROS nodes throttle the transmission of TF information and provide precomputed transforms for the Interactive Markers client. The 3D meshes and textures needed for the robot model are provided by an HTTP file server. In addition, the depth and color information from the Kinect is jointly encoded into a compressed video stream that is provided via HTTP. In order to increase the dynamic range of the streamed depth image, it is split into two individual frames that encode the captured depth information from 0 to 3 meters and from 3 to 6 meters, respectively. Furthermore, compression artifacts are reduced by filling areas of unknown depth information with interpolated sample data. A binary mask is used to detect and omit these samples during decoding. Once this video stream is received by the web browser, it is assigned to a WebGL texture object which allows for fast rendering of the point cloud on the GPU. Here, a vertex shader is used to reassemble the depth and color data followed by generating a colored point cloud. In addition, a filter based on local depth variance is used to further reduce the impact of video compression distortion.

Using the ros3d.js javascript libraries for Interactive Markers, TF and point cloud streaming, we have created a web-based teleoperation interface for the PR2 that can be run in any modern web browser without the need of installing additional software. The provided toolkit makes it easy for developers to port their existing ROS-based applications to the web.