Prototype of real-time mixed reality previews of VR experiences using a depth camera.

A live VR experience at exhibitions, large events, demos, etc is typically observed through first person camera or a third person camera on a TV screen. A more complex setup is sometimes used when the floor and walls feature projections of the VR world (similar principle to CAVE - http://goo.gl/i59uKF), or a green-screen setup with real-time compositing (along the lines of the process described meticulously by Kert Gartner - http://goo.gl/bWuHvu and Northways Games for streaming live - http://goo.gl/aJGxeA). The latter two methods require changes to the physical environment - adding projections or green-screen setup - often a costly or visually unattractive option.

To provide a real-time mixed reality feed to an audience, retain the physical design of the space and avoid issues with foreground and background compositing, a RGB+Depth camera can be used to insert the user’s point cloud or mesh into the virtual space. Once the matrices of virtual and real cameras are matched, the virtual camera’s feed becomes a mixed reality feed.

This technique is similar to Google’s approach for recording artist performances in Tilt Brush for Virtual Art Sessions (https://g.co/VirtualArtSessions), using two Kinects for depth data.

We built a quick prototype using one of HTC Vive’s controllers for camera tracking, Microsoft Kinect v2, a Kinect v2 plugin (https://goo.gl/K5B7nQ) and Unity running on 2x machines. The server ran the actual VR scene and the client extracted data from the Kinect, placed the point cloud into the scene, resulting in a mixed reality feed. The depth threshold was altered dynamically based on the position of the headset.

In theory, any RGB+D camera set up to be tracked by Vive (through SteamVR Tracking - https://partner.steamgames.com/vrtracking/), or a RGB+D camera that can accurately ‘understand' the physical space (i.e. Hololens or Tango) can be used as a mixed reality camera.

The downside of this technique is the quality level of the point cloud (notice rough edges of the point cloud in the prototype). Another downside is the requirement from the VR game developer to implement mechanism of connecting a depth camera into the game and render the point cloud into a separate scene from the one the user is experiencing, so they don’t see themselves in the scene.

- Note on interference between Kinect's infrared and Vive lighthouses:
We received some pointers that Kinect's IR field may cause issues with Vive tracking. Oddly, we didn't experience any major issues. The only issue we have come across was the controller used by the user would 'slip' away in virtual space (for a second or so), but it happened only a couple of times over the course of development and as it is something we've encountered previously during other development, we assumed it was an issue of visibility of the controller by the lighthouse. It now makes sense that it was possibly the Kinect that was interfering with the tracking as the controller used for the Kinect position was never affected.

Music from Jukedeck - create your own at http://jukedeck.com
(AI-driven music compositing)

Alex Anpilogov, Timothy Burdairon & Dan Baczynski

FITCH London, July 2016


Leave a Reply