Walkable 360° Video
Geometric AI allows you to step inside 360° VR photos and videos
Today’s virtual reality (VR) experiences fall short of the realism users expect. Current VR technology essentially falls into two categories: those based on 3D models with computer generated imagery (CGI); and those based on panoramic 360° imagery.
CGI-based experiences are interactive and support novel views but are far from photorealistic, are costly to build, and exhibit high data volumes and compute costs.
VR experiences based on 360° imagery are photorealistic, easy to acquire, exhibit low data volumes and compute costs, but are not interactive and do not support novel views.
Ideally we would like the best of both worlds, where VR experiences are both photorealistic and interactive.
360° cameras capture a single viewpoint only
While 360° content, i.e. panoramic photos and videos, are photorealistic by their nature — they are, after all, based on photography — they are acquired by a camera at a given point in space at each moment in time. At each moment the camera captures this fixed viewpoint only.
360° content can therefore only be viewed in VR from the original camera viewpoint. It is not possible to move around in the space to explore novel views that were not captured by the camera at the time of filming.
In 360° VR, no matter how you move in the real physical world, you remain frozen in the virtual world.
We would ideally like to unlock the user’s view from the camera viewpoint, thereby providing VR experiences that are photorealistic, being based on 360° imagery, while also interactive, allowing the user to move above. Artifical intelligence (AI) techniques can help in achieving this goal.
Geometric AI to the rescue
While AI has experienced a revolution in the past decade, most AI approaches are based on standard (Euclidean) data, such as common 2D images. However, 360° photos and video have geometric structure since they are defined on the 360° sphere.
To apply AI to 360° photos and videos, we need AI techniques defined specifically on the 360° sphere. The emerging field of geometric AI addresses precisely this challenge (see this article for an introduction to geometric AI for spherical 360° data and this article for an introduction to some of the underlying concepts).
Synthesising motion with geometric AI
Leveraging geometric AI techniques for 360° data, it is possible to realise VR experiences that are both photorealistic and interactive.
At Kagenova, a startup company working towards building the photorealistic metaverse of the future, we have developed copernic360 to deliver walkable 360° video.
copernic360 leverages geometric spherical AI techniques developed at Kagenova to estimate the 3D geometry representing the scene from a single 360° image (photo or video frame).
The copernic360 viewer system uses this geometry, along with the original 360° video or photo, to render a 3D textured representation of the scene. The user is then able to move about freely in the reconstructed scene. Not only does this greatly enhance the realism of the VR experience, it also eliminates the main cause of cybermotion sickness.
The embedded video below demonstrates the walkable 360° photos and videos realised by copernic360.
A photorealistic metaverse
By combining imagery with powerful AI techniques, it is possible to realise virtual experiences that are both photorealistic and interactive.
The walkable 360° video experiences delivered by copernic360 provide only a first step in this direction. The field of neural rendering is developing at pace and offers the potential to provide even more realistic experiences to power the photorealistic metaverse of the future — which we are actively pursuing at Kagenova!
In the meantime, copernic360 provides walkable 360° photos and videos that can be deployed in production today, to enhace existing 360° experiences, apps and platforms.
