Solving Cybersickness with AI
How geometric AI techniques can be used to alleviate cybersickness
Today’s 360° virtual reality (VR) experiences have great potential, allowing you to be transported anywhere in the world. They are photo-realistic and relatively easy and cheap to acquire — requiring only an off-the-shelf 360° camera — but they lack immersion and interactivity. Critically, this lack of interactivity can also induce cybersickness for many users.
3 DOF vs 6 DOF
While 360˚ VR experiences allow you to look around in all directions, with current technology you cannot move about in the virtual environment. For example, you can’t walk around or lean forward to take a closer look at something.
Since 360° content is captured from a camera at a given position, there is only one viewpoint available when viewing the experience in VR and so it’s not possible to move about freely to enjoy different views. No matter how you move in the real physical world, you remain frozen in the virtual world.
In other words, current 360° VR experiences support what is called 3 degree-of-freedom (3 DOF) rotational motion, where you can look around (i.e. rotate your head) but you cannot move. To allow you to move about in a virtual environment 6 DOF motion must be supported. 6 DOF motion includes 3 DOF from rotational motion (looking about) and 3 DOF from positional motion (moving about).
The latest standalone VR headsets, such as the Oculus Quest 2, are now opening up the VR market and making it more accessible for consumer and enterprise users alike, eliminating the need for a high-end computer to power a tethered VR device. These standalone devices now also support 6 DOF at the hardware level. However, content remains 3 DOF, which has lead to a disconnect between the capabilities of the latest hardware (6 DOF support) and content (3 DOF support only). This means 6 DOF 360° VR experiences cannot be delivered to users today.
Visual-vestibular conflict
While lack of support for 6 DOF motion in today’s 360° VR experiences limits immersion and realism, it also induces a greater problem: cybermotion sickness.
In everyday life the sensory systems of our body operate in harmony. Cybermotion sickness arises when the different sensory systems of the body are in conflict.
Visual-vestibular conflict is the most common cause of cybersickness. No one holds their head perfectly rigid when in VR and so some physical movement is inevitable. When you move physically your sense of balance, your vestibular system, experiences that motion. But if you don’t see yourself move through the scene when in a virtual environment, your visual system doesn’t experience the same motion. The disconnect been the visual and vestibular systems causes a sensory conflict that is experienced as nausea.
To learn more about cybersickness take a listen to the Voices of VR podcast, where Kent Bye interviews Jason Jerald, an Adjunct Professor at Duke University. Jason explains the various causes of cybersickness, highlighting visual-vestibular conflict as the most significant.
Synthesising movement in VR with AI
To eliminate visual-vestibular conflict in VR it is necessary to restore harmony between the visual and vestibular systems of the body. This can be achieved if your visual system experiences movement in VR to match physical motion experienced by the vestibular system. The problem is, when the 360° footage was taken the camera captured only one viewpoint of the scene at each moment in time.
Using artificial intelligence (AI) this problem can be solved by synthesising novel viewpoints of the scene that were never captured when shooting the 360° footage. By rendering these novel viewpoints on the fly as you move about, you get the sensation of moving within the virtual world. Technically speaking, synthesising movement in 360° VR environments using AI enhances the experience by providing 6 DOF motion — in a sense retro-fitting 3 DOF 360° content with 6 DOF support.
We have developed precisely such a system at Kagenova, a startup company developing geometric AI techniques for 360° photo and video content. Our copernic360 system achieves this effect through two sub-systems working together: a back-end AI-based cloud processing system; and a front-end viewer system.
The AI-based cloud system processes 360° VR video or photo content to recover an estimate of 3D geometry representing the scene. The viewer system uses the geometry computed by the AI system, along with the original 360° video or photo content, to render a 3D textured representation of the scene. The user is then able to move about in the reconstructed scene with full 6 DOF motion, where novel synthetic viewpoints of the scene are rendered and served to the user depending on their position in the scene.
Scientific study
While it’s clear intuitively that eliminating visual-vestibular conflict will alleviate cybersickness, it’s nevertheless important to test this scientifically. Recently, a team of experts in vestibular neuroscience at Royal Holloway University in London put this hypothesis to the test.
They devised an experiment to essentially perform an A-B test with a number of users in a highly controlled manner. Users experienced a 360° VR scene with 3 DOF using standard 360° VR technology and also with 6 DOF provided by Kagenova’s copernic360. In both scenarios a number of indicators of cybersickness were measured, including both qualitative (e.g. verbal user feedback) and quantitative (e.g. heart rate) indicators.
The results of the study have just been reported in an academic article submitted to Multisensory Research (preprint available here) and featured in New Scientist. The study shows that symptoms of cybersickness such as nausea were dramatically reduced when using Kagenova’s copernic360 6 DOF technology.
“The results confirmed a very effective reduction in cybersickness when using copernic360.” — Dr Elisa Ferré , Lead of the scientific study
Takeaway
While 360° VR can transport you anywhere in the world to enjoy, for example, virtual entertainment, tourism, cultural or educational experiences, today’s technology supports only 3 DOF motion, where you cannot move around in the virtual world. This limits realism and can induce cybersickness. Fortunately, recently developed geometric AI techniques tailored specifically to 360° photos and videos, realised in Kagenova’s copernic360 technology, can synthesise 6 DOF to allow you to move about in the virtual world, alleviating cybersickness and substantially enhancing realism.
References
[1] Iqra Arshad, Paulo De Mello, Martin Ender, Jason D. McEwen, Elisa R. Ferré, Reducing Cybersickness in 360-degree Virtual Reality (2021), arXiv:2103.03898
[2] Oliver J. Cobb, Christopher G. R. Wallis, Augustine N. Mavor-Parker, Augustin Marignier, Matthew A. Price, Mayeul d’Avezac, Jason D. McEwen, Efficient Generalized Spherical CNNs (2021), ICLR, arXiv:2010.11661
[3] Jason D. McEwen, Christopher G. R. Wallis, Augustine N. Mavor-Parker, Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs (2021), arXiv:2102.02828
[4] Matthew Sparkes, AI can stop the cybersickness some people get when using VR headsets, New Scientist, https://www.newscientist.com/article/2271020-ai-can-stop-the-cybersickness-some-people-get-when-using-vr-headsets/#ixzz6onYARkim
[5] copernic360, Walk into 360° VR Content, https://www.kagenova.com/products/copernic360/
[6] fourpiAI, Geometric AI for Spherical Data, https://www.kagenova.com/products/fourpiAI/
