How Video Upscaling Could Dramatically Improve Your VR Headset

Your virtual reality (VR) headset graphics could soon look a lot better without having to buy a newer model. 

Meta has introduced a new feature, Super Resolution, designed to boost the image quality of VR apps and games on the Quest VR headset. The software upgrade can improve graphics without increasing hardware requirements. 

"VR applications aim to provide immersive experiences which rely on low motion-to-photon latency and high rendering quality," Tian Guo, a computer science professor at Worcester Polytechnic Institute, told Lifewire in an email interview. "Super-resolution techniques have the potential to impact both aspects by upscaling the content that will be directly consumed by VR users wearing head-mounted devices."

Better VR Graphics Through Software

Meta said that Super Resolution dramatically improves image quality over current video sharpening methods. 

"Super Resolution is up to 2x faster than the 'quality' sharpening algorithm currently used for Link Sharpening+ on Meta Quest 2 and Quest Pro," the company wrote on its web page aimed at developers. 

Squeezing every bit of processing power out of headsets is crucial. VR applications demand real-time rendering of complex 3D environments, which can strain hardware resources and affect visual fidelity, Vladimir Fomenko, the director of the tech firm Infatica, said in an email. 

"Using upscaling techniques like Meta's Super Resolution, VR content can achieve improved resolution and clarity without significantly increasing computational power," he added. "This translates to more immersive and detailed VR experiences, enabling users to feel fully present in the virtual world."

But trying to wring top-notch graphics from the current generation of VR headsets is tricky. One constraint is the unpredictable motion-to-photon latency when streaming VR video over the internet, Guo said. The latency can cause motion sickness in some VR users. 

A key challenge lies in capturing and rendering high-resolution, high-fidelity VR content, software developer Youssef El Achab said in an email to Lifewire.  Recording immersive 360-degree video with sufficient resolution and image quality can be technically demanding, leading to compromises in the final output. 

"Another limitation is the visual acuity within the headset itself," he added. "While VR headsets have made great strides in resolution and pixel density, there is still room for improvement to eliminate the screen-door effect, where users can perceive the gaps between pixels."

Guo said it is also difficult for headsets to deliver 360-degree videos and display the corresponding field-of-view based on VR user head movements with high frames per second. That's because 360-degree videos often require sending more data over the network than traditional video streaming, which can take longer if not done carefully. 

"Researchers have been looking at reducing the network time by trying to predict user head movement and only sending the cropped VR frames (i.e., smaller amount of data)," he added. 

AI Could Improve VR Graphics

The future looks bright for VR graphics. Companies, including Qualcomm, are working on improving rendering quality without impacting the frame rate so that "VR users could enjoy higher resolution VR videos that are less subject to variable network bandwidth," Guo said. 

Another promising area of research is using artificial intelligence to improve VR graphics. Meta is trying to use neural networks to achieve 16x upscaling for its headsets. 

"As AR/VR displays reach toward higher resolutions, faster frame rates, and enhanced photorealism, neural supersampling methods may be key for reproducing sharp details by inferring them from scene data, rather than directly rendering them," according to a blog post by Meta researcher Lei Xiao. "This work points toward a future for high-resolution VR that isn't just about the displays, but also the algorithms required to practically drive them."

Better graphics might help users who tend to feel sick when using VR. Recent research found that nausea in VR is caused by changes in how people process sensory information. Scientists at the University of Waterloo assessed users' perceptions of the vertical before and after playing two VR games, one high-intensity, and one low-intensity.

"Our findings suggest that the severity of a person's cybersickness is affected by how our senses adjust to the conflict between reality and virtual reality," researcher Michael Barnett-Cowan said in a news release. "This knowledge could be invaluable for developers and designers of VR experiences, enabling them to create more comfortable and enjoyable environments for users."