Hacking Timewarp to Fix Blurry Rotation in VR Games

28 Mar 2023, 20:12 UTC

VR requires a refresh rate of at least 90Hz in order to provide a convincing sensation of being "in the world" (aka immersion). Below this, there is too much lag in head movement. John Carmack, genius that he is, figured out that you can get away with a lower framerate as long as you interpolate up to 90Hz (or, more usually, 120Hz). For instance, you can render your game at 60Hz and do 2:1 interpolation to move it to 120Hz. Carmack figured out that actually it's mostly the rotation that needs fixing up in this interpolation (as opposed to parallax caused by non-rotational head movement). So, based on HMD sensor readings, the intermediate frame can be computed from the previous frame using a blit operation to essentially pan and rotate the previous frame into the predicted head alignment for the current frame. This only requires rendering a little further outside the view bounds, to accommodate possible rotation of the view rectangle into the non-rendered region. Peripheral vision is not perfect anyway, so when the head turns too quickly the system can just copy pixels at the boundary (e.g. using "texture mirror"). This system is commonly referred to as "timewarp", as it computes frames at a different time to the rendered frame, and for a static scene it works remarkably well.

Unfortunately, this kind of upsampling presents two major knock-on issues, caused by the fact that the game itself is running at a reduced framerate. Both these issues can immediately be seen on, for instance, "Horizon: Call of the Mountain" on PSVR2.

Firstly, when an animated character is being observed, the 60Hz update of the animation, combined with head movement, manifests as obvious "blurriness" of the moving character compared to the surrounding (quazi-120Hz) static environment. This is disconcerting, although not fatal.

But what in my opion is fatal is the behaviour of the user's panning/rotation control - e.g. the classic left/right stick push which rotates the player left or right in the world. Many VR games avoid this entirely by using "snap" rotation, but this has its own annoyances and once a user is familiar with VR they will often turn it off and have continuous rotation on the stick instead, which is what they are used to from 2D games. But this rotation is also at 60Hz, so it has the same blurry characteristic as character animation, except now the entire world is blurry. You are now causing actual discomfort. The sensation is not only of blurriness but it also quite juddering/jarring, and (judging from Facebook comments) can give the user the impression that the headset is malfunctioning and/or lower-quality than it really is.

BUT here's the thing ... when the user presses the stick, it begins rotating the camera around an axis at a fixed angular speed, or at least a perfectly forward-predictable angular speed. The angle the camera should be pointing is therefore perfectly known for intermediate frames. Therefore, it can be combined with the head rotation prediction during timewarp processing, and the user can effectively be rotating in the scene at 120Hz rather than at 60Hz. All this needs is for the timewarp system to have an input from the game that gives the angular velocity and acceleration of the camera in a given frame, and (perhaps) for the game to dynamically increase it's over-render region to accommodate the additional inter-frame rotation (in fact the game can conceivably bias its overdraw instead e.g. if the user is already panning left anyway, the game can overdraw less on the right and more on the left, but have the same amount of overdraw in total).