0

I'm a researcher working on improving video quality in various contexts. For my current problem, I need to collect videos which have both RGB data as well as depth data. Video can be monocular or stereo. However, if there is egomotion, I'll need the camera pose (rotation and translation) as well. Is there any way to record all these information in any kind of games? Games can be on any platform: PC games, mobile games or VR games.

In short, I want to record gameplay but with additional information such as depth map and camera pose.

4
  • for some games probably, but for many probably not, what game in specific do you want to record?
    – Topcode
    Commented Mar 29, 2021 at 15:59
  • I'm flexible regarding that. First person shooter games or car racing/driving are preferred. Like, GTA V, Counter Strike, are good. Commented Mar 29, 2021 at 22:59
  • You might want to rephrase this question so it doesn't attract close votes on the basis that it's solliciting suggestions for games. "In a video game" instead of 'in any games' makes it a generic question instead of a shopping question. There will still be many possible answers, but only one correct answer is required to close the question as 'yes', whatever the game may be.
    – aphid
    Commented Mar 30, 2021 at 14:46
  • Hi @aphid, I didn't understand the change you suggested. I used the phrase "in any kind of games" to emphasize that I'm not restricted to a single game. If I change it to "in a video game", wouldn't it be confusing for a reader? I mean they may get confused as to which game I'm referring to. Commented Mar 31, 2021 at 8:51

1 Answer 1

2

Games with 3D cameras can certainly provide you with position information (X,Y,Z coordinates and Angle (yaw,pitch,roll) of the camera each frame). How this is stored (format) calculated and used to render the scene to the player will be different from game engine to game engine. Thus if you want to include multiple games in your research, you may have to transform your data to be in invariant units.

For example, would be a good candidate for a game that supports exposing this data to the player through the getpos console command, which you could program to run in the background in a private server context every frame, which gives you the necessary X,Y,Z,phi,psi data for the camera position and rotation.

Another good candidate is open-source games in general. Take a look at Super Tux Kart (Racing Game) or Xonotic (FPS). Since the source code is open, you can simply reprogram the game to e.g. output the information you want to a log file while it's running. Fork or Clone the game's github, modify the code to do what you want, and run your tests.

A full depth map is unlikely to be available for most games, because construcing one would essentially involve raytracing. Raytracing is very new1 and support has only very recently been added to GPUs. I don't know of many titles that fit your criteria. Certainly GTA V and CS:GO do not have the support, they're far too old. Most open-source titles lack the manpower to implement this as it's seen as nice-to-have.

However, ID software and NVidia did a demonstrative collaboration for the recently released GeForce-2000 series cards in which was given a Raytracing overhaul. The source code and modification is even publicly available, making it an even better fit.


Footnote

Open-source games are more conductive to research in general, because there is no corporate entity which might be interested in keeping the internal workings of a game secret, and which can go to great lengths to do so. In particular, the information you want can be useful to Cheat in various ways, so it's unlikely that any AAA game will expose it outside of a private server or Single-player context.

1: While the technique has been known and demonstrated for a long time, hardware capabilities on desktop computers haven't been sufficient to get real-time raytracing to an acceptable performance level for gaming at high resolutions to where it is a viable alternative to the traditional render model until quite recently.

2
  • 1
    this very well written response would be best served in gamedev.stackexchange.com I think.
    – Cole Busby
    Commented Mar 30, 2021 at 22:37
  • Thank you very much for the detailed answer. I also found something called as gamehook/scripthook which researchers have been using to generate required ground truth data for their research. I'll wait for another day before accepting your answer to see if I get any other answers. Commented Mar 31, 2021 at 8:52

Not the answer you're looking for? Browse other questions tagged .