Samsung researchers create AI which generates realistic 3D renders of video scenes
Three researchers at Samsung have created an AI which can generate realistic 3D renders of video scenes.
In a paper detailing the neural network behind the AI, the researchers explained the inefficient process of creating virtual scenes today:
“Creating virtual models of real scenes usually involves a lengthy pipeline of operations. Such modeling usually starts with a scanning process, where the photometric properties are captured using camera images and the raw scene geometry is captured using depth scanners or dense stereo matching.
The latter process usually provides noisy and incomplete point cloud that needs to be further processed by applying certain surface reconstruction and meshing approaches. Given the mesh, the texturing and material estimation processes determine the photometric properties of surface fragments and store them in the form of 2D parameterized maps, such as texture maps, bump maps, view-dependent textures, surface lightfields.
Finally, generating photorealistic views of the modeled scene involves computationally-heavy rendering process such as ray tracing and/or radiance transfer estimation.”
A video input is converted into points which represent the geometry of the scene. These geometry points are then rendered into computer graphics using a neural network, vastly speeding up the process of rendering a photorealistic 3D scene.
Here’s the result in a video of a 3D scene created by the AI:
Such a solution could one-day help game development, especially video game counterparts of movies that are already being filmed. Footage from a film set could provide a replica 3D environment for game developers to create interactive experiences in. Or, perhaps, you could relive events like your wedding day using just an old video and a VR headset.
Before such a point is reached, some advancements still need to be made. Current scenes cannot be altered and any large deviations from the original viewpoint results in artifacts. Still, it’s a fascinating early insight at what could be possible in a not-so-distant future.
You can read the full paper here or find the project’s Github page here.
Interested in hearing industry leaders discuss subjects like this and their use cases? Attend the co-located AI & Big Data Expo events with upcoming shows in Silicon Valley, London, and Amsterdam to learn more. Co-located with the IoT Tech Expo, Blockchain Expo, and Cyber Security & Cloud Expo.
