The AdMiRe project involves partners with different backgrounds and expertise. The Interactive Technologies Group (UPF-GTI) in Universitat Pompeu Fabra, specializes in real-time 3D rendering technologies and web-based 3D graphics editor applications.
In recent years, within the EU H2020 project HDR4EU, where the coordinator of AdMiRe, BRAINSTORM, has been a partner, UPF-GTI has been working on developing web based tools to support the introduction and use of High Dynamic Range (HDR) images in low-end environments. Within the AdMiRe project, we plan to re-use some of these tools to guide the user through the teleporting steps which are part of one of the use cases of the project.
In particular, the Real-time Chroma Keying tools give users the possibility to mix real photos/video captured with chroma with the 3D scene, taking into account the HDR environments illumination in the colorization of the character. This is a complex process as 2D images do not have orientation nor depth information. To solve this some fake orientation information (normalmap) based on the color information of the image is generated, which although not being 100% accurate, can help to improve the quality of the image and integrate it better with the surroundings.
We have combined normal map estimation with chroma-keying to re illuminate an image. The goal is to merge a 2D source (e.g. from a video call with the mighty Thor) with a 3D HDR environment. #webgl #javascript pic.twitter.com/hDyIYYxsfY
— GTI UPF – Interactive Technologies Group (@GTI_UPF) June 2, 2020
This could be improved within AdMiRe with the depth information available from some cameras, or pre-processing with algorithms with better quality results. We plan to explore client-side background extraction and HDR-based relighting in order to improve the user experience. The relighting involves imposing one illumination set from a certain place blending with the second place illumination or virtual lighting. These technologies should improve the blending of the captured user image with the illumination of the TV stage or virtual set.
The whole range of open tools, some of them implemented as graphs to facilitate re-use, and proposed file formats to deal with HDR images developed by UPF-GTI within the HDR4EU project can be visited as interactive demos or GitHub resources.
Node-based programming
UPF-GTI (Interactive Technologies Group)
Leave A Comment