ALL.VP

ALL.VP: AI-Powered Lighting in Live Virtual Production

Funder: Innovate UK, through the BridgeAI programme (Collaborative R&D, ref. 10102363)
Partners: dock10 (lead), University of York, 2LE Media
Dates: April 2024 to March 2025

In green-screen virtual production, light does not travel between the real performers and the virtual set. Shadows, reflections and virtual light falling on actors therefore have to be faked by hand, which cannot be done in real time for live or “as-live” TV. ALL.VP used computer vision and generative AI, trained on synthesised green-screen data, to model how light passes between the physical and virtual parts of a scene.

The project produced AI “decomposition” tools. They separate green-screen footage into 3D shape, material properties and direct and ambient lighting, so performers can be realistically relit to match a virtual environment. The tools fit into existing green-screen workflows and were tested in demonstrator productions with dock10 and 2LE Media.

The project built on an earlier XR Network+ proof of concept with dock10. Its research continues in the spin-out PXLD AI.

Coverage: Broadcast ยท Innovate UK Business Connect case study