NVIDIA Flora adds action descriptions to videos for robot learning
NVIDIA’s Flora examines videos of people or robots working and produces action times, descriptions and assessments of the result. A researcher could use footage of picking up a block to help prepare examples for robot learning. Its output can be wrong and is not intended to directly control a robot.
Video divided into action intervals, with descriptions for the actions and an assessment of the outcome.
View full-size imageWhy add descriptions to a video?
A learning program finds patterns in examples. Video alone does not always provide everything needed for robot training: a researcher also needs to know what happened and when. Flora-TempSeg finds action intervals, Flora-Captioner describes them and Flora-Rewarder evaluates quality or outcome. These annotations help prepare learning data; they are not an already learned robot skill.
What is actually available?
The Decoder covered Flora’s release on 25 October. NVIDIA’s repository provides sample code for running the models, rather than training code or its optimized production system. The licence permits non-commercial research and evaluation only. NVIDIA says outputs need checking on separate videos and must not be the sole safety monitor for a physical system.