Project idea

The idea of Neodroid is to create a reality-ready robot brain in virtual reality. We specifically focus on creating a robot brain capable of humanoid visual-motor ability. Visual-motor ability is the integration between visual perception and motor skills. More specifically, it is the ability to perform constructive tasks integrating both visual perception and motor skills. The motivation behind Neodroid is to enable robots to assist humans in performing such tasks.

Humans have the only known visual-motor function that we truly understand through experience. Even so, we have not been able to reproduce human-level visual-motor function in a robot, despite the super-human non-visual motor skills in robots. Humans learn through a relatively slow process that unfolds over years, from the baby stage, through childhood and into adulthood. In Neodroid, we seek to develop humanoid visual-motor function in an accelerated learning environment, so that a robot can learn in days or weeks what may take a human years to learn.

The perfect accelerated learning environment is virtual reality (VR), where time can be sped up, and robots can move and learn faster and safer than in reality. The vision behind Neodroid is to teach robots how to perform visual-motor tasks, by training their brains in virtual reality. This is similar in philosophy to how Neo in the movie “The Matrix” learnt kung fu over the course of several hours within a simulator – in just a few seconds in the real world. Similarly, in Neodroid we will create a robot brain by training it – using deep learning – in virtual reality, much faster and safer than possible in the real world.

The essence of Neodroid is shown in Figure 1. The ‘baby’ phase is shown to the left in the picture, and involves growing the basic visual-motor ability of the brain in virtual reality, so that it can represent the connections between visual perception and motion. Similar to the baby phase in humans, where the baby brain transitions from a blank brain – tabula rasa – to a brain capable of representing the basic structures of reality. There is no human teacher present in the baby phase – in which environment-assisted learning takes place. Environment-assisted learning implies learning purely by observation of the environment. In Figure 1 this is illustrated by the virtual robot just observing images of the falling strawberries together with environment-provided information on the underlying 3D position and pose of each strawberry. The ‘school’ phase follows, in which a human enters virtual reality and demonstrates the visual-motor associations and motion patterns that define different visual-motor tasks. In the school phase, the brain structure develops further and is capable of representing complex spatio-temporal sequences of visual-motor response patterns. After the school phase, the brain is transferred from the virtual to the real robot, so learnt visual-motor skills can be applied in the real world.