ACCRUE (Nvidia, KAIST): a robot that reuses what it has learnt reduces its actual training time by between 31 and 68.5 per cent

This continuous reinforcement learning framework transfers a light-weight agent and a light-weight critic from one assembly task to the next, whilst retaining almost all the knowledge gained from previous tasks.

Training a robot by reinforcement directly on real hardware is expensive in terms of machine time, and each new task often starts from scratch. Researchers from Nvidia and KAIST (Changyeon Kim, Kimin Lee, Yashraj Narang, Yijie Guo) propose ACCRUE, a method that is based on a robotic foundation model and maintains a small actor network and a critical network from one task to the next. The first provides already acquired reflexes, the second an estimate of the value of the actions, which accelerates the learning of the next task.

Learning without forgetting

In order not to lose previous skills, the critic is adjusted only on the data of the task in progress, while the actor is brought back to his old behaviors thanks to a distillation over a few preserved episodes. On assembly sequences rich in contact, in simulation as on a real robot, the interaction required drops from 31 to 68.5% on AutoMate tasks and from 38.7 to 42.9% on NIST tasks, with 98.3% average success maintained on six tasks already learned (118 trials out of 120).