How Large Language Models Enable Home Robots to Fix Errors Autonomously

There are many reasons why home robots have struggled to achieve success since the advent of the Roomba. Factors such as pricing, practicality, form factors, and mapping issues have led to numerous failures in the sector. Even when these aspects are resolved, a significant challenge remains: how to manage a system when it inevitably makes a mistake.

This challenge extends to industrial applications as well, but large corporations typically have the resources to tackle problems as they arise. Conversely, we cannot expect everyday consumers to learn programming or hire experts every time an issue occurs. Fortunately, emerging research from MIT points to a promising application of large language models (LLMs) in the field of robotics.

A study scheduled for presentation at the International Conference on Learning Representations (ICLR) in May aims to incorporate a degree of “common sense” into the error-correction process for robots.

“It turns out that robots are excellent mimics,” the research highlights. “However, unless engineers specifically program robots to adapt to every potential bump or nudge, they may not know how to handle these instances, aside from restarting the entire task.”

Traditionally, when a robot faces a challenge, it relies on its pre-programmed instructions before requiring human assistance. This is especially problematic in unstructured environments like homes, where varying conditions can disrupt a robot's functionality.

The study’s researchers observe that while imitation learning—learning a task by observing others—is popular in home robotics, it often falls short when dealing with the myriad of minor environmental changes that can hinder normal operation. This frequently requires the robot to revert to the initial task. The new research tackles this issue by dividing tasks into smaller subsets rather than treating them as a continuous sequence.

This is where large language models become significant, as they eliminate the need for programmers to label and categorize each sub-action manually.

“LLMs can articulate the steps of a task in natural language. A human's continuous demonstration serves as a physical representation of those steps,” explains graduate student Tsun-Hsuan Wang. “We wanted to bridge the two, allowing the robot to automatically recognize its stage in a task and replan or recover independently.”

The specific demonstration highlighted in the study involves training a robot to scoop marbles and pour them into a bowl—an easy, repeatable task for humans, but one that consists of multiple small steps for robots. LLMs assist in identifying and labeling these subtasks. During the demonstrations, researchers intentionally disrupted the process by nudging the robot or knocking marbles from its spoon. The system adapted by self-correcting these smaller tasks instead of starting from the beginning.

“With our method, when the robot makes mistakes, there's no need to rely on humans for programming or additional demonstrations on how to rectify them,” Wang adds.

This innovative approach offers a valuable solution to help prevent the total loss of control—one could say it helps keep one's marbles intact.