NeatNet: A model that can learn people’s tidying up preferences

As robots become increasingly advanced and affordable, more people could start introducing them into their homes. Many roboticists have thus been trying to develop systems that can effectively assist humans with house chores, such as cooking, cleaning and tidying up.

Researchers at The Robot Learning Lab at Imperial College London have recently developed NeatNet, an innovative machine-learning tool that could allow robots to tidy up home environments in ways that match a user’s individual preferences. This model, presented in a paper pre-published on arXiv, is based on a new variational autoencoder architecture with graph neural network layers.

“Everyone arranges their home in a unique and personal way, which is influenced by whether someone is left or right-handed, their aesthetic taste, their habits, and even their cultural background,” Dr. Edward Johns, one of the researchers who carried out the study, told TechXplore. “We developed a method for learning people’s preferences for how they like their home to be arranged, so that a robot could then tidy their home in a personalized way.”

NeatNet, the technique developed by Dr. Johns and his student Ivan Kapelyukh, allows robots to learn a user’s unique tidying up preferences, simply by observing how they arrange furniture and objects in their home. The robot can then use these preferences as guidance to tidy up the user’s home in ways that reflect his/her preferences.

“For example, suppose that you prefer your desk to be arranged in a compact way so that everything is easily reachable,” Dr. Johns said. “You might want the robot to learn this preference. After the robot tidies it, your desk will be arranged in a way that is convenient for you specifically.”

NeatNet draws inspiration from recommender systems, machine learning tools used by streaming platforms (e.g., Netflix, YouTube, Spotify) or other websites to recommend new content to users. Recommender systems work by learning a user’s preferences based on what content they watched, listened to, or accessed in the past.

“If a new film was watched by many users with preferences similar to yours, then Netflix might recommend that same film to you as well,” Dr. Johns explained. “This is how these methods make personalized recommendations.”

The new recommender-like tool for learning tidying preferences was based on Kapelyukh’s MEng thesis at Imperial College London, which was supervised by Dr. Johns. Kapelyukh, who is now a Ph.D. student at the university, presented this new paper together with his supervisor at this year’s Conference on Robot Learning (CoRL), which took place in London between 8 and 11 November.

“So far, we have presented our results using computer simulations of the robot and its environment, but in our future work, we plan to implement this on a real robot in the real world,” Dr. Johns said.

Essentially, NeatNet processes scenes that were arranged and tidied up by users. From these scenes, it learns a user’s tidying up preferences, which are represented as a sequence of numbers. Finally, it uses these numerical sequences to arrange any group of objects in a personalized way.

“Since the robot does not know in advance how many objects it will encounter in a scene, NeatNet uses a graph neural network to process scenes,” Dr. Johns said. “This means that rather than learning directly from images of the scene, it models a scene as a graph, where each object is a node (or a point), and all the nodes are connected together.”

Using a graph neural network, NeatNet is also able to learn the relationships between different objects. For instance, it could learn that a keyboard and mouse are usually placed side-by-side, or that cutlery is placed on the side of plates.

In addition to learning general object-to-object relations, NeatNet looks at individual user preferences. It could, for example, learn on what side of the plate users usually place their cutlery, as left and right-handed people might have different preferences.

Dr. Johns and Kapelyukh evaluated their technique in a series of experiments, using room arrangement examples created using a tidying simulator, which captured the preferences of 75 different users. In these tests, NeatNet consistently produced neat and personalized room arrangements.

“We found that tidying scenes in a personalized way was more pleasing for users than just tidying in the same way for everybody,” Dr. Johns said. “This was true even for simple scenes, and in real-world homes with hundreds of objects, there are many more options for how to arrange each room.”

When robots become more widespread, their ability to complete tasks in ways that are aligned with the preferences of individual users could be particularly valuable. NeatNet could thus prove to be particularly useful, particularly for enhancing the performance of home assistants and robots.

“Another interesting finding is that, although the neural network represented a user’s preferences as a sequence of numbers, we were still able to find some meaningful patterns,” Dr. Johns said. “For example, NeatNet decided to group left and right-handed users separately based on how they arranged a dinner table. This sheds some light on how the model works on the inside, which is often difficult to determine when using neural networks.”

NeatNet has so far only been tested in simulations, but it achieved highly promising results. The researchers are now conducting a follow-up study aimed at applying and evaluating their method on real robots.

“We will use cameras on the robot, the robot’s ‘eyes,’ to detect where the objects are in a room,” Dr. Johns said. “Additionally, we will consider how to ensure that the suggested arrangements are always safe under the laws of physics. For example, a plate should not tip over the edge of the table. We will also explore methods which take into account how long it would take the robot to complete this tidying. Rather than rearranging the whole house, for instance, we might like the robot to only tidy the few items which are significantly out of place.”

© 2021 Science X Network