AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot

Xiao Wang, Lu Dong, Sahana Rangasrinivasan, Ifeoma Nwogu, Srirangaraj Setlur, Venugopal Govindaraju

Abstract

The social robot’s open API allows users to customize open-domain interactions. However, it remains inaccessible to those without programming experience. In this work, we introduce AutoMisty, the first multi- agent collaboration framework powered by large language models (LLMs), enabling the seamless generation of exe- cutable Misty robot code from natural language instructions. AutoMisty manages task decomposition, assignment, problem-solving, and result synthesis through four specialized agent modules. The Planner Agent formulates action plans, determines execution sequences, and assigns tasks to the appropriate agents. The Action, Touch, and Audiovisual Agents further refine actions and generate high-quality executable code. To enhance code reliability and structure, we optimize the original APIs to improve their interpretability for LLMs. Additionally, each agent incorporates a two-layer optimization mechanism: self-reflection for iterative refinement and human-in-the-loop for better alignment with user preferences. AutoMisty ensures a transparent reasoning process, allowing users to iteratively refine tasks through natural language feedback for precise execution. To evaluate AutoMisty’s effectiveness, we designed a benchmark task set spanning four levels of complexity and conducted experiments in a real Misty robot environment. Extensive evaluations demonstrate that AutoMisty not only consistently generates high-quality code but also enables precise code control, significantly outperforming direct reasoning with ChatGPT-4o and ChatGPT-o1.

Method

Video Presentation

Type: Conference Paper

International Conference on Intelligent Robots and Systems (IROS2025 under review)