Embodied Agents Meet Personalization: Exploring Memory Utilization for Personalized Assistance

Embodied agents empowered by large language models (LLMs) have shown strongperformance in household object rearrangement tasks. However, these tasksprimarily focus on single-turn interactions with simplified instructions, whichdo not truly reflect the challenges of providing meaningful assistance tousers. To provide personalized assistance, embodied agents must understand theunique semantics that users assign to the physical world (e.g., favorite cup,breakfast routine) by leveraging prior interaction history to interpretdynamic, real-world instructions. Yet, the effectiveness of embodied agents inutilizing memory for personalized assistance remains largely underexplored. Toaddress this gap, we present MEMENTO, a personalized embodied agent evaluationframework designed to comprehensively assess memory utilization capabilities toprovide personalized assistance. Our framework consists of a two-stage memoryevaluation process design that enables quantifying the impact of memoryutilization on task performance. This process enables the evaluation of agents'understanding of personalized knowledge in object rearrangement tasks byfocusing on its role in goal interpretation: (1) the ability to identify targetobjects based on personal meaning (object semantics), and (2) the ability toinfer object-location configurations from consistent user patterns, such asroutines (user patterns). Our experiments across various LLMs revealsignificant limitations in memory utilization, with even frontier models likeGPT-4o experiencing a 30.5% performance drop when required to referencemultiple memories, particularly in tasks involving user patterns. Thesefindings, along with our detailed analyses and case studies, provide valuableinsights for future research in developing more effective personalized embodiedagents. Project website: https://connoriginal.github.io/MEMENTO