A team of researchers at Penn State has developed NaviSense, an AI-powered smartphone application designed to help visually impaired individuals locate objects in real time. The tool uses voice commands and combines audio and vibrational feedback to guide users to specific items in their environment. It was created after the team conducted interviews with visually impaired individuals to understand their real-world challenges and tailor the app’s features accordingly. Unlike many existing assistive tools that rely on preloaded object models or require human support, NaviSense leverages large-language models (LLMs) and vision-language models (VLMs) hosted on an external server. This allows the app to recognize objects dynamically based on spoken requests without needing prior data storage, offering greater flexibility and efficiency. The system works by scanning the environment, filtering out irrelevant objects based on user input, and asking follow-up questions if the request is unclear. It also tracks the user’s hand movements through the phone’s motion sensors, providing real-time feedback on the object’s location relative to the hand—such as left, right, up, or down—helping users reach for items more accurately. During testing, 12 participants compared NaviSense to two commercial assistive tools. Results showed that NaviSense significantly reduced the time needed to find objects and improved detection accuracy. Users also reported a more intuitive and satisfying experience, with one noting the app’s directional cues made it feel like “bullseye, boom, you got it.” The team presented NaviSense at the Association for Computing Machinery’s SIGACCESS ASSETS '25 conference, where it won the Best Audience Choice Poster Award. The research was published in the Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility. While the current version shows strong promise, the researchers are working to improve power efficiency to reduce battery drain and enhance the performance of the underlying AI models. They believe the technology is close to commercial release and are committed to refining it based on user feedback and prior prototypes to ensure it meets the needs of the visually impaired community.