For people with blindness or who have low vision, moving around independently is a daily challenge. Harvard researchers have developed a new tool that could help such people navigate more quickly and safely, using nothing but a device they likely already have – a smartphone.
Engineers at the John A. Paulson School of Engineering and Applied Sciences (SEAS) have invented a new smartphone app called Mobilio that combines machine learning; algorithms that streamline input from multiple sensors; and personalized audio cues to provide a blind or low-vision person turn-by-turn directions, path guidance, and obstacle avoidance. The research was led by Ph.D. student Raymond Liu in collaboration with Patrick Slade, assistant professor of bioengineering. The research is published in Nature Biomedical Engineering .
To start, the researchers surveyed over 100 blind or low-vision people to understand their needs. Respondents highlighted three essential capabilities for navigation technologies: reliable turn-by-turn directions, continuous guidance along sidewalks and paths, and obstacle detection and avoidance.
Existing tools like canes, guide dogs, and electronic travel aids address parts of these problems but rarely all three together, and many such systems are expensive and inaccessible. While GPS map software has turn-by-turn navigation, it is not accurate enough for a person who is blind or low-vision to safely use.
By contrast, most people already have smartphones. The Harvard team saw an opportunity to deliver a navigation aid through a standalone app that could one day be downloaded from an app store and runs entirely on the phone.
Mobilio utilizes the phone’s built-in sensors, including the camera; GPS; motion- and orientation-sensitive inertial measurement unit; and (if available) the LiDAR sensor. A central technological advance is a custom computer-vision model that analyzes the live camera feed from the perspective of a pedestrian to find walkable paths in the user’s environment.
“Our system uses the sensors in a smartphone, GPS information, and the actual motion of the person to create what’s essentially a small autonomous vehicle plan on how to navigate them from where they are to where they want to go,” Slade said.
Since most publicly available street-scene datasets are captured from cars, many standard models misclassify pedestrian-relevant infrastructure, explained first author Liu. To address this issue, Liu trained a semantic segmentation model specifically on pedestrian-view images so it could reliably recognize sidewalks, crosswalks, roads, and other key surfaces during walking.
“There’s a need for more data that comes from the perspective of humans walking along sidewalks, or walking outdoors,” Liu added.
Liu said a major design challenge was ensuring all the algorithms and models could run in real time on a standard smartphone, which has far less computing power than a laptop or specialized hardware.
Instead of just spoken prompts, Mobilio provides continuous audio cues in the form of beeps as the user walks. The beeps originate from a direction, such as the user’s left or right, to indicate steering, and the guidance is personalized in real time using human-in-the-loop optimization. As a person walks, the system measures how accurately they follow the audio cues and automatically adjusts the cues and pitch patterns to work best for that individual.
To evaluate the app, the team ran experiments with 14 volunteer participants recruited from Carroll Center for the Blind in Newton, Massachusetts. Each participant completed navigation tasks using a baseline condition of Google Maps turn-by-turn navigation plus a white mobiliity cane, and again with the Mobilio app plus the cane. They walked along a pre-planned outdoor path as well as an indoor course with obstacles that mimic common environmental challenges.
Compared to using Google Maps and a cane, participants using Mobilio and a cane completed the outdoor route about 13% faster, and reduced contact with obstacles in the indoor route by about 41%. In terms of reliability, or how consistently the system guided users to the destination without major errors, Mobilio performed comparably to a human guide.
Beyond the technical contribution, the project is informed by personal experience. Liu has an older brother who is blind, and supporting his sibling’s independent navigation strongly shaped the goals of the work.
The team emphasized that both quantitative and qualitative feedback from real users was a must.
“Having actual people test out your device is absolutely crucial because it’s impossible to predict how somebody is going to react to your device,” Liu said. “Especially within blindness and visual impairment, which is a huge, diverse set of impairments.”
Next, the researchers plan to test Mobilio in a wider range of realistic settings around the Boston area to further evaluate how well the system generalizes, and how it integrates with people’s everyday habits. The researchers are supported by the Harvard Grid Accelerator to help translate the technology and make it broadly accessible as a future product.
Funders of the project included the National Science Foundation Graduate Research Fellowship DGE-2140743, Harvard Grid Accelerator, Amazon Greater Boston Tech Initiative, Harvard Dean's Competitive Fund for Promising Scholarship, and the Kempner Institute.
The Harvard Office of Technology Development has protected the innovations associated with this research and is actively pursuing commercialization opportunities.
Nature Biomedical Engineering
Experimental study
People
Improving outdoor navigation for people with blindness using an AI-driven smartphone application and personalized audio guidance
24-Aug-2026