How AI-powered robot guide dogs could change mobility for people with vision impairment
Gretchen Fisher Orr knows how powerful a guide dog can be for someone who is blind or has low vision. As supervisor of admissions and client services at the Fidelco Guide Dog Foundation, she helps match people with the organization’s German shepherds and works to create successful partnerships with like-minded organizations.
Guide dogs are a critical lifeline, helping their human handlers navigate life with greater independence. However, it can take as long as six months to two years to match someone with a guide dog since they’re in such high demand.
And placement isn’t always guaranteed — organizations want to ensure compatibility with temperament and lifestyle between the humans and dogs.
To address this gap, computer scientists at several universities are building a different kind of guide dog: one that walks with a metallic clank, runs on batteries and relies on software. These robotic guide dogs could offer a unique set of benefits. They may perform functions a live animal cannot, like describing their surroundings or answering questions about the route they’re about to take.
It’s important to note that robotic guide dogs are in the prototype stage and being tested with people who are blind or have low vision. Early progress shows they can navigate indoor environments and, in some cases, use their speaking ability to provide humans with helpful information.
Good robot
Donghyun Kim, PhD, an assistant professor at UMass Amherst’s Manning College of Information and Computer Sciences, directs the Dynamic and Autonomous Robotic Systems Laboratory. He wants robotic guide dogs to be helpful, natural partners to people who are blind or have low vision. A key to building the right type of robot is working with blind and visually impaired users to capture the nuances of using a guide dog.
“Instead of immediately focusing on navigation algorithms, we met guide-dog handlers and trainers at guide-dog training centers and walked with animal guide dogs to understand how they actually operate,” Dr. Kim said. Handlers must sense the dog’s movements and understand its behavior to interpret its meaning.
Orr tried an early prototype of one of the UMass Amherst robotic guide dogs. She echoed the importance of capturing the nuances of dog-human interaction. Human handlers rely on sensing the dog’s movements through the dog’s harness. The most subtle shifts can mean stopping, going or changing direction.
Extensive training is critical for a successful partnership between guide dog and human. As a guide dog mobility instructor and certified orientation and mobility specialist — a professional who teaches route planning, traffic judgment and cane technique — Orr works with both the dog and handler to help set everyone up for success.
The Fidelco Guide Dog Foundation’s German shepherds undergo comprehensive training and enrichment from birth to 24 months of age. When making a match with a blind applicant, a Fidelco trainer will provide intensive instruction, even staying in a nearby hotel if necessary, to be close enough during the early training period. The organization also provides support as needed throughout the dog’s placement.
The guide dog’s harness is a critical link to the dog. The human handler learns to sense the dog’s movements and rely on their canine companion for critical navigation information.
“As a dog navigates environmental changes and hazards, it is constantly making tiny adjustments in its movement, its pace, and the position of its body based on the movement of the human and the environment. The human is also responding and making their own movements. It is a closely coordinated dance,” Orr said.
Orr added that “trust is everything” when it comes to the dog/human relationship. This is built over time and shaped by environmental factors. If the handler has children, they’ll need a dog that takes to them well. It’s a relationship that will last up to a decade, Orr said, that needs to be a great personality match for long-term success.
One challenge she sees for developing robots is formulating that same sense of connection and intuition that a human handler and guide dog usually develop.
“The coordination is totally a dance between the dog and the human. With a robot dog, how do you create that level of coordination? It is more than somebody holding on to something and following at one pace,” she said.
Where a robotic guide dog, by comparison, can serve as a differentiator is through how its software opens new tools and its ability to communicate with the user. Dr. Kim said that the ideal state is a dog that harnesses what’s possible through artificial intelligence combined with assessing the environment.
“Because it is a robotic system, it can also do much more than provide navigation,” Dr. Kim said. “It can describe a scene, communicate and access the internet.”
Dr. Kim’s team is developing language capabilities to eventually integrate into the robotic guide dogs. His team is incorporating vision-language models that would enable the robot to communicate with its handler. The critical component is the human experience, ensuring that a robot benefits the user.
When robotic guide dogs talk back
At Binghamton University, another four-legged prototype isn’t just leading humans where they need to go. It’s having a dialogue on the way.
Together with his students, Shiqi Zhang, PhD, an associate professor of computer science at the School of Computing, Binghamton University, developed the university’s robotic guide dog system. One potential advantage the robotic guide dog offers over canine counterparts is its ability to engage in two-way conversation in English.
“The robot initiates the conversation and tells the human what it can do,” Dr. Zhang said. “The human says, ‘I want to go somewhere.’ The robot may ask some clarification questions, and then they go all the way to the goal.”
Dr. Zhang’s team has been testing this experience with legally blind participants — meaning visual acuity of 20/200 or worse in the better eye with the best possible correction, or a field of vision narrowed to 20 degrees or less — to help them navigate indoor spaces such as multi-room office environments. The robotic guide dog would ask users where they wished to go and offer potential routes to the destination. The user could then choose the route that sounded best and be guided by the dog along the one they selected.
Such a conversation is more natural, and doesn’t require the user to have memorized a series of commands. The Binghamton University system incorporates a large language model to enable the guide dogs to understand conversation and speech. Onboard cameras and computer vision technology help the robotic guide dogs identify people, paths, doors and obstacles.
Dr. Zhang describes the robotic guide dog’s process as three connected capabilities: perceiving the environment, understanding the surroundings and acting on a decision.
When guiding the user on the journey, the robotic dog narrates helpful details about the route and its surroundings.
“When they’re navigating a long corridor, the robot will say, ‘This is a long corridor, and going to the end of it needs one and a half minutes,’” Dr. Zhang said.
How to build a better robot (dog)
It will be some time before the robotic dogs are released from their digital kennel and put into the world for everyday use.
Dr. Zhang noted that several challenges must be addressed in both hardware and software. The Binghamton system is primarily developed for indoor environments. Dr. Zhang’s team is now working to increase its autonomy and prepare it to navigate longer distances indoors and outdoors.
Dr. Kim, of UMass Amherst, said his team continues to look for ways to make the robotic guide dogs quieter and more suited for everyday environments. Robotic guide dogs that are too loud can interfere with the human handler's ability to perceive the environment.
Physical sensitivity is also key to how a human interprets where their guide dog is leading them. Subtle motions in the harness tell someone where to follow or when to stop.
Dr. Kim said his team has developed locomotion, navigation, obstacle-avoidance and scene-description systems, but has tested them separately. Their next task is to combine those components into a single self-contained robot that an individual can use without extensive instructions. Dr. Kim also wants the robot dog to be light enough that someone could carry it if they need to for transport — his goal is a weight less than 22 pounds.
“We want the robot to be usable by an ordinary person without a lengthy explanation of what it does and what the person should or should not ask it to do,” Dr. Kim said.
Researchers are working on a critical concept to get the robotic guide dogs to exhibit “intelligent disobedience.” Guide dogs use this ability to disobey a handler’s instructions when safety is at risk. For example, if a handler wants to go forward into a busy intersection, a guide dog must be able to observe when there’s danger and disobey the command.
“That's something very important for guide dogs, biological or robotic, because people make mistakes,” Dr. Zhang said. “But at the same time, this kind of intelligent disobedience is quite a challenge, because it requires a robot to understand a few things very accurately.”
“One is human intent. The robot needs to understand what the human is commanding the robot to do. The robot needs to understand the environment to detect potential risks and hazards. And the robot needs to communicate the potential risks back to the human, because ultimately the decision is up to the human. So quite a few things together make this intelligent disobedience capability pretty challenging,” Dr. Zhang said.
Testing this capability is also challenging, as researchers don’t want to put human test subjects in danger to evaluate software performance. Intelligent disobedience remains a topic of interest, Dr. Zhang said, but more work is needed.
Creating a match between humans, dogs or one day, robots
Orr of the Fidelco Guide Dog Foundation tested an early version of the robotic guide dogs from UMass Amherst. She said it’s an interesting concept, and one that could serve people for whom a traditional guide dog isn’t a good fit. Some people may have allergies and aren’t able to have a dog. Others’ lifestyles may not work with the needs of caring for a guide dog.
“I do not think guide dogs are going out of fashion,” Orr said. “At this stage, if you need a working partner and dogs are right for you, that relationship and connection are not going to be replaced by a robot anytime soon.”
So much about connecting someone who wants a guide dog to a companion is finding the right fit. Orr said that getting someone a guide dog is far more complex than just going through a list and selecting the next person. When pairing someone with a guide dog, her organization considers the pace of the human and the dog.
The environment matters as well. A very active dog should be paired with someone with a strong level of fitness.
“Personality also matters. If we have a dog that prefers lots of affection and physical contact and a person who is not especially touch-oriented, or vice versa, that affects the match,” she said.
Building an AI-powered robotic guide dog is a topic of interest at other universities too. Similar work is underway at Georgia Tech. Researchers at the University of Glasgow have also developed and tested a prototype.
Each project takes a different approach to constructing the best robotic dog. They face several challenges in making the robot perform in different environments and be a responsive and reliable companion for the human it serves.
While their approach may differ, these projects share the goal of expanding mobility options for people who are blind or have low vision.
“I absolutely think there is a place for something like this, because people who cannot see need as many tools as they can have,” Orr said.
