New Tech Will Change How People Use Mobile Devices

By Lorena Anderson, UC Merced

March 9, 2020

Professor Arif (center) and two of his Ph.D. students, Gulnar Rakhmetulla (right) and Laxmi Pandey (left), have developed new tech for mobile users.

Two projects from a UC Merced engineering lab are making interactions with mobile devices easier and quicker, especially for blind people.

Computer science Professor Ahmed Arif and his students have developed a new way for blind people to be faster and more accurate when entering text, and a new way for all users to work with numbers.

The average typing speed for sighted, experienced people is 40 words per minute on a mobile device. But blind people using mobile keyboards can’t see what they are typing, Arif said, so their typing is slower and more error-prone.

A member of the Department of Computer Science and Engineering, Arif focuses on human-computer interactions. His lab group delves into such topics as speech recognition; artificial intelligence; virtual reality; technology for children; and assistive technologies.

One of his students was exploring how blind people enter text on mobile keyboards, and Arif was inspired to look to the community for research opportunities. He connected with the Center of Vision Enhancement (COVE) in Merced, a resource and learning center for the blind and low-sighted that offers training on technology use.

“But blind people are motivated to learn a new method because the common QWERTY keyboard is so ineffective for them,” Arif said.

See how chording works in this demonstration of Senorita.

Expert blind users on Braille-enabled mobile devices averaged 23.2 words per minute on mobile keyboards, according to a 2012 study published in the Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services.

“Most text entry solutions for blind people rely on their knowledge of Braille, but Braille literacy in this population is only 10 percent,” third-year graduate student Gulnar Rakhmetulla said. “Additionally, the findings from the 2017 National Health Interview Survey show that 26.9 million adult Americans reported experiencing vision loss.”

Also, Braille keyboards for mobile devices are expensive, Arif said.

“We wanted to find a better interaction with inclusive design, so we developed Senorita, motivated by the ‘design for all’ philosophy,” Rakhmetulla said.

Senorita is a virtual, chorded, two-thumb keyboard that features eight keys in a single row. Tapping a key enters the letter assigned to that key, while tapping two keys together — chording — enters the common letter between the keys.

So far, the chording keyboard has allowed test users to surpass their typing times by 32 percent after just a short period of practice. COVE clients said they found Senorita effective and playful and wanted to keep using it.

The current version is only available on limited devices in limited settings — not in any messaging apps. The next big step would be to make it generally available as a software keyboard so it could be selected on any mobile device as an input choice, Arif said.

“We will also explore whether user performance, experience and learning can be improved by adding tactile feedback, such as vibration,” Rakhmetulla said. The project will be presented at the prestigious Association for Computing Machinery Human-Computer Interaction (ACM CHI 2020) Conference on Human Factors in Computing Systems in Honolulu in late April.

See a video demonstration of predictive numbers.

The second project, predictive number entry and editing, is already an award winner and is getting notice within the field. Working with numbers on mobile devices requires the tedious cutting/copying and pasting between apps, Arif said.

Graduate students Laxmi Pandey and Azar Alizadeh, Arif and the lab came up with the idea to include predictive number entry and editing on the suggestion bars of mobile-device keyboards. Using deep learning, text-based querying and regular expression to suggest the most probable next numeric actions — along with word suggestions — significantly increases number entry and editing speed and accuracy.

“It reduces the number of actions needed per task, and also significantly reduces the time and effort needed to fix errors,” Arif said.

The researchers conducted two user studies, and almost all the participants found the method faster, more reliable and easier to use than the conventional method, and they wanted to keep using it on their mobile devices.

“We found that all of our participants were unhappy with the conventional method of entering and editing numbers using third-party apps,” Pandey said. “They all wanted a better approach that requires fewer actions.”

Pandey, a second-year Ph.D. student, received two awards from the Association for Computing Machinery (ACM) – an ACM’s Council on Women (ACM-W) scholarship and an ACM Special Interest Group on Information Retrieval (SIGIR) student travel grant. She has been asked to present the research at the prestigious ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2020) conference this month in Vancouver, B.C.

Pandey said the next step would be to make more personalized suggestions for users based on their habits, and she and the other researchers believe the project has some potential in medical technology. Existing number-entry methods on medical devices are not reliable, which can cause problems in drug-dosing and delivery.

“We want to investigate whether the proposed method can mitigate this issue by suggesting numbers and auto-corrections based on previous inputs and prescriptions,” Pandey said.