Apps to interact with people who are deaf and cannot speak

At work, teams may have people who are deaf. Jay, my teammate, is deaf and cannot speak.

While Jay uses the Indian sign language to talk to those who understand it, he uses an instant messenger to talk to the rest of us. This led me to explore options for better interaction between us.

(I was not sure which term to use and turned to the internet. The term "deaf" is not considered derogatory and is not to be grouped with other offensive terms that denote impairment. The term "hearing impaired" is not specific to people who cannot hear at all - they do not consider it appropriate. But if you find the usage offensive, I apologise sincerely and request you to leave a comment).

Google Translate and ListNote

Most of my colleagues use Android, so I didn't look at apps for iOS.

The Google Translate app can be set to translate from English to English. It understands Indian accent, but not that well. Translate sends the voice over a data/wi-fi connection to Google's servers, where it is converted to text and sent back to Jay. Translate stops listening to speech when it senses a pause, making it an inconvenient option. Translate may have been just intended to translate phrases.

ListNote for Android does not stop listening when it senses a pause. But just like Google Translate, it wasn't that good with Indian English.

Things are looking up

Apps like Transcense and MotionSavvy are changing the landscape - these are being developed especially for people who cannot hear or speak.

Both companies are run mainly by qualified and motivated people who are deaf themselves or have family members who are deaf. Both are raising funds from the public on the indiegogo web site (crowdfunding).

Transcense

Transcense is a two-way communication tool for Jay. Its Android app is in beta testing. See the video below (Turn on captions in the video. I have also described it below with additional information).

In this beta stage, Jay can buy one of the "perks" i.e. plans listed on the indiegogo site. Usually, these are cheaper in the beta period and cost more when the product is in generally released later.

Jay could opt for the US$60 perk to get the Transcense app and become a beta tester (others can buy it as a gift for Jay too). After Jay installs the app, he can use it to invite people to join a conversation. The app sends a request to the mobile devices of his colleagues. They can just tap a button to accept and join the conversation.

When other people speak, their speech is transmitted via their mobile's mic to Jay's Transcense app. The app converts it to text and displays it on Jay's mobile (if you would like to know the technical details, the speech is actually sent over an internet connection to a server. The server converts speech to text and sends it back; With a good connection, all this happens in a second).

Now, not only can Jay clearly follow who says what, he can also type his response in Transcense, which converts text to speech. As far as I know, there is no other tool that can help Jay to participate in a group discussion like this.

Transcense is working on Indian English as well. They can help Jay and his team of deaf and hearing people to get a good success rate in translating Indian English to text (and text to speech).

The Infinity plan (unlimited data) would cost US$29/month - other limited data plans might come cheaper, though I could not see any right now. Beta testers will get a discount on this monthly rate.

MotionSavvy's UNI

MotionSavvy's UNI also allows effective two-way communication. The difference from Trancense is that instead of typing a reply, Jay can use American Sign Language . UNI converts sign language to speech. See this video for a clear explanation (it's a silent one with captions).

UNI is a bundle of software and hardware. It consists of:

• A regular Dell tablet
• A special case for the tablet that contains a motion sensor to detect sign language (sensor is also called a controller)
• The UNI app

The UNI app does two things. When Jay's sits in front of it and uses sign language, UNI translates it to speech. Secondly, when the presenter replies, it converts his speech to text.

Signs can also be added to a dictionary. The dictionary can then be shared among users. This implies that support for Indo-Pak Sign Language can also be added by users themselves. It is not clear how difficult or easy this is.

UNI does not need a data connection and so can be used anywhere. UNI is likely to be available for both Windows and Android platforms in 2015 and cost US$19.99/month. Unfortunately, it may support Indian English only by 2017.

Dragon Naturally Speaking

Dragon Naturally Speaking (DNS) is Windows software that can be used on the desktop or laptop.

DNS understands Indian English. The presenter's speech is captured and converted to text and displayed in a Windows application like Word. It is amazing to see the high success rate on YouTube video, recorded by individuals (video-1, video-2).

Speech-to-text software work best with good quality audio, so one of the following is recommended:

• Buy a good quality bluetooth headset and ask the speaker to wear it. Get a hands-free headset so that the presenter can move about freely - for example, to write on the board. This is the best approach for most situations.
• If the presenter is seated and will not move around, Jay can sit next to him so that his laptop mic can pick up the speech well.
• If your mobile can connect to your office wi-fi network, install Dragon's free iOS or Android app on it. This will connect your mobile to the Dragon Windows app. Now you can talk on your mobile and the audio will be passed on to Dragon, which will convert it to text.

There is another video from the maker of Dragon, Nuance, on improving accuracy.

The limitation in DNS is that there is no way for Jay to reply - he has to fall back on existing options or try other text to speech software. I could not find any reference to a text to speech feature in DNS.

If you cannot use DNS on the laptop, DNS has an app for iOS. It needs an internet connection to work. The system requirements list OS 4.0 - this might mean iOS 4.0. India and Indian English are not listed in the product support page - wonder what this implies.

Conclusion

Unfortunately, I left Jay's organization by the time this was written. Hopefully Jay and his colleagues can dig more into the subject. Unless other solutions come to notice, my suggestion would be this:

• DNS can be used till a two-way solution becomes available for Indian English. That solution must be affordable in the Indian context too.
• DNS needs high quality audio and hence may only be suitable for situations where one person speaks at a time (single speaker or speakers take turns presenting etc.).
• Transcense's solution might be the best where there are many speakers.
• It supports Indian English - test and help to improve its accuracy.
• Check how good the round trip time is.
• UNI may be a good alternative for some as it provides another way of two-way communication.
• Check if UNI works when multiple speakers are involved, as in a meeting.
• Keep track of it to know when they support Indian English.
• Find out how easy it is add support for Indian Sign Language.