By Stephanie Steinberg
Daily Staff Reporter  On  February 3rd, 2009

 

For most students, a laptop or a college-ruled notebook is enough to get through most lecture classes at the University. For hearing impaired students, the process is much more challenging. They need both a translator to transcribe the lecture and then extra time to read through the notes once class is over.

Using a cell phone, two University students have a plan to completely change the way hearing impaired students attend class.  Jason Gilbert and Judy Yu, graduate students in the College of Engineering's Department of Atmospheric, Oceanic and Space Sciences, have developed a system that translates sign language into speech using cell phone technology. They say that their invention, Mobile Sign Language Systems (MSigns), will revolutionize communication for the hearing impaired. "The idea behind it was to create some software that would take spoken English on any handheld device like a smart phone and translate it into a video of the sign language like an interpreter you can carry in your hand," Gilbert said. "This will be really useful for deaf and hard-of-hearing people who needed to communicate on the spot and didn't have a professional interpreter with them at the time."

 

Gilbert and Yu began working on MSigns in January 2006. They entered their idea in a program at the Digital Media Commons in the Duderstadt Center, which awards cash grants to students who submit project ideas that incorporate digital media.
Gilbert, who is fluent in sign language and Yu, who has computer-programming experience, collaborated to develop the project.  As an undergraduate student at Brigham Young University, Gilbert worked as a sign language translator to help pay his tuition bills. He said many deaf and hearing-impaired people told him they preferred sign language to written speech because it was easier to understand.

Gilbert realized it was possible to create an application for a smart phone, which would translate speech into a video image of sign language.  "We'd like it to be something you can use on your phone like more along the lines of a Google app, or an iPhone app or an add-on program to an install phone plan - or even a service you could use on the Internet," he said. While a few products exist that can translate speech to sign language, they are expensive - including the iCommunicator, which costs $6,500.  "It's out of the range of the everyday person, and that's who we want to bring this to," he said.

 

Though the MSign's technology isn't currently compatible with cell phones, it's designed to work with PC computers. Gilbert and Yu said they plan to develop a model the public can use within the next month and distribute the product by next fall.  In a later version of the technology, Yu said that using a camera phone, the goal is to develop the product to translate sign language into words.  Currently, the program can only translate English into sign language, but Yu said she intends to add more languages. "Sign language, like spoken language, is different in every country so we'd have to basically redo the whole process with another set of vocabulary," Yu said. "It's certainly something to think of but (English speakers) are by far the largest sign language community right now."

To film their current prototype, Gilbert and Yu used the Motion Capture Studio at the University's 3D laboratory to record the videos.
"We really want to do a motion capture version of sign language rather than just video taping a human," he said. "We think there's a lot we can do with the digital data in the long run." So far, deaf and hearing-impaired high school students who attended a College of Engineering summer program have been involved in the testing of MSigns. "They were excited to see it," Yu said. "It wasn't the perfect solution because it only does one way right now, but they were glad to see there was something being developed that was targeted for them because that doesn't happen very often."

 

Yu said she hoped MSigns will have a positive impact on the education and employment prospects for the hearing impaired.  "Transcribers, when they take notes for the students, it's still pretty difficult for hearing impaired students to interact inside a classroom or with students," she said. "We want to fill that gap so they can more fully integrate into society."