In one past inter­dis­ci­pli­nary col­lab­o­ra­tion, Rupal Patel and Isabel Meirelles teamed up to develop a web-​​based tool that teaches chil­dren how to read with appro­priate expres­sion using visual cues.

Their newest project, funded by a Provost’s Tier 1 Inter­dis­ci­pli­nary Seed Grant, tar­gets a trickier pop­u­la­tion: teenagers who want to learn Eng­lish as a second language.

Ado­les­cents who are not native to Eng­lish often lag in reading com­pre­hen­sion because the prosody, which is the melody of speech, is dif­ferent in their lan­guage,” said Patel, an asso­ciate pro­fessor of speech lan­guage pathology and audi­ology in the Bouvé Col­lege of Health Sci­ences with a joint appoint­ment in the Col­lege of Com­puter and Infor­ma­tion Sci­ence. “We want to help them both to sound more fluent and non-​​native and also to help them with lan­guage comprehension.”

But the new task pre­sented a new set of visual chal­lenges, noted Meirelles, an asso­ciate pro­fessor of graphic design in the Col­lege of Arts, Media & Design. “We spent a lot of time trying to devise a struc­tural metaphor for how to engage people,” she said. “Because it’s a dif­fi­cult age group, we didn’t want them to feel like they were doing work even though it is work.”

In the end, Meirelles and Patel appealed to that uni­versal fea­ture of puberty: nar­cis­sism. “This is the period in a person’s life when they go from where the family is a unit to where they are the unit,” Patel explained.

A still from the pro­gram. Cour­tesy photo.

The pro­gram, she said, turns teenagers into tele­vi­sion stars. With an inter­face mod­eled after a tele­vi­sion set, users can enter any of a series of “recording stu­dios” and become any­thing from a sports­caster to a fashion critic.

Here’s how it works: Upon entering the room, a user hears a native speaker reading a script. Then, while looking at the visual cues for each sen­tence, the user records her­self reading it. Finally, she plays back her own recording.

Patel said an impor­tant aspect of the com­pu­ta­tional pro­gram is its scal­a­bility. “There isn’t an inven­tory of rules,” she explained. “It doesn’t see a ques­tion mark and say okay, for the ques­tion mark, raise your pitch, because in Eng­lish that’s not always true. The ques­tion mark itself doesn’t tell you enough.”

Instead, the pro­gram uses speech acoustics and actual speech data to render the visual cues of native-​​speaker record­ings. Even­tu­ally, the team hopes to intro­duce a voice-​​recognition tool, allowing the pro­gram to imme­di­ately render users’ record­ings so they can com­pare the visual rep­re­sen­ta­tion of their own speech with that of the native speaker.

Ulti­mately Patel and Meirelles view the pro­gram as a web appli­ca­tion fea­turing record­ings that could be shared among users.

First, how­ever, they will test the user inter­face in a proof of con­cept pilot study. This data will help the design team deter­mine which visual cues need more devel­op­ment before deploying the pro­gram in a larger effi­cacy study.