A Kinect Game for Gesture and Speech

Research Category: Engineering and Technology
Presenter: Pooja Jhaveri
Additional Authors: Steven Troetti, Prashant Naidu, Christopher Barton, Kenny Jean-Baptiste, Kevin Do, Danielle Levac, Kathryn Connaghan, Waleed Meleis
PI: Kathryn Connaghan, Waleed Meleis

Introduction: Cerebral Palsy (CP) is a disorder frequently characterized by impaired limb and speech motor control. Therapy to address these impairments typically involves repetition of motor and speech tasks over many sessions. A reward-based game application is therefore an ideal therapeutic tool to encourage exercise participation and maintain motivation.

Methods: We have developed a Unity-based, Kinect-enabled video game designed to provide physical and speech therapy to children with CP through an engaging, interactive virtual environment. The player assumes the role of a young spy-in-training and is given tasks and mini-games to progress through his/her training. Tasks and mini-games elicit expansive physical motions of the upper extremities, upright sitting posture, and speech projection with the aim of increasing speech volume and strengthening motor control. Data is collected and formatted on a webpage accessible to clinicians and researchers.

Results: While development is ongoing, the project currently consists of a game scene where a player is asked to capture an object and verbally describe it. Such tasks are designed to elicit increasing range of motion and speech intensity. The game platform sends task data to a server and these data are ultimately displayed on a web user interface.

Conclusions: Our next steps are to evaluate game usability with a small group of end-users. Following larger-scale trials, this platform will serve as a clinically relevant tool to encourage repetitive practice and collect clinical data to explore the links between expansive gestures and speech production.