New brain-computer interface lets paralyzed patients speak and gesture via a virtual avatar

A research group from the University of California, San Francisco, has created a brain-computer interface that can decode speech and gestural intent at the same time, enabling users to control a custom virtual avatar. Electrodes placed on the brain’s surface capture sensorimotor activity, which machine-learning algorithms convert into text and corresponding avatar motions in real time. In a pilot study, three severely paralyzed participants attempted phrases and gestures; two achieved 100 % accuracy in both domains across multiple trials, successfully making the avatar shrug, nod, or pump a fist while speaking.

The system was built using the Unreal Engine platform, and participants helped design the avatar’s appearance. Authors of the study, published in Nature Neuroscience, view this as an early step toward more expressive communication for individuals affected by stroke or motor-neuron disease, though larger trials are needed before clinical use. Stroke Association medical director Deb Lowe praised the innovation while noting its early stage.