When we communicate with each other, we use our whole bodies. Although spoken words are often seen as the most relevant part of human communication, we also emphasize and express context through movements, such as nodding and shaking our heads or accompanying our words with hand gestures.
Many neurodegenerative diseases can take away this complexity and uniqueness of human conversation by not only paralyzing the anatomy involved in speech but also affecting the rest of the body. Brain-computer interfaces (BCIs) are designed to help restore this ability, but so far, they have mostly excelled at translating either speech or movement, not both at the same time.
Researchers from the Department of Neurological Surgery at the University of California, San Francisco, have now found a way to combine both functions in a single brain implant. Their proof-of-concept device is the first to read brain activity and translate it into speech and movement simultaneously, as described in their paper published in Nature Neuroscience.
Challenges of Translating Speech and Movement
People who suffer a stroke or neurodegenerative diseases, such as amyotrophic lateral sclerosis (ALS), can develop problems with speech or nonverbal gestures, depending on which areas of the brain are affected. As a result, some people may struggle with or completely lose their ability to communicate with the world, significantly reducing their quality of life.
Explore All OptionsRecent advances in neurotechnology have made it possible to track brain activity and decode speech, facial movements, and even more sophisticated actions, such as controlling robotic limbs. However, most of these technologies focus on restoring one ability at a time.
Decoding several types of brain activity at once is more complicated. Brain activity expresses differently when movements are part of communication than when they are performed on their own. For example, waving while saying hello involves a different combination of signals than simply moving your hand. The new BCI is designed to tell these patterns apart.
Read More: Blood-Vessel-on-a-Chip Models Can Now Grow and Redirect New Capillaries
New Implant Covers a Larger Portion of the Brain
To test the system, researchers worked with two participants. The implant consisted of an array of 253 electrodes placed on the surface of the brain’s cortex, covering a large portion of the sensorimotor cortex. This allowed it to pick up signals linked to both speech and body movements.
One of the participants showed impaired speech and movement after a stroke in the brainstem. He silently attempted five phrases and movements, including waving, nodding, shaking his hands, and clapping. The second participant had ALS, which had led to progressive speech paralysis and motor impairments. He vocalized 10 phrases and imagined 10 gestures, including a shrug and thumbs-up, while keeping his body still apart from his facial muscles.
The researchers recorded brain activity as the participants attempted speech and gestures separately and together, like combining “hello” with a wave or “yes” with a nod. They then trained artificial intelligence to convert these neural signals into text, gestures, or both simultaneously.
From Brain Signals to Virtual Avatars
The researchers connected the BCI to a personalized full-body virtual avatar that appeared on a screen and could reproduce the participant’s intended movements.
During conversation tests, the participants responded to questions and comments in real time using phrases, gestures, or combinations they had practiced. The system achieved accuracy rates ranging from 75 to 100 percent.
There is still plenty of room for improvement. The speech decoding in this study was less accurate than what previous BCIs have achieved. The team plans to expand the system’s vocabulary and give users more control over individual body parts, including up to 15 joints in the hand, as explained in a statement.
For now, the system is a proof of concept, but it could be an important step toward BCIs that better capture the complexity of human communication.
Read More: Organ‑on‑a‑Chip Models Now Include the Cervix — Revealing New Clues About STIs
Article Sources
Our writers at Discovermagazine.com use peer-reviewed studies and high-quality sources for our articles, and our editors review for scientific accuracy and editorial standards. Review the sources used below for this article:
- This article references information from a study published in Nature Neuroscience: Simultaneous speech and gesture decoding for multimodal communication in paralysis
