Researchers have successfully used a brain implant in conjunction with AI to decode and communicate the thoughts of a bilingual person who is unable to speak.

This system decodes in real time what the patient is trying to say in either Spanish or English, simply by reading his brain activity. This remarkable achievement represents a significant leap forward in brain-computer interface (BCI) technology.

Brain to Auto-translate

The study centered around a single patient named Pancho, who suffered a stroke at a young age that left him paralyzed and unable to form clear speech. The man can only moan or grunt, but articulating even the simplest words is essentially impossible.

Partnering with a team at the University of California, San Francisco led by Dr. Edward Chang, Pancho underwent brain surgery to implant electrodes that recorded his neural activity.

While an earlier version of this technology allowed Pancho to communicate in English, this new iteration incorporates an AI system specifically designed to decipher his thoughts in both Spanish (his native language) and English, which he learned much later in life during his twenties.

The AI system works by analyzing the distinct neural patterns generated by Pancho’s attempts to speak. The training part was the most challenging. Pancho had to struggle to say nearly 200 words from a list given to him by the researchers. Although he couldn’t speak, the mental effort while he concentrated on articulating a word formed a distinct neural pattern that could be read by the electrodes implanted in his brain and later interpreted by the AI.

The AI system, comprised of separate modules for Spanish and English, assigns probabilities to potential words based on the neural activity and their logical sequence within a sentence. For instance, “I am” is much more likely to appear in this sequence than “I not”. The final output displays the sentence with the highest overall probability score on a screen, allowing Pancho to communicate effectively in either language.

“Speech decoding has primarily been shown for monolinguals but half the world is bilingual with each language contributing to a person’s personality and worldview,” Chang’s research group said on X. “There is a need to develop decoders that let bilinguals communicate with both languages.”

During one experiment, the modules were able to distinguish between English and Spanish on a first-word basis with 88% accuracy. The correct sentence was decoded with an accuracy of 75%. According to the researchers, Pancho can now use this system to form phrases (albeit slowly) and have unscripted conversations.

This research not only offers a promising avenue for restoring speech to individuals with communication disabilities but also sheds light on the brain’s language processing mechanisms. Unlike what previous research had suggested, the new findings show that Spanish and English utilized largely the same cortical area in Pancho’s brain. This means there may be a shared neural foundation for bilingualism, challenging previous assumptions about distinct brain regions for different languages.

While the study involved a single participant, the researchers are optimistic about the technology’s potential for wider application. Future steps include incorporating additional participants and exploring languages with vastly different structures, such as Mandarin or Japanese. Ultimately, the goal is to refine this BCI technology to enable natural and unrestricted communication for individuals who have lost their ability to speak.

The findings appeared in the journal Nature Biomedical Engineering.