HyperAIHyperAI

Command Palette

Search for a command to run...

China Achieves Real-Time Brain-to-Speech Decoding for Mandarin

Language is a cornerstone of human civilization and a primary means of communication. However, major brain disorders such as stroke and amyotrophic lateral sclerosis (ALS) can impair or completely disrupt a person’s ability to speak, severely affecting their quality of life and placing a significant burden on families and society. Brain-computer interface (BCI) technology offers a promising solution by restoring communication capabilities in individuals with language impairments, significantly improving their social integration and well-being. Globally, BCI systems have achieved notable progress in decoding and synthesizing English speech and text. However, research on decoding Chinese—a language with unique linguistic characteristics—has remained limited. Recently, scientists from the Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, and collaborating institutions have developed an implanted high-throughput flexible BCI system and a real-time neural decoding algorithm specifically tailored for the Chinese language. This breakthrough marks the first successful demonstration of real-time decoding and sentence synthesis using brain signals for spoken Chinese. Chinese differs fundamentally from English in structure and phonology. While English is a multi-syllabic, toneless language with a vast vocabulary—around 20,000 commonly used words—Chinese is a monosyllabic, tonal language. Remarkably, approximately 400 Chinese syllables combined with four tones can generate over 3,500 commonly used characters sufficient for everyday communication. Leveraging this linguistic efficiency, the research team identified these 400 syllables and four tones as stable intermediate decoding units. By decoding these units, the system can extrapolate to reconstruct full Chinese characters and sentences. The system captures both brain signals and corresponding speech production data simultaneously during sentence-generation tasks. Researchers employed a multi-stage real-time decoding pipeline, extracting high-frequency gamma-band (70–170 Hz) neural signals using a 50-millisecond sliding window. These signals were precisely aligned with the onset of speech, enabling a dual-stream decoder to simultaneously estimate the probabilities of syllables and tones. These outputs were then fused with a language model to select the most contextually appropriate sentence. After just nine days of training, the system achieved an average syllable decoding accuracy of 71.2% (excluding rare, unrecognized syllables), with a decoding latency of only 65 milliseconds per syllable. The real-time sentence decoding rate reached 49.6 characters per minute—sufficient for practical communication. Beyond decoding, the team integrated the BCI technology with artificial intelligence and embodied intelligence systems. Using the real-time Chinese decoding capability and a self-developed universal BCI operating system, participants successfully controlled a digital avatar and engaged in dialogue with large AI models. Furthermore, the decoded neural signals were translated into commands that enabled real-time control of a dexterous robotic hand, demonstrating advanced human-machine interaction. These findings were published in Science Advances. The research was supported by the Chinese Academy of Sciences and the Shanghai Municipal Government. The study presents a transformative step toward restoring communication for individuals with severe speech impairments and opens new avenues for integrating neural decoding with intelligent systems in healthcare and beyond.

Related Links

China Achieves Real-Time Brain-to-Speech Decoding for Mandarin | Trending Stories | HyperAI