Why Did LeCun’s Younger Brother Bengio Win the Award Together With His Predecessor?
Scenario description:Bengio's research in deep learning has promoted the advancement of machine translation, and his achievements ultimately earned him the 2018 Turing Award.
Keywords:Deep Learning Machine Translation Turing Award
In 1988, Yoshua Bengio, a doctoral student majoring in computer engineering at McGill University in Canada, probably did not expect that more than 30 years later, he would win the Turing Award together with his idol Geoffrey Hinton.
Born in France, Bengio grew up in Montreal, a French-speaking part of Canada. He was drawn to science fiction since childhood, such as Philip K. Dick’s Do Androids Dream of Electric Sheep, which was the novel that impressed him the most. It tells the story of a sentient robot created by a large corporation that eventually turned into a villain due to environmental influences.
When Bengio read one of Hinton's papers, he felt like he was electrocuted because he found the feeling of the science fiction stories he loved so much in his childhood, and it helped him find his own research direction.
After obtaining a postdoctoral degree from MIT, Bengio joined Bell Labs and became a member of LeCun's team. This was when his relationship with the Turing Award officially began.
Before Bengio stepped in, machine translation was just a makeshift
When Bengio chose his research field, he probably had no idea how big an impact he would have. It all started with IBM's first translation machine.
This translator was invented in the 1980s, when the mainstream technology was rule-based machine translation (RBMT). The most common approach was to translate word for word directly from a dictionary, and then add syntactic rules to correct it. However, the results were disappointing because they looked very "stupid", such as:
For this slogan, if we don’t consider the context, the results of manual translation are all different: “Shall we be chickens?” “We have the right to be chickens”! “We are the right half of the chicken”… Let alone rule-based machine translation.
For example, KFC’s slogan:"We Do Chicken Right", the rule-based machine translation result is:"We do chicken right",or,"We are a chicken couple"…
Therefore, by the 1980s this method of translation had disappeared.
Later, corpus-based machine translation was developed, which is divided into statistical-based (SBMT) and example-based machine translation (EBMT) methods. In layman's terms, it is a sentence extraction model. When you enter a sentence to be translated, it will search for similar sentences in the bilingual corpus and then replace the different vocabulary translations.
For example:
I gave Xiao Ming a pen.
I gave Li Ming an apple.
For such sentences, you can extract their similar parts and directly replace the words in the different places.
However, this method has high requirements on the bilingual corpus, the division of phrase fragment granularity, the alignment of example pairs, and the reorganization of fragments, and there are many problems. Therefore, there was still a big gap between machine translation and human translation at that time.
Attention mechanism, reforming machine translation
Over the past half century, the development of machine translation has gone through many twists and turns, but there has been no significant improvement.
Five or six years ago, Google Translate was still based on phrase-based statistical machine translation (SMT), and SMT has always been regarded as the most advanced machine translation method. However, for many people, machine translation is just a "makeshift" method.
It wasn't until 2014 that machine learning finally achieved a historic breakthrough thanks to neural networks.
And this is thanks to Yoshua Bengio.
In 2001, Yoshua Bengio et al. published a landmark paper at NIPS (renamed NeurIPS in November last year)."A Neural Probabilistic Language Model", using high-dimensional word vectors to represent natural language. His team also introduced the attention mechanism, which made a breakthrough in machine translation and became an important technology for deep learning to process sequences.
In 2014, Yoshua Bengio's paper「Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation」In 2011, he laid the foundation for the basic architecture of deep learning technology for machine translation. He mainly used sequence-based recurrent neural networks (RNNs) to allow the machine to automatically capture the word features between sentences and then automatically write the translation results into another language.
As soon as this paper came out, Google was overjoyed. Soon, with sufficient supply of gunpowder and the blessing of the gods, the paper was published on ArXiv.org in September 2016."Google`s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation".
At the same time, Google officially announced that all statistical machine translations would be removed from the shelves, and neural network machine translation would take over and become the absolute mainstream of modern machine translation.
Google announced that after about 27 years, machine translation has officially transitioned from the 1989 IBM machine translation model (PBMT, phrase-based machine translation) to the neural network machine translation model, which has increased the translation accuracy to 87%. Google claims that in multiple language pairs such as "English-French", "English-Chinese", and "English-Spanish", the error rate has been reduced by 60% compared with the previous statistical machine translation system.
This accuracy is already very close to that of ordinary people. MIT TR reported that it is "almost indistinguishable from that of humans" and "excellently demonstrates the power of neural network machine translation."
This news caused a stir in the technology circle at the time, and Google's Neural Machine Translation System (GNMT) became a major milestone in the history of machine translation. It must be mentioned that GNMT uses the Neural Machine Translation (NMT) technology pioneered by Yoshua Bengio and his deep learning team at the University of Montreal, Canada.
Therefore, it can be said that Yoshua Bengio has promoted the advancement of natural language understanding and processing technology, and the research he participated in was later incorporated into many machine translation and artificial intelligence assistant products.
After NMT, a Hundred Schools of Thought
In a 2016 interview, Yoshua Bengio talked about past research that made him proud."To give you a relatively recent example, there have been some recent advances in machine translation, much of which is due to the work of our research group, Neural Machine Translation, which was developed about two years ago and is now being used in labs around the world. It is the most advanced machine translation technology and the first time that neural networks have made a major breakthrough in machine translation. Previously, neural networks have made great breakthroughs in areas such as speech recognition and computer vision, but machine translation and natural language processing are still in their infancy. So this is an achievement that I think we can be proud of."
Based on NMT technology, various Internet technology companies have also taken new initiatives. In October 2017, Google launched a new hardware product, the Pixel Buds headset, which integrates Google Assistant, real-time translation and other functions, and supports real-time translation in 40 languages.
In the following years, domestic companies such as NetEase Youdao, iFLYTEK, and Sogou have also launched translation machine products, and the technology behind them is inseparable from NMT.
In March 2018, researchers at Microsoft Research Asia’s Redmond Research Institute announced that they had added a neural network machine translation architecture.Dual learningas well asDeliberation NetworksThe machine translation system developed by us has achieved a level comparable to that of human translation on the Chinese-English translation test set newstest2017, a general news report test set.
Imagine if neural network translation had not been proposed, we might still be stuck in the stage where large sections of translation are full of grammatical errors and the results are "terrible". And the super neural AI editor might also face the risk of being fired because of poor English and poor translation...
So, thanks to Bengio, thanks to neural machine translation, the Turing Award is well deserved.