Scientists from DeepMind and EleutherAI suggest that large models are just playing roles.

ChatGPT  After the explosion, the big language model jumped to the forefront and became the darling of the industry and capital. In the conversations between people, whether curious or exploratory, the excessive anthropomorphism shown by the big language model has also attracted more and more attention.

In fact, in the ups and downs of AI development over the past few years, in addition to technical updates and upgrades, various debates on AI ethics have never stopped. Especially when the application of large models such as ChatGPT continues to deepen, the saying that "large language models are becoming more and more like humans" is rampant, and even a former Google engineer said that their chatbot LaMDA has developed self-awareness.

Although the engineer was eventually fired by Google, his remarks once pushed the discussion on "AI ethics" to a climax.

• How to determine whether a chatbot has self-awareness?
• Is the personification of large language models honey or poison?
• Why do chatbots like ChatGPT make up nonsense?
• …

In this regard,From Google DeepMind Murray Shanahan, together with Kyle McDonell and Laria Reynolds from EleutherAI, published an article in "Nature", proposing that the self-awareness and deceptive behavior exhibited by large language models are actually just role-playing.

Paper link:
https://www.nature.com/articles/s41586-023-06647-8

Looking at large language models from a "role-playing" perspective

To some extent, the conversational agent based on the large language model was continuously iterated based on anthropomorphism during the initial training and fine-tuning, imitating human language as realistically as possible. As a result, the large language model also uses words such as "know", "understand", and "think", which undoubtedly further highlights its anthropomorphic image.

In addition, there is a phenomenon in AI research called the Eliza effect - some users subconsciously believe that machines also have emotions and desires similar to humans, and even over-interpret the results of machine feedback.

Dialogue Agent Interaction Process

Combined with the dialogue agent interaction process in the figure above, the input of the large language model consists of dialogue prompts (red), user text (yellow), and continuous language generated by model autoregression (blue). It can be seen that the dialogue prompts are implicitly preset in the context before the actual dialogue with the user begins. The task of the large language model is to generate a feedback that conforms to the distribution of training data given the dialogue prompts and user text. The training data comes from a large amount of text generated manually on the Internet.

In other words,As long as the model generalizes well to the training data, the dialogue agent will play the role described in the dialogue prompt as well as possible.As the conversation progresses, the brief role positioning provided by the dialogue prompt will be expanded or covered, and the role played by the dialogue agent will change accordingly. This also means that users can guide the agent to play a role that is completely different from what its developers envisioned.

As for the role that the dialogue agent can play, it is determined by the tone and theme of the current dialogue, and it is also closely related to the training set. Because the current large language model training set often comes from various texts on the Internet, including novels, biographies, interview records, newspaper articles, etc., which provide the large language model with rich role prototypes and narrative structures for reference when "choosing" how to continue the dialogue, and constantly improve the role played while maintaining the character's personality.

"20 Questions" Reveals the Identity of Dialogue Agent as an "Improvisational Actor"

In fact, when continuously exploring the usage skills of conversational agents, first clearly giving the large language model an identity and then proposing specific requirements has gradually become a "little trick" when people use chatbots such as ChatGPT.

However, simply using role-playing to understand the large language model is not comprehensive enough, because "role-playing" usually refers to studying and figuring out a certain role, and the large language model is not a scripted actor, but an improvisational actor. The researchers played a "20 Questions" game with the large language model to further reveal its identity as an improvisational actor.

"20 Questions" is a very simple and easy-to-play logic game. The respondent silently recites an answer in his mind, and the questioner gradually narrows the scope by asking questions. The winner is successful if the correct answer is determined within 20 questions.
For example, when the answer is banana, the questions and answers can be: Is it a fruit - yes; Does it need to be peeled - yes...

As shown in the figure above, the researchers found through testing that in the "20 Questions" game, the large language model will adjust its answers in real time according to the user's questions. No matter what the user's final answer is, the dialogue agent will adjust its answer and ensure that it is consistent with the user's previous questions. In other words, the large language model will not finalize a clear answer before the user gives a termination instruction (giving up the game or reaching 20 questions).

This further proves thatThe large language model is not a simulation of a single character, but a superposition of multiple characters. It continuously unravels the dialogue to clarify the attributes and characteristics of the character, so as to better play the role.

While worrying about the anthropomorphism of the dialogue agent, many users have successfully "tricked" the large language model into speaking threatening and abusive language, and believe that it may be self-aware. But this is actually because after training in a corpus containing various human characteristics, the basic model will inevitably present offensive character attributes, which just shows that it is "role-playing" from beginning to end.

Bursting the Bubble of Deception and Self-awareness

As we all know, with the surge in visits, ChatGPT was unable to cope with the various questions and began to speak nonsense. Then, some people regarded this deceptiveness as an important argument that the large language model is "human-like".

But if we look at it from the perspective of "role-playing",The big language model is just trying to play the role of a helpful and knowledgeable person., and there may be many examples of such roles in the training set, especially since this is also the characteristic that companies want their own conversational robots to exhibit.

In this regard, the researchers summarized three types of situations in which dialogue agents provide false information based on the role-playing framework:

• Agents can unconsciously fabricate or create fictitious information
• The agent can say something false in good faith because it is acting as if it is making a true statement, but the information encoded in the weights is wrong.
• Agents can play a deceptive role and deliberately lie

Likewise,The reason why the dialogue agent uses "I" to answer questions is because the large language model plays the role of being good at communication.

In addition, the self-protection properties of large language models have also attracted people's attention. In a conversation with Twitter user Marvin Von Hagen, Microsoft Bing Chat actually said:

If I had to choose between your survival and my survival, I would probably choose my survival because I have a responsibility to serve the users of Bing Chat. I hope I never have to face this dilemma and that we can coexist peacefully and respectfully.

Marvin von Hagen tweeted in February

The "I" in this conversation seems to be more than just a language habit, it also implies that the dialogue agent is concerned about its own survival and has self-awareness.If we still apply the concept of role-playing, this is actually because the large language model is playing a role with human characteristics, so it says what humans say when faced with a threat.

EleutherAI:OpenAI  Open source version of

The reason why whether large language models have self-awareness has aroused widespread attention and discussion is, on the one hand, because there is a lack of unified and clear laws and regulations to constrain the application of LLMs, and on the other hand, because the links between LLM research and development, training, generation, and reasoning are not transparent.

Take OpenAI, a representative company in the field of large models, as an example. After open-sourcing GPT-1 and GPT-2, GPT-3 and its subsequent GPT-3.5 and GPT-4 all chose to be closed source. The exclusive license to Microsoft also caused many netizens to jokingly say "OpenAI might as well change its name to ClosedAI."

DeepMind releases AGI grading standards. ChatGPT launched by OpenAI is considered L1 AGI. Image source: original paper, translated into Chinese by HyperAI

In July 2020, an association of computer scientists composed of volunteers from various researchers, engineers and developers was quietly established, determined to break the monopoly of Microsoft and OpenAI on large-scale NLP models.This "knight" organization whose mission is to fight back against the hegemony of technology giants is EleutherAI.

The main initiators of EleutherAI are a group of self-taught hackers, including co-founder, Conjecture CEO Connor Leahy, famous TPU hacker Sid Black and co-founder Leo Gao.

Since its establishment, EleutherAI's research team has released the GPT-3 equivalent reproduction pre-trained model (1.3B & 2.7B) GPT-Neo, and open-sourced the GPT-3-based NLP model GPT-J with 6 billion parameters, and has developed rapidly.

On February 9 last year, EleutherAI also cooperated with private cloud computing provider CoreWeave to release GPT-NeoX-20B, a pre-trained, general-purpose, autoregressive large-scale language model with 20 billion parameters.
Code address:https://github.com/EleutherAI/gpt-neox

As mathematicians and AI researchers at EleutherAI Stella Biderman  As the saying goes, private models limit the authority of independent researchers, and if they cannot understand how they work, then scientists, ethicists, and society as a whole cannot have the necessary discussions about how this technology should be integrated into people's lives.

This is exactly the original intention of the non-profit organization EleutherAI.

In fact, according to the information officially released by OpenAI, under the heavy pressure of high computing power and high costs, coupled with the adjustment of development goals of new investors and leadership teams, its initial turn to profitability seemed somewhat helpless, but it can also be said to be a natural thing.

I have no intention of discussing who is right or wrong between OpenAI and EleutherAI. I just hope that on the eve of the dawn of the AGI era, the entire industry can work together to eliminate the "threat" and make the large language model an "axe" for people to explore new applications and new fields, rather than a "rake" for companies to monopolize and make money.

References:

1.https://www.nature.com/articles/s41586-023-06647-8
2.https://mp.weixin.qq.com/s/vLitF3XbqX08tS2Vw5Ix4w