Starting With the IJCAL Review Controversy, Why Are Top Conferences Always Controversial?
Recently, the review results of IJCAI 2019 have been criticized. We conducted an in-depth analysis to see if IJCAI is as bad as many netizens say. In addition, in order to improve the efficiency of paper review, China has been trying to use AI tools to assist in review.
The IJCAI 2019 paper acceptance results have recently sparked a wave of complaints on the Internet.
On May 9, the top conference in the field of artificial intelligence IJCAI 2019 announced the results of paper acceptance: this year, a total of 4752 Effectively submitted papers, finally included 850 The acceptance rate is 17.9%, which is lower than the average 20% admission rate in previous years.
After the results were announced, many contributors were dissatisfied. The authors whose papers were rejected posted their experiences online, questioning the review, scoring and rebuttal processes.
In response to this, Sarit Kraus, the program chair of IJCAI 2019, published an open letter on May 10. She said in the letter:
I understand everyone's doubts about the review process, fairness, and randomness. I have also received many rejection letters. In this review, I have tried my best to reduce the randomness of the review process, reduce the number of papers received by each PC and SPC, and increase the review time.
However, with a total of 4,752 papers submitted to this conference, the review task is extremely difficult. I think the concept of rigorous review of academic conferences needs to be rethought and innovated. IJCAI will also listen to everyone's ideas and build a better journal.
Some scholars believe that this response is quite pertinent and understandable and acceptable. After all, after so many years, there should be more innovation.
After the anger subsides, we need to analyze IJCAI calmly and objectively. Is it really "getting worse and worse" as some authors complain?
Be objective and listen to multiple voices
We should not listen to only one side of the story in any dispute. Overall, IJCAI was wronged this time.
1. There is a lot of complaints at home, but the response abroad is mediocre
First, the areas with the heaviest resentment are concentrated in the domestic question-and-answer community "Zhihu", while no related questions were seen on foreign platforms such as Twitter, Reddit, and Quaro.
A user said on Weibo that there is a clique culture in the domestic academic circle, where reviewers will give high scores to authors they know, and may give less objective and fair evaluations to authors they have no relationship with. In addition, this year I heard for the first time that AAAI and IJCAI checked reviewer interest conflicts, and the results showed that all the highly suspected reviewers in 90% were Chinese.
Professor Wan Xiaojun of Peking University also mentioned on Weibo: "It seems that some authors are using connections to influence the results of paper reviews... I don't know when this bad trend started."
From this, we can speculate that perhaps it is precisely because of my own bad habits that the reviewers have become biased against Chinese people and thus failed to provide objective reviews?
2. If you win, it’s good, and if you don’t win, it’s bad?
Secondly, regarding 「How to view the IJCAI 2019 acceptance results」The comments were not entirely one-sided in their condemnation of IJCAI. In general, most of the complaints came from authors whose papers were not accepted.
One user said:
“Before submitting to IJCAI, you should be prepared for random review. The review results are not the same this year. In addition, because IJCAI has too many paper topics and is not focused enough, reviewers may not understand. Therefore,Contributors should submit to conferences that are more professional in their research field.In today’s world of paper inflation, it is better to work hard to write good articles. ”
This comment received more than 80 likes. Other netizens also responded:
“If your paper is not accepted, you may blame the review for the poor results, but why don’t you think about whether your work is good enough?”
An author whose paper was accepted said:
“It’s still sad to see the results of my hard work being belittled. Objectively speaking, the quality of my own review is still quite good, and the requirements are very strict. And the reviewers I met are also quite responsible.”
Tsinghua University professor Liu Zhiyuan responded to him in the comments:
"After all, there are only a few unreliable reviewers. Don't worry too much about it."
Another accepted author also gave his opinion on the complainers:
In addition, some netizens said that this situation is caused by many people saying that IJCAI and AAAI are too weak, but at the same time submitting their own papers that are obviously not up to the level of top A-level conferences.
According to the data, in recent years, the number of submissions and acceptances to IJCAI has been on the rise, and the number of accepted papers this year is even higher than last year.Added 37%,Reached an all-time high.However, in contrast, this yearThe admission rate is the lowest in recent years.For the first time, this is lower than the average admission rate of 20% in previous years.
Reference data (IJCAI paper acceptance and acceptance rate in the past 4 years):
In 2015, we received 1996 submissions, 575 of which were accepted, with an acceptance rate of 28.8%;
In 2016, a total of 2294 submissions were received, with an acceptance rate lower than 25%;
In 2017, a total of 2,540 submissions were received, 660 were accepted, and the acceptance rate was 25.9%. Among these 2,540 paper submissions, the largest number came from China, accounting for 37% of the total number of submissions; Europe and the United States each accounted for 18%;
In 2018, a total of 3,470 submissions were received, 710 papers were accepted, and the acceptance rate was 20.5%. Among the accepted papers, 325 authors were from China, accounting for 46% of the total, far ahead of all other countries and regions.
Perhaps, as the netizen mentioned above said, too many researchers are not sure about the quality of their papers, but authors submit them with a gamble mentality.This has resulted in a waste of IJCAI review resources.This is also one of the reasons for the increase in review bugs.
Therefore, it is reasonable to criticize the results of this acceptance rationally, but it is not good to complain blindly. Both reviewers and contributors should reflect on this.
There is no top conference that is not criticized
In fact, IJCAI is not the only top conference that has been criticized. ICML, AAAI, NeurIPS They all had the same "experience".Being criticized has become a routine.
No, just today, NLP conference ACL The results have just been released, and they have attracted a lot of criticism. ICML 2019 Twitter users were also furious about the review results.He mocked her severely.
Therefore, facing the situation that top conferences in the AI field are generally criticized, we should spend more time to find our own shortcomings and write better papers.
Professor Zhou Zhihua has previously commented on various conferences in the field of AI. Here is his evaluation of IJCAI for your reference:
IJCAI: The best comprehensive conference on AI, started in 1969, held every two years, in odd-numbered years. Because AI is so huge, although each session can basically record more than 100 papers (now more than 200 papers), there are only a few papers in each field. For such a large field as Machine Learning and Computer Vision, there are only about 10 papers each time, so it is very difficult.
However, the acceptance rate is not too low, basically around 20%, because insiders will weigh the weight of the application, and if there is no hope, don't waste the reviewer's time.
Recently, there has been a flood of papers submitted to international conferences from mainland China, and because there are few research groups in China that can review papers themselves, many conferences have complained that the low-quality papers from China have seriously hampered the efficiency of PCs. Under these circumstances, it is estimated that the acceptance rate of international conferences will drop in the next few years.
IJCAI is the leading conference in AI
In terms of seniority, IJCAI can be regarded as a veteran among many top conferences in the field of AI.
Looking at the top international conferences in the field of artificial intelligence,IJCAI is one of the oldest conferences.The first session was held in the United States in 1969, and has been held every two years since then, until 2015, when it was changed to an annual session. 50th birthday.
IJCAI is recognized by the China Computer Federation (CCF) as one of the recommended international academic conferences. Category A meetings,It has always held a very authoritative position in the field of artificial intelligence. Its admission standards are always relatively strict, generally maintained at around 20%.
At a glance, the list of winners of IJCAI's highest honor, the "Excellence in Research Award", is mostly made up of well-known names in the industry. For example:
In 1985, John McCarthy, the father of artificial intelligence, won the 1971 Tuling Award. (Interestingly, IJCAI also has an award named after him, which is intended to recognize mid-career researchers.)
In 2005, Geoffrey E. Hinton, the father of neural networks and the winner of the 2018 Graph Award.
In 2016, Michael I. Jordan, a member of three American academies and a leader in artificial intelligence.
If you are interested, you can search for the detailed list of previous winners. They are all the admirable AI gods.
In addition, the sponsors of the conference can also reflect its level to a certain extent. From the sponsors of IJCAI this year, there are many giant technology companies, including Baidu, Huawei, Alibaba, Tencent, etc. in China, and IBM Research AI, SONY, BOSCH, HITACHI and Microsoft abroad. The lineup is still very strong.
Solution: Use AI to review AI papers
The discussion about the IJCAI 2019 admission results has been on the Zhihu hot list, and some people lamented: It seems that it is reallyThe Era of AI for All.
In recent years, China has made rapid progress in the field of AI. Among the growing number of papers in various AI conferences, a large part comes from domestic contributions. In the past 20 years, China has produced a total of 100,000 papers in the field of artificial intelligence. 369,000 articles,The output is located atNo.1 in the world,The United States ranks second, 3.8 times that of the third-place United Kingdom.
It is no wonder that IJCAI complains that the huge amount of papers reduces the efficiency of PCs.
In response to the current situation, researchers have introduced an efficient and unbiased AI Grant Reviewer.
Yesterday, an article published in Nature stated,National Natural Science Foundation of China (NSFC)An AI tool is being trialled that selects researchers to review grant applications, making the process more accessible.More efficient, faster and fairer.
The system will utilizeNatural Language ProcessingCollect online scientific literature databases and scientists' personal web pages to collect detailed information about potential reviewers' publications or research projects. Use semantic analysis of text to compare grant applications and identify the best match for reviewers.
Some researchers say the NSFC's approach is world-leading, but they remain skeptical about whether AI can really improve the process.
More top conferences are using AI
last month,Norwegian Research CouncilWe began using natural language processing techniques to group approximately 3,000 research proposals and match them with the best review panels.
Swiss-based academic publisher Frontiers(Frontiers) Through AI tools AIRA(Artificial Intelligence Review Assistant) assists reviewers and editors to improve efficiency.
The AIRA system is built from in-house custom algorithms as well as industry-leading tools such as Google, CrossRef's iThenticate, and Editage's Ada.
AIRA currently carries out two key peer review tasks:Quality control and reviewer identification.Its algorithm quickly and accurately evaluates submitted manuscripts based on a set of quality indicators, including text overlap, language, the presence of human images, and other ethical considerations. Manuscripts that meet the established quality threshold will be passed to the editor, while those with any potential problems will be sent to the review team for further investigation.
We hope that when AI tools join the paper review team, reviewers will be able to save more time from other links, so that they can carefully review our papers and give each contributor a satisfactory response.
IJCAI 2019 will be held in Macau, China from August 10 to 16 this year. This is the second time it will be held in China after the 2013 Beijing conference.
There are still 3 months to the opening of the conference, but it is already so lively and it will probably be very interesting. So, do you want to attend?