Study Ranks Claude Sonnet, GPT, and Gemini Pro Top in Synthetic Data Generation Across Multiple Languages

RWS's TrainAI LLM Benchmarking Study Ranks Claude Sonnet, GPT, and Gemini Pro as Leaders in Synthetic Data Generation MAIDENHEAD, England--(BUSINESS WIRE)--A recent study by TrainAI, part of RWS Holdings plc, has ranked Claude Sonnet, GPT, and Gemini Pro as the top performers in synthetic data generation among large language models (LLMs). Unlike traditional benchmarks that rely on automated tests and focus on closed-ended questions, TrainAI’s study utilized human expert evaluators to assess the models' capacity to generate sentences and conversations, providing a comprehensive evaluation of their natural language processing (NLP) proficiency across multiple languages and tasks. The primary motivation behind the study was the observation that leading companies such as OpenAI, Anthropic, and Google may be facing data scarcity issues, which are crucial for training and fine-tuning their AI models. These companies are increasingly exploring the use of synthetic data—data generated by the models themselves rather than by humans—to overcome this challenge. To better understand the effectiveness of this approach, TrainAI designed a benchmarking study that evaluated nine popular LLMs on six different data generation tasks, spanning eight languages with varied levels of representation. The study involved generating 38,000 sentences and collecting 115,000 annotations and 250,000 ratings from 27 linguists worldwide. Each language was assessed by three native-speaking specialists, who rated the model outputs based on criteria such as grammar, naturalness, instruction adherence, creativity, speed, and cost. The results revealed that no single model outperformed all others consistently across languages and tasks, but Claude Sonnet, GPT, and Gemini Pro demonstrated superior performance in key areas. "AI is designed for human usage, so we decided to use human evaluators instead of AI systems to assess the models' performance," explained Vasagi Kothandapani, President of Enterprise Services at RWS. "Our findings show that while some models excel in certain criteria, none is universally the best. This highlights the need to carefully evaluate multiple LLMs for specific use cases to realize genuine value and a positive business impact." RWS Holdings plc, founded in 1958 and headquartered in the UK, is a world leader in technology-enabled language, content, and intellectual property services. The company combines AI and human expertise to help clients grow by ensuring their messages are understood globally, in any language. Over the past 20 years, RWS has developed its own AI solutions and supported clients in exploring, building, and using multilingual AI applications. With more than 45 AI-related patents and over 100 peer-reviewed papers, RWS has a strong track record in the field. RWS works with over 80% of the world’s top 100 brands, including more than three-quarters of Fortune’s 20 'Most Admired Companies,' and nearly all of the top pharmaceutical companies, investment banks, law firms, and patent filers. Its global network of 60+ locations across five continents serves clients in the automotive, chemical, financial, legal, medical, pharmaceutical, technology, and telecommunications sectors. For more information about RWS Holdings plc, visit www.rws.com.