Japanese Open-Source Model Shisa V2 405B Surpasses GPT-4

Japan's AI is making waves globally with the release of Shisa V2 405B, a groundbreaking open-source model that outperforms GPT-4 in Japanese tasks. This article delves into the latest advancements from Shisa.AI, a Tokyo-based startup focused on developing and deploying advanced open-source AI language and speech models for the Japanese market. Shisa.AI has recently unveiled Shisa V2 405B, a large language model built on Llama 3.1, which is hailed as "the strongest large language model ever trained in Japan." This open-source model excels not only in Japanese tasks but also retains powerful English capabilities, demonstrating its exceptional performance in a bilingual setting. Test results show that Shisa V2 405B surpasses both GPT-4 and GPT-4 Turbo in several Japanese benchmarks, and matches the performance of the latest GPT-4o and DeepSeek-V3 in Japanese-language tasks. This achievement marks a significant rise of Japan's AI labs in the global competition and opens new possibilities for Japanese AI applications. Focused Optimization and Advanced Fine-Tuning Shisa.AI’s approach to model development centers on optimizing post-training processes rather than the costly continuous pre-training and expanding tokenizers used in earlier models. By leveraging synthetic data, they have significantly enhanced the performance of their models. The core dataset, ultra-orca-boros-en-ja-v1, has been meticulously filtered, regenerated, and resampled, making it one of the most robust Japanese-English bilingual datasets available. This dataset is freely available under the Apache 2.0 license, offering valuable resources to developers worldwide. Diverse Model Family from 7B to 405B Parameters The Shisa V2 series includes models ranging from 7 billion parameters (7B) to 405 billion parameters (405B), catering to a wide spectrum of device and computing requirements. These models excel in various Japanese tasks such as grammar, character portrayal, and translation. For instance, in tests like shisa-jp-ifeval (Japanese instruction-following evaluation), shisa-jp-rp-bench (Japanese role-playing benchmark), and shisa-jp-tl-bench (Japanese-English translation benchmark), the Shisa V2 series outperformed their foundational counterparts. Notably, Shisa V2 405B incorporates a small amount of Korean and Traditional Chinese data, enhancing its multilingual capabilities and broadening its application scope in cross-lingual scenarios. Open Source Spirit Driving Global Innovation Shisa.AI’s contributions extend beyond Japanese AI performance improvements; they also foster innovation within the global AI community. The training logs for the Shisa V2 series are publicly available on the Weights and Biases platform, providing transparency into the development process. The models were trained using an AWS Sagemaker cluster of four H100 nodes, along with cutting-edge technologies like Axolotl, DeepSpeed, and Liger Kernel, ensuring efficient model development. Moreover, Shisa.AI plans to open-source its Japanese-specific benchmarking tools, which will aid researchers and developers in evaluating and advancing large language models for Japanese. This initiative further supports global collaboration and development in the AI ecosystem. Future Prospects: Strengthening Japan's Global AI Competitiveness Shisa.AI’s success demonstrates that even smaller AI labs can make substantial contributions to the global AI landscape. By releasing their models and datasets, they have provided strong support for the widespread adoption of Japanese AI applications. AIbase anticipates that as Shisa.AI continues to update and refine their offerings, Japan’s position in the global AI arena will become even more solidified. For developers dealing with complex Japanese tasks, the Shisa V2 series represents a powerful tool worth exploring. AIbase recommends monitoring Shisa.AI’s official website and HuggingFace page for detailed technical information and opportunities to test the models. In summary, through the Shisa V2 series, Shisa.AI has showcased Japan’s innovative potential in AI. These open-source models pave the way for future advancements in both academic research and commercial applications, solidifying Japan’s growing influence in the global AI community.