Audio understanding is a multimodal task that uses artificial intelligence to analyze, recognize, and understand audio information. Its goal is to enable machines not only to process sound signals but also to understand the language content, environmental information, emotional features, and semantic relationships contained within them. This technology integrates Automatic Speech Recognition (ASR), audio event recognition, natural language processing, and multimodal learning, and is a crucial foundation for achieving intelligent voice interaction.

The development of audio understanding is built upon long-term research in multiple fields, including speech processing and machine learning. Early systems typically treated speech recognition, audio classification, and language understanding as independent tasks, making it difficult to establish connections between sound signals and high-level semantics. With the development of large-scale pre-trained models and multimodal learning, researchers began exploring combining audio encoders with language models, enabling the model to process both sound and text information simultaneously. In 2023, Google researchers published a paper... AudioPaLM: A Large Language Model That Can Speak and Listen The AudioPaLM model was proposed, which combines the language understanding ability of PaLM-2 with the audio processing ability, and explores a large language model framework with auditory understanding and speech generation capabilities. It has become one of the important representative studies in the field of audio language model (ALM).

Modern audio understanding systems typically extract sound features through audio encoders and map them to a linguistic semantic space using multimodal models, thereby performing tasks such as speech recognition, audio description generation, sound event analysis, and cross-modal reasoning. In recent years, with the development of fundamental audio models and large-scale multimodal models, this technology has been widely applied in scenarios such as intelligent assistants, meeting transcription, audio search, content analysis, accessibility assistance, and human-computer interaction, providing crucial technical support for machines to understand sound information in the real world.