BoldVoice, an AI-powered accent coaching app for non-native English speakers, is exploring how machine learning models can understand and measure accent strength. Accents are complex patterns in speech that involve vowel shapes, timing, pitch, and other phonetic elements, typically analyzed by linguists. BoldVoice aims to make these insights accessible through AI. To achieve this, they generate an "accent fingerprint" by processing English speech recordings through a large-scale accented speech model. This fingerprint is represented as a 768-dimensional embedding, which is then analyzed in a latent space to understand accent characteristics. The Latent Space Analysis The Original Recordings Victor (Non-Native Speaker): His recording has a noticeable Chinese accent when speaking English. Eliza (Native Speaker): She reads the same passage with a standard American English accent, serving as the target. Visualizing Accents in the Latent Space To understand the machine's perception of accent strength, BoldVoice populated a latent space with 1,000 speech recordings varying in accent intensity. They applied Partial Least Squares (PLS) regression to identify dimensions most correlated with human-rated accent strength, followed by UMAP (Uniform Manifold Approximation and Projection) for 2D visualization. In this space: - Lower Left: Native or near-native English speakers. - Upper Right: Speakers with strong accents. Victor’s and Eliza’s embeddings were plotted: - Eliza: Located in the lower left, indicating a native-like accent. - Victor: Positioned in the upper right, reflecting a strong Chinese accent. Cleaning the Background Noise Initially, they hypothesized that cleaning background noise from Victor’s recording might help him focus better on accent nuances. After applying noise reduction, the cleaned recording remained in the same position in the latent space, confirming that background noise does not significantly affect accent strength metrics. Converting the Accent Next, they used an in-house accent conversion model to simulate Victor’s voice with Eliza’s accent: - Converted Recording: Placed very close to Eliza’s original position, suggesting the model successfully transferred the accent while maintaining Victor’s voice characteristics. Practical Application: Practicing the Accent Victor practiced mimicking the converted recording. After about 10 minutes, his new recording showed notable improvements: - Improved Recording: Positioned at the border between Intermediate and Advanced levels in the latent space, indicating significant progress. To reach native-like proficiency, the app offers sound-by-sound phonetic analysis, helping users understand and apply specific patterns of pronunciation and stress. Key Insights and Future Directions Quantitative Tracking: The model provides a way to quantitatively measure a user’s accent journey by tracking their distance from a target accent profile in the latent space. Evaluating ASR Systems: This approach can rigorously evaluate Automatic Speech Recognition (ASR) systems for performance variations across different accent strengths. Monitoring TTS Systems: Similarly, it can monitor Text-to-Speech (TTS) systems for unwanted changes in accent, known as "accent drift." Industry Evaluation and Company Profile Industry insiders commend BoldVoice for its innovative use of AI to enhance accent coaching, which can significantly benefit global communication by making English more accessible and understandable for non-native speakers. The company’s focus on creating practical applications like real-time feedback and accent conversion models sets it apart in the rapidly evolving AI-driven language learning market. BoldVoice, founded by a team of linguists and AI experts, leverages cutting-edge technology to bridge the gap between linguistic theory and practical speech improvement. Their robust data set and sophisticated models are positioning them as leaders in the field of accent modification and speech recognition. Future posts promise to delve deeper into accent fingerprints and explore how they vary across different language backgrounds.