The Open ASR Leaderboard, launched in September 2023 and visited over 710,000 times, has introduced a new strategy to combat benchmark-specific optimization known as benchmaxxing. Citing Goodhart's Law, which warns that measures become targets when pursued too strictly, the organizers announced the addition of high-quality private datasets from Appen Inc. and DataoceanAI. These datasets, covering scripted and conversational speech across multiple accents including American, British, Canadian, and Indian, will remain unpublished to prevent contamination of the test sets. The core challenge in maintaining the leaderboard involves balancing standardization and openness. While the project has successfully opened its evaluation scripts and unified test sets to ensure fair comparisons, this transparency has made the benchmark susceptible to overfitting. Developers may optimize models specifically for public test data, boosting leaderboard scores without improving real-world robustness. To address this, the new private datasets will be used to provide a more trustworthy measure of performance on diverse tasks, such as conversational speech and non-American accents, which are often underrepresented in current models. Appen and DataoceanAI have curated these datasets with varied characteristics, including different speaker ratios, styles, and transcription qualities. The Open ASR team will not release the raw data to developers. Instead, model developers must submit their models via the project's GitHub repository. The organizers will then verify results on public datasets and automatically compute metrics on the private sets. To prevent these private sets from artificially skewing the overall ranking, the default macroaverage Word Error Rate (WER) calculation excludes private data. Users can toggle this feature on to see the impact of private datasets on the rankings, which helps identify models that perform well under more nuanced conditions. The initiative aims to highlight the limitations of a single top-performing model, acknowledging that different applications prioritize different capabilities such as speed, multilingual support, or handling of disfluencies. By separating private and public evaluation tracks, the leaderboard can better expose biases between controlled settings and real-world scenarios. The organizers emphasized that while some models might gain an advantage if trained on data distributions similar to the private sets, using multiple data providers helps balance this advantage. Future developments will focus on evaluating models in noisy real-world conditions and ensuring consistent audio and transcript quality across all splits. The team invites community feedback on the new toggling features to help developers select models best suited to their specific use cases.