Modern data centers rely heavily on Baseboard Management Controllers (BMCs) for remote administration, allowing IT professionals to manage server configurations, monitor hardware health, and apply firmware updates, even when systems are powered off. BMCs play a crucial role in maintaining the efficiency and reliability of data centers, particularly in hyperscale and physically inaccessible environments. However, their powerful capabilities also make them significant security risks, expanding the attack surface for potential breaches. Recently, the NVIDIA Offensive Security Research (OSR) team conducted an in-depth analysis of BMC firmware used in data center servers, uncovering 18 vulnerabilities and developing nine working exploits to assess their real-world impact. This highlights the critical need to secure BMCs, as they can provide attackers with persistent and undetectable access to an entire fleet of devices. How BMCs Work and Why They Are Risky BMCs are essentially embedded service processors that operate independently of the host operating system. They provide remote keyboard-video-mouse (KVM) access, enabling administrators to modify BIOS settings, apply firmware updates, and control boot behavior. These capabilities are essential for managing large-scale infrastructure, but they also introduce potential security gaps. BMCs often run outside traditional security monitoring systems, exposing dedicated management interfaces and relying on third-party firmware stacks. If compromised, they can serve as a pivot point for deeper system intrusions, making them a prime target for cyberattacks. Inside the BMC: Unveiling Vulnerabilities Leaking Credentials with Side Channels The NVIDIA OSR team investigated the IPMI authentication process and found that it remains susceptible to a hash-leak vulnerability (CVE-2013-4786). This vulnerability can be exploited if the attacker knows a valid username, which the BMC's response timing inadvertently revealed through a classic timing oracle side channel. The team used the leaked hash and standard word lists to brute force passwords offline, gaining full username and password access. Full Remote Access Through Insecure APIs The firmware image showed that the user database is managed by Redis, with encrypted passwords and keys stored alongside them. The team discovered an API that allowed querying the Redis database, decrypting passwords, and obtaining the complete user database. Another API provided read/write access to the IPMI server process's virtual memory without proper checks, indicating a lack of Address Space Layout Randomization (ASLR). This enabled the team to locate and toggle a hidden configuration flag, enabling additional features like file downloads and facilitating the exploitation of other vulnerabilities. Pivoting to the Host System With full access to the BMC, the researchers explored interactions with the host system. Using the BMC's KVM functionality, they modified bootloader parameters to gain shell access to the host operating system without user credentials. This was possible because Secure Boot was not enabled. Even if Secure Boot were active, the BMC could adjust Unified Extensible Firmware Interface (UEFI) settings, potentially bypassing protection. The team also found an exposed API that allowed the BMC to read and write directly to the host's SPI flash, modifying NVRAM entries and disabling Secure Boot. Classic Memory Exploits, No Modern Mitigations In the authentication code, the team discovered a stack-based buffer overflow due to the use of strcpy for copying unvalidated input into a fixed-size buffer. This vulnerability, combined with the absence of standard modern mitigations such as stack canaries, non-executable stack, and ASLR, made it relatively easy to exploit. The team developed an exploit that fully hijacked the control flow, enabling the execution of injected shellcode during login attempts. Driving Fixes Across the Ecosystem Upon validating the vulnerabilities, the NVIDIA OSR team collaborated closely with American Megatrends Inc. (AMI), the vendor responsible for the affected BMC firmware. Detailed technical reports were provided to assist AMI in patching the issues and coordinating fixes with their customer base. Since this firmware is widely used across the industry, the team issued their own Common Vulnerabilities and Exposures (CVEs) to expedite awareness and remediation efforts. This proactive approach not only protected NVIDIA customers but also contributed to enhancing BMC security across the entire data center ecosystem. What Security Teams Should Do Now Incorporate BMCs into Security Models: BMCs are highly privileged systems that control critical aspects of hardware. Security teams must treat them with the same level of scrutiny as other IT infrastructure components. Enable Strong Configuration Management: Ensure that Secure Boot and other security features are enabled and properly configured. Regularly audit and update BMC firmware to include the latest security patches. Implement Monitoring and Controls: Integrate BMCs into existing security monitoring systems. Monitor for unusual activity and implement access controls to limit who can interact with these controllers. Use Modern Mitigations: Ensure BMC firmware includes modern security measures such as stack canaries, non-executable stack, and ASLR to protect against common exploitation techniques. Evaluation by Industry Insiders and Company Profiles Industry experts commend NVIDIA's proactive approach to securing BMCs, emphasizing the importance of identifying and addressing vulnerabilities in often-overlooked components. The research underscores the need for robust security practices and continuous monitoring across all layers of infrastructure. NVIDIA, known for its leadership in GPU technology and data center solutions, continues to prioritize security in its products and ecosystems, driving innovation and trust in the tech community. The full research paper, "Breaking BMC: The Forgotten Key to the Kingdom," provides detailed insights and recommendations for further reading.