Financial Services Firms Lead AI Adoption, Prioritizing In-House Training for Control and Security

Financial services firms are increasingly prioritizing in-house AI training to commercialize generative AI technologies, driven by the need for control, security, and regulatory compliance. While tech giants like Meta, Google, and OpenAI focus on theoretical advancements in AI models, financial institutions are at the forefront of applying these innovations to real-world workflows. This trend mirrors their historical role in adopting disruptive technologies, from ATMs to mobile banking, but now with a heightened focus on navigating the complexities of AI integration in highly regulated environments. Scott Hebner, a principle analyst at theCUBE Research, highlights that financial services companies (FSIs) are early adopters of AI due to their high-stakes operations. These firms handle vast, high-dimensional datasets involving “other people’s money,” necessitating rigorous safeguards for trustworthiness, explainability, and compliance. While they have embraced predictive AI for years, they are now experimenting with generative AI (GenAI) to enhance tasks like regulatory reporting, investment decisions, and claims processing. However, the path to adoption is not straightforward. A critical challenge lies in where FSIs will train their GenAI models. Licensing cloud-based solutions offers convenience but risks losing control and incurring high costs. For example, a single Nvidia H100 SXM GPU, paired with supporting infrastructure, costs around $53,750 over three years, while cloud providers like AWS and Azure charge 2.6 to 6 times more for similar compute power. This economic disparity, combined with security concerns and latency requirements for inference tasks, is pushing many FSIs to invest in on-premises or co-location facilities. Vik Malyala of Supermicro notes that FSIs are hesitant to rely on cloud infrastructure for GPU-heavy workloads, citing expenses and the need to optimize data integration. Many are retraining models internally or using co-location setups, where they can manage data and models without exposing them to external risks. This approach is particularly vital for FSIs developing proprietary AI systems, which are as sensitive as their traditional trading algorithms. Training GenAI models requires massive infrastructure. Unlike high-frequency trading systems, which operate on single nodes, AI training demands hundreds to thousands of GPUs working in unison. These systems require high-bandwidth memory, low-latency networking, and specialized cooling. For instance, a single rack with eight H100 GPU nodes can consume 40.8 kilowatts, while newer systems with Blackwell GPUs may reach 120 kilowatts. Such power demands are incompatible with most existing datacenters, making liquid cooling essential for future scalability. Supermicro’s Grace-Hopper and Grace-Blackwell systems are gaining traction in the sector, designed for datacenters without liquid cooling capabilities. These systems, with configurations like the GH200 NVL2 (two Hopper GPUs) and GB200 NVL4 (four Blackwell GPUs), are being used for both training and inference. However, FSIs remain secretive about their specific applications, with some customers opting for AMD and Intel processors for trading tasks. Regulatory constraints, such as the Volcker Rule, differentiate banks from hedge funds and proprietary trading firms. While banks are limited in using deposit funds for speculative trades, hedge funds operate with more flexibility, enabling them to experiment with AI-driven strategies. This regulatory landscape has fueled the growth of trading companies, which now leverage GenAI to analyze complex financial datasets—often hundreds of terabytes daily—replacing traditional algorithmic methods. GenAI’s potential to improve decision-making, such as identifying trading patterns or assessing loan risks, is driving adoption. However, the transition to AI requires careful planning. While banks initially focus on text-based tasks like compliance, hedge funds are investing in large GPU clusters to refine their models. The shift to lower-precision formats like FP8 and FP4, which reduce training costs and time, is also accelerating. Looking ahead, FSIs are preparing for the rise of liquid-cooled, high-density AI systems. These setups, which may consume up to 1 megawatt of power, will require efficient heat management. Liquid cooling, once common in mainframes, is becoming a necessity as AI workloads grow. Over time, these advancements could set a precedent for other industries, as FSIs continue to lead in deploying AI supercomputers and agentic architectures. As the AI race intensifies, financial services firms are balancing innovation with caution, ensuring their systems meet regulatory standards while leveraging cutting-edge hardware. Their choices today may shape the future of AI infrastructure across the broader tech landscape.