AI engineering is not a replacement for software engineering—it’s an evolution. As someone working in the AI space, I’ve come to realize that most of my day-to-day work isn’t about training models from scratch. Instead, it’s about integrating powerful pre-trained models into real-world applications, solving business problems, and ensuring those systems are reliable, scalable, and useful. The key insight I’ve taken from recent reading, especially O’Reilly’s AI Engineering, is that AI engineering is fundamentally software engineering with AI models layered into the stack. This means we’re still dealing with core software concerns—deployment, monitoring, scaling, observability—but now with added complexity from probabilistic outputs, context management, and model behavior. I’ve been thinking deeply about one of the most critical but under-discussed aspects of this new stack: evaluations. In traditional software, testing helps catch regressions—when a new change breaks something previously working. In AI, the same risk exists, but it’s far more subtle. A prompt tweak might improve accuracy on one task while reducing helpfulness in another. A RAG pipeline update could make responses more factual but less conversational. Without proper evaluation, these trade-offs go unnoticed. Evaluations are the safety net. They’re what allow us to move fast without breaking things. But unlike software tests, AI evaluations are hard. Models are complex, outputs are often open-ended, and there’s no single “right” answer. A summary can be fluent and grammatically correct but miss the main point. A chatbot can sound polite while giving dangerously misleading advice. So how do we evaluate effectively? I’ve found it helpful to break evaluations into two categories: quantitative and qualitative. Quantitative evaluations are rule-based and measurable. Did the model solve the math problem correctly? Did the code run without errors? These can often be automated and scaled, making them ideal for continuous integration workflows. Qualitative evaluations are judgment-based. Is the tone appropriate? Does the response feel natural? Is the summary coherent and relevant? These require human insight—but we can’t rely solely on humans at scale. That’s where AI judges come in. By defining clear criteria—helpfulness, clarity, factual accuracy, tone—we can use another model to score outputs. This approach mimics human evaluation but at scale. Run over thousands of examples, these AI judges can detect subtle drifts in performance after model updates, prompt changes, or pipeline refinements. The real power comes from eval-driven development—defining what success looks like before building anything. Just as test-driven development ensures code meets requirements, eval-driven development ensures AI systems deliver real value. This means asking: What does “good” mean in this context? Who is the user? What are the business outcomes? It’s not enough to be correct. A model that generates accurate code but takes minutes to run isn’t practical. We need to evaluate runtime, memory use, and cost. For generative tasks, fluency and coherence matter as much as correctness. And for factual accuracy, we need both local consistency (with provided context) and global consistency (with external knowledge). Safety is another layer. Beyond filtering profanity, we must guard against data leaks, prompt injection, and harmful outputs. Evaluations should test for these risks systematically. In the end, as AI systems grow more capable, the need for robust, thoughtful evaluation only increases. It’s not just about measuring performance—it’s about ensuring reliability, trust, and real-world impact. The shift isn’t just technical. It’s a change in mindset. We’re no longer just writing code. We’re designing systems that reason, respond, and adapt. And with that comes the responsibility to measure what matters—not just what’s easy to measure. The future of AI engineering isn’t about building smarter models. It’s about building smarter ways to evaluate them.