Managing effective incident response in technology projects, particularly within constrained budgets, is a challenge many teams face. One vivid example of this was during a cloud modernization project I worked on as a backend architect. Our team was pressured to minimize service-level logging to cut costs on our observability platform. This decision, while initially successful, backfired when we encountered unexpected issues in production. Without the detailed logs we had previously relied on, pinpointing the root cause became nearly impossible. Hours were wasted, uncertainty and anxiety spread, and team morale plummeted. These logs, such as the one below, are essential for understanding complex system behavior: { "timestamp": "2025-03-21T14:05:03Z", "service": "preference-engine", "level": "ERROR", "message": "Worker queue overflow: unable to dispatch to worker pool", "requestId": "abc123", "userId": "admin_42" } A simple query in a logging platform could have quickly identified and resolved the issue: _sourceCategory=prod/preference-engine "Worker queue overlap" | count by userId, requestId However, the absence of these logs rendered our troubleshooting efforts inefficient and ineffective. This exposure to uncertainty undermined our confidence and revealed gaps in our testing and observability strategies. Lean Doesn’t Have to Mean Starved for Resources The "lean" approach can be beneficial, encouraging focus and efficiency. However, it's crucial to maintain essential tools and resources, especially in high-stakes environments. Our cloud migration had a strict deadline tied to a browser deprecation date, necessitating rapid and smooth service transitions. Detailed logs were our primary safety net for resolving issues and understanding unexpected scenarios. Initially, we inserted necessary logging while working on each method, but the cost-cutting mandate stripped away this protective layer. Without logs, reproducing incidents locally was nearly impossible due to the variability in real-world use cases. Partial logging reintroduced the same costs and failed to deliver useful insights when needed most. MTTR Versus Signal Quality During incident retrospectives, mean time to recovery (MTTR) is often a key metric. However, low MTTR alone doesn’t guarantee successful incident response; it depends on the quality of the signals. Industry benchmarks show elite teams achieving sub-hour MTTR, but speed isn’t just about automation—it’s about high-fidelity, contextual signals. Generic error messages or delayed alerts from aggregate metrics introduce ambiguity and waste critical triage time. Structured logs with userId, requestId, and service traces provide clear, actionable insights, reducing MTTR effectively. An observability platform can enhance MTTR, but only if the ingested data is meaningful and actionable. How Sumo Logic Could Have Made a Difference What could have improved our situation? Better log analytics and application performance monitoring (APM) would have been invaluable. A pay-per-analysis model like Sumo Logic’s allows continuous log ingestion without incurring high costs until analysis is needed. This approach: Reduces Anxiety: Teams can log extensively, knowing they won’t face unnecessary financial burdens. Improves MTTR: High-fidelity, contextual data enables faster identification and resolution of issues. Enhances Budget Management: Costs are tied directly to the impact of the analysis, making it more justifiable. With Sumo Logic, we could have set up our observability stack to include APM, log management, service monitors, alerts, and metrics aligned with system success. For every new feature, an associated metric would ensure its proper function. Machine-Assisted Triage Beyond unlimited log ingestion, machine-assisted triage tools are essential. These tools automatically group anomalies, detect outliers, and correlate signals across services. For example, the command: _sourceCategory=prod/* error | logreduce clusters noisy log data into actionable categories, such as: Error: Worker queue overflow Error: Auth token expired for user * Error: Timeout in service * From there, teams can drill down to specific details: | where message matches "Auth token expired*" | count by userId, region This workflow streamlines the search process, accelerates decision-making, and minimizes stress during incidents. Conclusion The "do more with less" philosophy can be a double-edged sword. While it encourages resourcefulness, it must not compromise crucial observability tools. A zero-cost ingestion model combined with machine-assisted triage provides a balanced solution. It empowers teams to log comprehensively, ensuring they can quickly and accurately identify root causes, reduce downtime, and manage budgets effectively. My advice, as I’ve shared in my previous writings, is to focus on delivering high-value features while leveraging specialized tools for other critical tasks. This approach not only enhances productivity but also reduces anxiety and stress during incident response. Let’s keep those logs! Industry Insights: J. Vester, a seasoned IT professional, emphasizes the importance of balancing lean principles with the necessity for robust observability tools. Sumo Logic’s innovative pay-per-analysis model addresses the budgetary constraints while maintaining high-quality data logging. This model has gained traction among tech companies for its ability to improve MTTR and overall system reliability without incurring prohibitive costs. Company Background: Sumo Logic is a leading cloud-native observability and security platform. By providing comprehensive log management and analysis, it enables organizations to gain real-time visibility into their applications and infrastructure, thereby enhancing operational efficiency and security.