Hybrid LLM Judge Grades Identity Resolution at 4x Lower Cost

Hightouch has evaluated large language models as judges for identity resolution, finding that a hybrid system pairing the decision model Jev with the frontier model Claude Opus can approximate leading accuracy while reducing evaluation costs by more than four times. The company tested whether models could audit customer profiles for over-merging, a common failure in which records belonging to different people are incorrectly assigned to one identity. Identity resolution is essential for marketing, personalization and analytics, but verifying its accuracy is difficult because production data lacks a definitive answer key. The benchmark used synthetic data generated with pseudopeople to simulate a U.S. town and create 1,000 profiles, half representing a single person with noisy records and half containing deliberate over-merges. Data errors included typos, nicknames, blank fields, shared household values, altered addresses and inconsistent dates. Hightouch asked models to decide whether each profile described one real person and whether it represented over-merging. Out of the box, GPT-6 Luna reached 97.3% accuracy at about $0.10 per 1,000 evaluations, while Jev reached 95.6% at about $0.05 per 1,000. Frontier models such as Claude Opus performed better, reaching 99.0% accuracy, but at higher cost and latency. Hightouch adjusted confidence thresholds, setting Jev at 60% and Luna at 70% to improve accuracy. When the test was narrowed to harder cases, including similar names, twins and senior-junior profiles, Luna’s performance declined to 88.8%, while Claude Opus reached 95.1%. Hightouch then built hybrid evaluators that used a cheaper model first and sent uncertain or difficult profiles to Claude Opus for a second opinion. The hybrid Jev system achieved 98.8% accuracy on easier cases at $0.88 per 1,000 evaluations and 95.3% on harder cases at $2.94, compared with $5.39 for Claude Opus alone on the harder set. The hybrid Luna system was cheaper but less accurate on difficult data, reaching 91.6% and $1.15. A key finding was that model calibration mattered more than raw score. Jev assigned uncertainty more reliably, allowing the hybrid system to escalate difficult profiles to the expensive model when needed. Luna was often overconfident, which limited its usefulness in harder cases. Hightouch concluded that well-calibrated low-cost decision models can serve as scalable first-line judges for identity resolution audits, while frontier models can be reserved for edge cases. The approach could speed up customer-data setup and maintenance by reducing reliance on slow manual review. All test data was synthetic, with no real individuals represented.