Calibration
Can you trust the judge? Label a few conversations yourself on the same five criteria, then see how often the LLM judge agrees. Cohen’s κ corrects for chance, so a judge that always says “pass” cannot look good.
Can you trust the judge? Label a few conversations yourself on the same five criteria, then see how often the LLM judge agrees. Cohen’s κ corrects for chance, so a judge that always says “pass” cannot look good.