A review prompt includes the criterion: 'The response should be professional.' This criterion produces inconsistent results — reviewers disagree on whether specific responses pass or fail. What is the fix?
Answer: Replace the adjective criterion with 2–3 binary rules: 'contains no informal language (slang, expletives),' 'does not address the user by first name in an opening greeting,' 'does not use contractions in formal mode'
Adjective criteria produce inconsistent results because 'professional' has no definitive threshold — different reviewers (and model runs) draw the line differently. Binary criteria have clear pass/fail conditions: either the response contains informal language or it doesn't. Replacing the adjective with specific binary rules produces consistent, auditable results. Defining 'professional' (A) makes the criterion clearer but still leaves a subjective threshold. A 1–10 score (C) is useful for genuinely continuous qualities but doesn't eliminate subjectivity for this criterion. Industry standard (D) introduces a reference that's undefined and variable.