A developer uses three few-shot examples that all reference 'Order 123' and 'Customer Alice.' The model produces correct output in testing but fails on new orders and customers. What is the likely cause?
Answer: The model has memorized the entity-specific mapping (Order 123 → specific output) rather than learning the underlying rule
When the same entities appear in all examples, the model can learn a shortcut: 'when I see Order 123 or Customer Alice, produce this output.' On new entities, the shortcut doesn't apply, so the model has no learned pattern to follow. Varying entities across examples forces the model to learn the rule rather than the entity mapping. Adding examples with more entities (A) is the correct fix, but the question asks for the cause, not the fix. The model doesn't need production data (C) — diverse synthetic examples work. Few-shot does work across entities (D) — when examples are designed correctly.