From Pixels to Concepts: Do Segmentation Models Understand What They Segment?

Segmentation is a fundamental vision task underlying numerous downstream applications. Recent promptable segmentation models, such as Segment Anything Model 3 (SAM3), extend segmentation from category-agnostic mask prediction to concept-guided localization conditioned on high-level textual prompts. However, existing benchmarks primarily evaluate mask accuracy or object presence, leaving unclear whether these models faithfully ground the queried concept or instead rely on visually salient but semantically misleading cues. We introduce CAFE: Counterfactual Attribute Factuality Evaluation, a benchmark for evaluating concept-faithful segmentation in promptable concept segmentation models. CAFE is built on attribute-level counterfactual manipulation: the target region and ground-truth mask are preserved, while attributes such as surface appearance, surrounding context, or material composition are modified to introduce misleading semantic cues. The benchmark contains 2,146 paired test samples across Superficial Mimicry (SM), Context Conflict (CC), and Ontological Conflict (OC). Experiments reveal a systematic gap between localization quality and concept discrimination: models often generate accurate masks even for misleading prompts, suggesting that strong mask prediction does not necessarily imply faithful semantic grounding.