There exist two widely used notions concerning the structure of phenomenal color space. The first is the notion of unique/binary hue structure, which maintains that there are four unique hues from which all other hues are composed. The second notion is the similarity structure of hues, which describes the interrelations between the hues and hence does not divide hues into two types as the first notion does. Philosophers have considered the existence of the unique/binary hue structure to be empirically and phenomenally well-grounded, and the structure has been considered to be primary because this can account for the similarity structure. Consequently, the unique/binary hue structure has played a central role in color philosophy. This calls for the assessment of the justification for its existence carried out in this paper. It is concluded that, despite the prevalent view among philosophers, none of their reasons for endorsing the existence of the unique/binary hue structure are justified. Since the notion of the unique/binary hue structure appears intuitively plausible for many, however, a sketch explaining this intuition is outlined at the end.