'Whiteness' itself is a construct - in recent discussion somewhere (I think misia's lj on panic over the white middle classes not breeding and being SWAMPED by philoprogenitive persons of other colours, I remarked that the Irish used to occup that position of Other. Also, within European context, people from around the Mediterranean often occupied a v liminal position, sometimes white and sometimes darker.

But yes, it's often hard to distinguish in large modern cities unless people are over-tuned to what they consider to be markers of The Other.