I'm vilified and don't get attacked

Ok, I know from previous questions that the whole reputation and karma thing of FA:NV is a bit wierd but now I can't make heads or tails from it. I have been vilified with Caesar's Legion pretty much from the start. Today I bumped into Cottonwood Cove from the south. It made for excellent sniping and sniped I did. Picked off any Legion member I could see. Then I started moving into the cove and got stopped by a guy who wanted to know what I was up to. An swift speech check later and I'm now waltzing about and no one cares. I could even buy some slaves their freedom. And I'm NOT wearing any Legion armor at all, just plain neutral power armor.

I thought that vilified status made that affiliation attack you on sight.