First, be careful not to generalize or make race-based assumptions (and people, do NOT let this conversation devolve into racism). Second, don’t assume that because they’re eyeballing you, it’s because they want to get in a fight. You’re most likely being observed by all men, you’re just noticing the more aggressive ones because they’re directly watching you.
Emperor raises your internal status. Your body language, personal energy aura, speech, etc. is changing, giving off a more dominant aura, even if you don’t realize it. These people are simply reacting to what seems to be an enigma or potential threat. In their eyes, you are the one being aggressive. It’s simply a consequence of gaining more value. People complain about this all time – I experienced it myself when running status based subs, and even natural ways of gaining more status (like opening a business).
Don’t let anyone shake you, but also don’t go around punching everyone that looks at you sideways.