You certainly wouldn't phrase it like that if you wanted to be really polite -- for example, if you sensed that a guest was tired or ill but didn't want to offend you by leaving early.
If you'd rather not stay any longer, ...
If you'd prefer to go on home now, ...
... please feel free to leave whenever you'd like to.
... please don't hesitate, go ahead and leave any time.
If you'd like to leave, please feel free to.
If you'd like to leave, please go ahead.
In a sentence like your example, yes, it's likely to sound annoyed or aggressive. It implies that you're unconcerned, or even annoyed because the other person seems bored or uninterested in the entertainment you're offering.
If you want to leave, feel free.
If you want to leave, go ahead.
If you want to leave, be my guest.
If you want to leave, go right ahead.
That said, if you as a non-native said 'Be my guest' with a friendly smile, people would surely understand that you weren't angry.
And cello0's examples are also good.
A: Do you mind if I play your piano?
B: No, of course not. Be my guest!
A: Is it okay if I sit here?
B: Certainly, please do. Be my guest!
I would phrase the one about dinner differently, though:
A: Let me pay for dinner.
B: No, no, you're my guest tonight.
... I'd like you to be my guest.
... it's on me.
... it's my treat.
... I insist. It's my pleasure.