Selfish?

I've been wondering...

Why does nobody say the person leaving someone is selfish?
Why is only "wishing someone to be with me" a selfish thing?

If both be selfish,
The former is more selfish than the latter.
But people do not say a lot about the former.
People always talk about the latter.

Perhaps both of them are selfish acts.
Perhaps not.

But the former is definitely more cruel.