What does it when the directional complement (趋向补语) is about the speaker himself ?

the Directional complement - 趋向补语 indicates that the verb is done in the same direction or the oppisite direction from the speaker
他上来 （he is going up to the speaker )
他上去 （he is going up but leaving the speaker ）

but what does it means when the speaker himself is talking about himself in the Directional complement sentence ? the direction of the speaker towards himself ? for example

我过去
我过来

我上去
我上来

Help me understand ?