Oh I get it now! It refers to salmon (fish) when it has to lay eggs, it returns to its original hatching place by swimming from the river to the ocean (upstream swimming) which is hard work. So I guess this phrase refers to that, he can't go back to his shrine that easily or find it since he's not a salmon returning upstream to spawn.
Hope you understand it even if it's a little bit haha! 😅