In English some people refer to their fathers as"My Old Man" and it is not considered an insult but if its directed at someone who is a stranger( depending on the tone) it is basically like calling them weak and pathetic which is an insult.
If it's an address, it's rude. "Hey, old man!" would be a very blunt and honestly rude way to call an elderly person in English. If it's just in narration, it's neutral: like in "there was an old man sitting on the bench," "an old man told me," etc, it's just a way to indicate that the person is old. Another (less frequent) use of "old man" is slang to talk about one's father: my old man = my father. But it's slang, so it's not super polite either. And it wouldn't be an address (you wouldn't directly call someone this), just a way to talk *about* that person.