How to effectively make others aware of my hearing impairment, which is non-obvious?

What complicates matters and makes a simple "I'm sorry I'm hard of hearing" difficult to say is that in most cases people don't get the fact that I can "hear" them when I'm looking at them but suddenly not when we are not facing each other.

That's correct. So you'll have to tell them something more. I've had several co-workers with hearing difficulties. One of them, just like you, isn't wearing a hearing aid but can mask the hearing problem quite easily. So I never knew he's completely deaf on one side until that one time he I was on his wrong side. Then, he explained this to me once in detail:

I'm sorry, I didn't mean to ignore you but I'm hard of hearing. You might not have noticed before because I can understand you as long as you look at me, or if I'm on your right side, but if we're not facing each other or switched around I can't follow the conversation. Could you please look at me while you're talking, or draw my attention so we can switch to the correct places?

His statement included:

• Something like 'I'm sorry' or 'Excuse me', just the basic politeness stuff. Not to apologize for this problem, but to give me some reassurance that he was not doing the ignoring intentionally.
• Statement of the problem: hard of hearing, in his right ear.
• Big point: a solution for the problem, such as being able to see the face of the speaker and read their lips will help follow the conversation/ the position of the good ear.

I can't speak for other people with regards to the rude/cold/pity/sympathy feelings because this is highly dependent on the person you're dealing with. But, personally, if you'll make a statement that includes these points, I won't find you rude or cold because you just explained there's a legitimate reason you could hear me at that time but not now. And I won't pity you either because you've just shown that it's easily remedied and doesn't have to be a big problem. You will get some sympathy from me though because sympathy also means 'understanding between people'. And you will need some understanding of your problem and its solution to get me to actually do what is required for the solution.

Of course, with regard to my co-worker, I often forget which ear is his good ear. But I know that if he seems to ignore me, I'm on the bad side and should switch places, or find some other way to draw his attention. A tap on his shoulder works fine, if you're not bothered by that, you can offer that as a solution to these people as well.