If I am understanding correctly you used the muzzle end of your firearm to lift your sleeve to look at your watch thereby displaying a firearm to someone who you presumed had the intention to rob you. You deduced their intention by them asking you what time it was..?
That isn't the most sound decision i've ever heard of - a simpler approach may be responding to the questioner "I don't know what time it is" - avoiding areas that you know to be hot spots for criminal activity and drugs. I myself am not a proponent of pocket carry. Not because I don't find it comfortable or somewhat practical but because my pockets are usually full of other stuff.
Aside from the short story long and the questionable decisions made you do have a point about if you're being robbed or having your pockets rummaged due to bad decisions it would be a bit of a problem to have your firearm taken away from you, an IWB or OWB under a jacket would likely be discovered as well though. Not enough of an issue for me to write off pocket carry.
I'd rethink your strategy.