The job of the weapon and holster, 99.99% of the time, is just to be available. If it's there, and doesn't do anything, then it has done its job for the day. I tend to be cautious and will try to position myself and my companions out of bad situations at the first hint of trouble. I avoid areas with high violence and try to watch my surroundings closely. There's a very good chance - hopefully - that I'll never encounter a real-life situation that requires me to draw.
At the same time, the more minutes in your life that you are covered by a readily accessible firearm, the less minutes there are in which you're vulnerable. It's better to have more coverage than less.
If I want to relax, I'm going to relax whether I'm carrying a P3AT under swimming shorts, or have the Moss covered by a towel under a deck chair. Since you're potentially vulnerable everywhere, it's best to just accept it and make sure that you're covered as much as possible. Even if you have to go into the ghetto, stay relaxed while being alert. This may require meditation, or whatever does it for you.
I think there's a greater chance of a car crashing into my house and killing me in the shower than an armed assailant randomly storming in. But, I'd feel extraordinarily stupid if I happened to be in the shower at the time, and couldn't access any one of the five firearms on the other side of the wall. I'd much rather that one of the guns be doing its job of being available for defensive use, than just sitting pointlessly in another room.