Are those the only two places you go outside the home? What about grocery shopping (or any kind of shopping for that matter), the bank, and all the car rides in between?
Asking the pastor is worth the effort IMO. Surely you, and he, have heard recent stories of churches being shot up. Just because you are among friends and aquaintences does not guarrentee your safety.
As for the restaurants, I'd still have it with me for the car ride. I'm not crazy about leaving it in the car, but if I'm going somewhere where I may not take it in, I'd rather leave it in the car than at home.
But, to each his own I guess.