I tried it once with a "concealable" model. It was a relatively thin model that had a white liner and velcro on the straps. There was also a pocket in the front for ceramic inserts. It was still pretty bulky, and tends to push up against an IWB, providing an additional obstruction to get through to access the firearm. You also have to wear a shirt under it, which adds additional complexity. It's sort of like wearing a fat suit, in that your clothes suddenly become smaller, and you have to adjust your car seat because you're suddenly closer to the steering wheel. It's also hot, because essentially you're wearing an insulated vest.
There are regular-looking jackets available that have had kevlar sewn into the lining. I'm not sure how they fit, but you might want to try one of those. It would be far more practical than trying to conceal it under your existing clothing, and shouldn't interfere too much with a concealed firearm. Obviously this is useful only at a time of year when you can have a reason to wear a jacket - but you wouldn't be able to conceal a vest under a t-shirt, anyway.
From a legal perspective, a ballistic vest may be asking for trouble. Say you shoot a perp in a borderline situation, and later have to explain to the judge why you were wearing body armor. It'll sound fishy in court any way you tell it. You've already got a strike against you because most members of the public aren't familiar with the idea of CWP, or open carry for that matter. Then they find out you were wearing body armor...well, that sounds like you were actively looking for trouble.
It's probably not a bad thing to have around, just in case - and you could wear it for a short period of time, or in a TEOTWAWKI situation. A more practical and all-around comfortable option might be to take some steel and armor your car in several strategic places - doors, backs of seats, trunk, etc - and just keep it light enough so it's not a burden on gas. You don't have to remember to put it on or physically carry it around, and it will protect others as well. If you're feeling ambitious, there is some relatively thin ballistic glass for cars available now.