Since your preference is the larger caliber, that is what I would carry. You can always reload (assuming you carry the extra ammo) but you CANNOT increase your impact size during the fight.
That said, it comes back to balance. I generally only carry a .380 because of concealment issues however, my largest caliber that I own is only a 9. If I truly felt more comfortable with a larger caliber then I would get one, to me it mentally does not make a difference. But you must carry what YOU believe is YOUR best round... so for you it sounds like the .44