"Per the US Supreme Court if they aren't fighting words, ie threats, it's in your best legal interest to walk away."
Definitely walk away (and don't turn your back). But keep in mind that "fighting words" do not have to be explicit threats ("I'm going to smash your face"). Words obviously intended to provoke a fighting response ("You stupid dirty n****r") are different from mere insults ("Nice parking job, idiot").
I think the question is, If a grievous insult to my wife constitutes fighting words (and it may well be that it does), what kind of response is justified and won't be considered escalation? If you say, "Hey, that's my wife. Shut up," and stand there waiting for a response, and then the situation gets physically violent, will the fact that you responded militate against you? I'd probably respond somehow but keep walking. In court, I want witnesses saying, "The guy was walking away," not "They both got into it."
Still, who wants to spend 20 grand to prove he's in the right? To me, the threat of extreme violence would have to be obvious before I even thought about touching that gun.