Yeah seriously, once you fight him back, I don't think either of you would back down. Shit might calm down, but it would be an explosive situation when your at the same party. If he attacks you out of the blue, you should defend yourself, but if he starts talking shit to you just ignore it. Generally in a situation like that the person will keep going and after a little bit everyone else will get tired of it and pissed at him, and he will feel like a douche.