It's not easy, and it's totally understandable. Like grumpywitch said, it's hard to be the lone voice of reason in a situation like that, especially when it's people you work with and have to have a lot of interaction with.

It's not always a matter of strength - it's hard to think fast sometimes when you're caught off guard and the comment is really unexpected.