With too much responsibility comes paralysis
To what degree is it your responsibility to make sure other people don’t end up in an uncomfortable situation?
Certainly you should be considerate of others’ feelings and not act like a jerk. But at some point, the consequences are out of your hands. Should you really be responsible for anticipating and avoiding every conceivable terrible future for everything you ever do?
“This all seemed really unlikely” should be a more convincing explanation for your choices than it usually is.