A fact is just that, a mere fact.
An acknowledged fact asks for an interpretation, otherwise the human mind finds it hard to accept its very existence.

An interpretation that seems to make sense becomes an understanding and regardless of that understanding being right or wrong it generates a belief.
Until that understanding is proven wrong and even then… eventually a new understanding is generated and, in its turn, it leads to a new belief.

That’s why we should indeed reserve the right Patton Oswalt speaks about and then use it sparingly, only when other believers tries to forcefully impose their beliefs on us.

In fact Oswalt is right. We don’t have to respect other people’s beliefs, only their right to have their own beliefs.