I rarely worry about that, but it's not unusual for the conversation to go:

Me: unusual word
Them: Is that really how that's pronounced? I've never heard it spoken and always assumed it was said a different way.
Me: I'm not sure I've ever heard anyone say it, either. Let's look it up.
Them: According to the OED, either is correct, but they give mine/yours top billing.
Us: Cool!

I still think dour should rhyme with flour.