Is it fair to have a lover outside marriage?
You use the words 'outside marriage' which implies it is a secret (else it would be 'another lover inside marriage')
It's not OK to have secrets from your partner that could affect the marriage. It is the dishonesty that is destructive.