The doctor sounds like a shitty man. I hate shitty people. I get why married folks have affairs outside their marriage.

However, if the person you are cheating with isn't comfortable with the affair, the wisest thing to do is to let them go not persist in having your way.