imagine he is a lady, and she was once married to your father for a short period of time. then she left your father to marry a Spanish bloke (Miguel), and they spent a long time together, having a right old time. then she moved in with a bit of a shady italian guy (Roberto).
one day you hear that she is interested in getting together with you. let's, for a second, say you get with her. in conversation with your family and friends would she be described as Miguel's ex-wife, or Roberto's ex-wife?
no; she would be "your dads ex-wife" ... "you are fucking your dads ex-wife" thats how it will be seen. and i dont want to fuck my dads ex-wife (metaphorically).