This so nearly happened to me as a driver recently, near the road works by the airport relief road, initially I was furious with the cyclist as he swerved into my path avoiding a pot hole, I had to really slam on the anchors to avoid him. I then thought about it and came to the conclusion that had I been in the same boat as the cyclist I'd have done the same - the alternative of hitting it and going arse over tit over the handlebars is not a good option.
Therefore; either the council has to get it's act together to get these things filled in. or cycling should, somehow, be restricted to suitable routes - A whole new debate, my view? Roads are not recreational spaces.