Further to my previous post, Id like to point out that there are ways of saying "no" to him
If you say "no, Im not going to" then this is bound to make him angry - he wants to go and cannot see any reason whey he should not, so in his mind you are just being selfish and unreasonable......
Try using "love lies" and starting the sentence with "yes", even though you are actually saying no...... Say - "yes, that would be nice, but its a bank holiday and all the shops are shut/they are doing road works and the roads are closed/theres been a huge accident and massive traffic jams/the car is broken and needs repairs/etc" - choose whichever best suits your purposes.
You may have tried all of this, and if he still gets angry cut the visit short.