Im sorry you are finding it so tough, but it takes a while to settle - at least a couple of months. My mum spent the first 6 weeks "packing to go home" but she settled really well eventually.I'm really struggling this week,he's been there 2 weeks tomorrow and the visits are no easier,he doesn't seem to have settled at all yet,
Honestly, if you are feeling that visiting is not doing either of you any good then it is perfectly OK not to go.
Have you asked the staff what he is like when you are not there? Often, seeing their relative is the trigger to remind them of home and the rest of the time they are OK.
Just hang in there. You know you cant go back, but this knowledge hasnt yet gone from your heart to your heart. It will take time for you to adjust too.