I don't mind admitting that in the early weeks and months I would often dread visiting, because of the endless, 'Have you come to take me home? I want to go home! etc., not to mention anger/accusations that we'd only put her there because we were after her money - a joke considering the CH fees! But of course there was no point in saying that, though I did just once resort to it.
It did get considerably easier eventually, though I'm afraid to say it was only after her dementia worsened and she'd forgotten her former home altogether. Later still, she forgot who I was, and although it sounds awful to say it, and of course it was heartbreaking in one way, those visits were perhaps the easiest, since I was just 'a nice lady' who made her cups of tea and brought her chocolate.
In the earlier days, when visits were so often difficult, I found them rather easier if anyone else - particularly my OH, who had always been something of a golden boy - came too. But that was not often.
Please don't feel guilty. Many of us will have felt the same.