Capacity is legally defined and medically certified so just being irrational doesn't qualify. The charity might also argue that if she has capacity she can do what she wants with her moneyI would hate to see the charity get away with this. Even if the dementia hadn't been mentioned they should give it back (in my opinion) but the fact that they knew about the diagnosis puts them in a very bad light. I would be shouting it from the rooftops to anyone who would listen.
There are ethics involved here. They are not allowed to accept money from criminals or anyone who lacks capacity and by her very actions your mother has confirmed that she lacks capacity.
I do have some sympathy for your poor dad though.