I think you're probably right in assuming your Mum's got dementia. I'm so sorry for you, wanting to help but unable to because you're not allowed to.
If you were in the UK, we'd probably be advising that you talked to the GP, explaining that you realised he / she couldn't discuss your Mum's condition BUT you wanted to say what you'd noticed in your own experiences of interacting with your Mum recently.
You might be able to open up the conversation with the doctor - and the doctor's thinking - by asking in "humble" patient mode "when my Mum does X, Y or Z [pick behaviours very typical of dementia patients], how should I react to her to calm her down / safeguard the relationship between us?"
At least your brother's involved (even if not helpfully involved) and at least he's talking to you. It could all be worse. Clearly it's still a very bad situation though ... and it isn't one that'll be at all easy to put right.
Please do your best to look after yourself, it's a ghastly time you're going through and it will be so difficult to stay strong and calm.
