I think that it's good to try new things, and if they don't work out, at least you've given it a go. The "Ladies Group", which I attended once, not only didn't make me feel welcome, but then discussed a book they'd read (nobody had mentioned they had a reading group), about someone with Alzheimer's!
And yes, I'd explained my position to the group leader both on the phone, and in an email. The Bereavement Group wasn't for me either, but then I enjoyed my stint with the Citizen's Advice Bureau, love my Community Choir, am starting another next week, and am so pleased I can strum a few basic ukulele chords.
Good for you for starting something new, and I'm sure your input will be much appreciated.