My youngest brother Alan is downs syndrome. He is 56 years old now, 2 years younger than myself.
Growing up together in Scotland was brilliant where he received almost celebrity status, the icecream man stopped outside our house and wouldn't accept payment for whatever Alan was having. Our doctor would often pop in for a cup of tea and the locals were brilliant as was the help and education Alan received from the local council.
Then we moved to England. I was 10 years' old and found myself attacking groups of people who were laughing at my brother. I got into a lot of bother with the authorities and had a reputation for violence. I was definitely scarred by some peoples' attitude towards my brother and never learnt to control my reactions till probably my late 20s' early 30s'.
I can now listen to and even laugh at jokes about downs syndrome so long as no malice is intended, depending on who tells' the joke I decide whether or not to tell them about Alan.
Alan does Elvis/karaoke in a local pub. I don't go cos to be honest his singing is dire but a few associates of mine keep an eye on him and make sure he is not abused. Also, I am still constantly on the lookout and notice when ignorant people give him so much as a sideways' look so can't be fully trusted not to react, albeit verbally now rather than physically.
Reading my post you will be able to tell that ignoring the assholes is far easier said than done johnboymartin, but you already know that.You are doing an amazing job mate, I couldn't have coped as well as you have.