Advice on Maths set teacher who I am struggling with

I hate to post about this because I don't want to sound like I'm whinging but would love some advice. DS is 9 in year 4 and has problems with speech, is dyspraxic and hypermobile. He has been given some extra time in English writing exams (not anything else). He is very happy at school and we have had no complaints at the school since he started in nursery. I have an issue here which I feel a bit embarrassed raising but it is kind of bugging me.

I have a problem with ds's maths teacher. I have to pick ds up one afternoon straight after Maths every week to take him to OT (off site) and then bring him back for his afternoon lessons. His classroom is a mobile classroom in the school playground and the very first time I picked him up some of the children saw me waiting (it was already 5 mins after the class should have finished and there were a number of children playing in the playground) and it seems they started saying, 'ooh look there's ds's mummy in the playground' or something to that effect. At this point ds's maths teacher stormed out and said quite rudely, "the lesson isn't over. Wait."

Not only was she rude but also mistaken as I was standing a good 15 metres away from the classroom, not close by so obviously one child had spotted me and then others had taken it up. When ds came out he was a bit upset with me and said, "mummy can you make sure Miss S doesn't see you when you come to get me."

Sorry for the long back story but I'm getting to the point! So anyway since then I have been picking ds up every week and dropping him back in the afternoon. I now wait around the corner out of sight of the classroom and then come out when I see the children leaving so that no-one can spot me from a window. Miss S is never friendly and ds has told me that he is scared of getting any questions wrong as she shouted at a boy for not understanding (he has never said this about any other teachers in the school). He also says that she is stressed because she has her own children and that he feels a bit sorry for them (I don't know who told him this!)

We were supposed to have parent's evening last night but she called yesterday and asked to reschedule as I had booked in for a late slot (I had to take ds for after-school physio so had booked in for a 18:00 slot and I think the last person before me was at 17:15). She said that she only works four days a week between 9-3:30 and since she had 'seen me hanging around the school during the day' could I come and meet her then. I just found it a bit cheeky of her to cancel at the last minute but as I was running around like a headless chicken getting ds to physio and then back to school for the meeting I agreed to meet her after drop off today.

In the meeting I found her manner quiet condescending, she told me ds got the second highest mark in the recent exams in his set (a 4a) but she was marking him a level down because he hesitates to answer questions in class and seems nervous. I explained to her that ds has a number of issues that the school is aware of and the hesitation in his speech is something that he is getting speech therapy for (which I know she has been informed of via the SENCO). She then said in a way I felt was condescending, "is he your only child? He is such a sensitive boy." Which sounded like a criticism for some reason.

The thing is ds has some major physical difficulties - he can't do most sport and he's an excruciatingly slow writer (he is typing now so that's fine). He has always been a very polite, quiet but happy and smiley little boy and other teachers have always described him as gentle and just lovely. It took us a long time to get a diagnosis for him because he is extremely stoical and never complains, just gets on with things. I find this teacher quite trunchball-esque and honestly I'm scared of her. He is doing well academically and under normal circumstances I'd be happy for him stay in the middle set which he's in but since his exam results are good enough for the top set and he is frankly scared of this teacher, I was thinking about asking to move him as the teacher for that set is his class teacher who knows him very well.

Sorry for the incredibly rambling nature of this post but any advice appreciated. I don't want to make things worse for ds by winding Miss S up but equally I'm sure his hesitation and lack of confidence in class, which she has marked him down for, is a direct result of the fact that he feels uncomfortable with her and that he would achieve better with someone else.