I have my Betta in a Aqueon Mini Bow 2.5 Gallon. I have a heater and it came with a filter. I have noticed lately my Betta has been being pushed down by the filter. I even baffled it and he was still stressed. I turned off the filter and saw a drastic improvement in a matter of minuets! I know I would have to do more water changes and I'm fine with that!
Currently, I do two 50% water changes a week with a gravel vacuum. Without a filter I would do one 100% one 50% and one 25% water change every week. All with a gravel vacuum.
So my questions are...
Can i take out the filter? Will the heater be affected by removing the filter? Anything else i should know about?

I don't have filters in my 2.5 and 3. I just do lots of water changes and gravel vacuuming. No problems so far. My guys hated the filters too. I even tried a sponge filter, the "small" one and it took up a quarter of the 3 gallon and looked terrible.

You don't need the filter, but for the cheapest route if you wanted the filter you could shave the impeller blades down to further reduce output or install a rheostat to adjust the input voltage. Replace the filters with the matted filter floss cut to size, this could also reduce flow and will help clean the tank water between water changes.

Having some circulation of the water helps keep the temperature evenly circulated throughout the tank.

I have the same tank and noticed the same issue with my betta. Poor little guy would only hang out over by the filter behind it where there was no disturbance in the water. I unplugged it and he's super happy now, swimming all around and coming to greet me (and my roommate) when we come to his tank (or are returning DVDs in the case of my roommate, but it made her happy).

I am looking into a filter like the one rlw suggested or another one I saw suggested on this site so I don't have to do water changes more than twice a week but for now I'm just letting Dedalus chill.

(I do dislike that the 2.5 I got advertised being a betta tank and the filter is way too strong for bettas)

Already mentioned, but yes you will have warmth near the heater which will hit temp quickly and shut off leaving the remaining water in the tank closer to room temp than ideal preset temp. This could potentially shorten the life of your heater due to frequent cycling on and off.

If you have no plans for your filter, take the impeller out and use a file to slowly file the impeller blades down. Test as you go until the current is minimal. The replacement impeller shouldn't be that costly if you mess up or decide later that you'd like to restore full flow rate.