For the best?

I’ve been thinking about breaking up with my therapist. I’ve gone 4 times now and have next week off. I don’t feel like I’ve made any progress, I actually feel like each session is making me worse. I went in looking for help with panic attacks and ended up going after the deep dark corners that I have well hidden, and for good reason. I’m going to tell him that I don’t want to delve into the past anymore and I just want to focus on the panic attacks. When I started, I wasn’t having too many problems, just the anxiety problems. Aside from that, I was generally very happy. Now, after 4 sessions I feel like I’m on the verge of a psychotic break, I’m either angry or depressed all the time and I’m only happy when I’m sleeping and I feel generally hopeless and despondent. I feel like if I stopped therapy I could get back to being generally happy again.