Help with my virgo man

i have known this virgo man for 6 yrs and in that time i have always known that he was chasing me as he told me problem is he has a partner and children and i was married at the time now i am single and over the past 4 mths he has been on the scene just past 3 weeks we have got together he has been lovely and although i know it is wrong to be meeting him i cant help myself everytime he phones i have butterflies he says all the right things and now he is sayin that he dosent know what to do he thinks he is falling in love with me i have slept with him once but past 3 nights he has been to meet me and we miss each other when we are apart he has asked me to tell him what i want and what to do but its to early for me but he says he would leave his partner tomorrow if its what i want i just dont ant to hurt anyone but i cant help but want him i know he is wrong for cheating on his partner and he knows to but sayin he is beyond caring cause he cant stop thinking about me what should i do im a cancer women