They shouldn't act against you, as in acting for you in the past they are aware of some confidential information about you that is relevant to the current matter. You should point this out to the firm and they should agree to act no further.