Kinaya means using words that are not exclusively prescribed for issuing divorce, but alludes and hints to it. Kinaya words do not effect divorce unless one intends divorce by them or it is determined by the circumstance one is in. (Hidayah, V2, P373, Raddul Muhtaar, V2, P501, Hindiyah, V1, P374)

With regards to your question, the phrase you said falls into the second type of ambiguous and allusive words (kinaya). Therefore, if you intended to give talaq, one talaq baain will occur. Otherwise, no talaq will occur. (Fatawa Darul Uloom Deoband p.400 v.9)

The same ruling will also apply if you say the same thing in the absence of your wife.