The mammary gland of male is said to be a vestigial organ. But it is a functional organ in female. So, my question is that how the mammary gland can be vestigial in male as vestigial organs are those that were functional before in our ancestors. Does it means that it was functional in the past.

Actually, if we go nitpicking here, mammary glands aren't completely inoperative in males, one could say that they are rather "dormant" - with proper hormonal treatment, certain men can even lactate. And like someone earlier said, men can also have breast cancer, which pretty much indicates the existence of minute amounts of breast tissue in males, too (in addition to the nipples).

I am not an evolutionary biologist, so the following is sort of an educated guess, but I think that during early fetal development, the underlying breast structures are "manufactured" on both female and male fetuses (like essentially all other body parts), and later things like hormonal effects dictate whether or not the mammary glands gain their functionality - after all, boys and girls have pretty much similar breast structure before puberty and the hormonal changes associated to that phase of human development.

your guess is correct. It's a question of evo-devo.
During human development from the zygote, genes located on the X chromosome and on certain autosomes direct the beginning of the development of a female. Nipples form, even rudiments of ovaries. But later on, if the embryo has a Y chromosome, the SRY gene will direct the regression of the ovaries and development of tests. from there, a boy develops. However, the nipples and rudiments of breasts have already formed.
Regards,
Andrew

"As a biologist, I firmly believe that when you're dead, you're dead. Except for what you live behind in history. That's the only afterlife" - J. Craig Venter