According to a 1998 poll of 7,441 Protestant clergy in the U.S., the following ministers said they didn’t believe in the virgin birth:American Lutherans, 19 percentAmerican Baptists, 34 percentEpiscopalians, 44 percentPresbyterians, 49 percentMethodists, 60 percentYet another poll, in 1999, surveyed 103 Roman Catholic priests, Anglican priests and Protestant ministers in the U.K. That poll found 25 percent did not believe in the virgin birth, according to ReligiousTolerance.org. A 2004 survey of ministers in the Church of Scotland found 37 percent don’t accept the virgin account.

The clergy are educated Christians and they have much more at stake in maintaining their faith as traditionally believed. We would not expect so many of them to reject the virgin birth. The fact so many of them do so shows the evidence for the virgin birth is not there.
What we know is the earliest testimonies to Jesus did not have a virgin birth story. It is not found in Q (a source of the 3 synoptic gospels), nor the apostle Paul, nor the gospel of Mark, nor the Gospel of Thomas. The evolutionary trend of the literature explains this fact since questions eventually arose as to the nature of the savior's birth.

The need to claim Jesus was born of a virgin came as the earliest Christians began to deify Jesus. If he was God in the flesh then the question to be answered was how did he make his entrance into the world, and the virgin birth narrative fit nicely. A culturally acceptable story was available for early Christians to borrow from that made a parallel case for Jesus' virgin birth, just as it was believed of Alexander the Great, as biblical scholar Richard C. Miller argues:

Another need for a virgin birth story was because of original sin. If Jesus was born naturally then he falls under the "guilt" of original sin. However, we now know from science there was never an Adam and Eve.

With no Adam & Eve there is no original sin, no need for a virgin birth, no need for an atonement on the cross either. Biblical scholar Robert Price (with Ed Suominen) explain the not so good options in the wake of this new evidence:

But just think about the supposed evidence itself. Mary could just have easily conceived out of wedlock and told a wild virgin birth tale that Joseph would have readily agreed to tell. After all, who wants a bastard child and the approbation involved? Tell instead a story where Joseph didn't believe Mary's story initially in order to make it sound more reasonable. For this reason alone we should not believe their testimony if we want to know the truth. But when that's gone what's left but some anonymous writers in gospels 90+ years later after the supposed events in question (i.e., Matthew & Luke). But then how might they objectively know Jesus was born of a virgin? Who presumably told them? The Holy Spirit? Why is it God always speaks to individuals in private subjective unevidenced whispers? Those claims are a penny a dozen and not something we can double-check based on objective evidence.

What we know is there were up to 8 million Jews in that day who did not believe Jesus was born of a virgin, 1-2 million of them living in Palestine. They believed in Yahweh, that he does miracles, and were experts in their own prophecies, yet they didn't believe. If they were there and they weren't convinced then why should we? Biblical scholar David C. Sim argues that at the end of the first century there were only 1000 Jewish Christians:

http://www.hts.org.za/index.php/HTS/article/viewFile/430/329

If they were there and didn't believe why should we? Why should anyone in our world? If your god misled the Jews in the prophecies then he purposely condemned them to hell and to the sufferings and torture handed out to them by Christians as the "Christ killers" they were. After all, if God misled them how could they believe differently?

The truth is there is not one prophecy of a virgin birth that is to be seen as a prophecy in its original context that points singularly to the virgin birth of Jesus. Not one. Just look how the gospel of Matthew treats the Old Testament, especially in the early chapters. He misinterprets and twists those prophecies several times so out of context that no one should trust him on the virgin birth either, after all, he's not quoting from the original Hebrew for young woman but a misinformed Greek translation, and he stole it completely out of context.

Just look at the context. Matthew has it absolutely wrong. Isaiah is referring to king Ahaz in his day. That's plain as day. Look for yourself and ask what YOU would say if some other cult (which Christianity was initially) claimed their messiah fulfills this prophecy:

Isaiah 7:

1 And it came to pass in the days of Ahaz the son of Jotham, the son of Uzziah, king of Judah, that Rezin the king of Syria, and Pekah the son of Remaliah, king of Israel, went up toward Jerusalem to war against it, but could not prevail against it.2 And it was told the house of David, saying, Syria is confederate with Ephraim. And his heart was moved, and the heart of his people, as the trees of the wood are moved with the wind.3 Then said the Lord unto Isaiah, Go forth now to meet Ahaz, thou, and Shearjashub thy son, at the end of the conduit of the upper pool in the highway of the fuller's field;4 And say unto him, Take heed, and be quiet; fear not, neither be fainthearted for the two tails of these smoking firebrands, for the fierce anger of Rezin with Syria, and of the son of Remaliah.5 Because Syria, Ephraim, and the son of Remaliah, have taken evil counsel against thee, saying,6 Let us go up against Judah, and vex it, and let us make a breach therein for us, and set a king in the midst of it, even the son of Tabeal:7 Thus saith the Lord God, It shall not stand, neither shall it come to pass.8 For the head of Syria is Damascus, and the head of Damascus is Rezin; and within threescore and five years shall Ephraim be broken, that it be not a people.9 And the head of Ephraim is Samaria, and the head of Samaria is Remaliah's son. If ye will not believe, surely ye shall not be established.10 Moreover the Lord spake again unto Ahaz, saying,11 Ask thee a sign of the Lord thy God; ask it either in the depth, or in the height above.12 But Ahaz said, I will not ask, neither will I tempt the Lord.13 And he said, Hear ye now, O house of David; Is it a small thing for you to weary men, but will ye weary my God also?14 Therefore the Lord himself shall give you a sign; Behold, a virgin shall conceive, and bear a son, and shall call his name Immanuel.15 Butter and honey shall he eat, that he may know to refuse the evil, and choose the good.16 For before the child shall know to refuse the evil, and choose the good, the land that thou abhorrest shall be forsaken of both her kings.17 The Lord shall bring upon thee, and upon thy people, and upon thy father's house, days that have not come, from the day that Ephraim departed from Judah; even the king of Assyria.18 And it shall come to pass in that day, that the Lord shall hiss for the fly that is in the uttermost part of the rivers of Egypt, and for the bee that is in the land of Assyria.19 And they shall come, and shall rest all of them in the desolate valleys, and in the holes of the rocks, and upon all thorns, and upon all bushes.20 In the same day shall the Lord shave with a razor that is hired, namely, by them beyond the river, by the king of Assyria, the head, and the hair of the feet: and it shall also consume the beard.21 And it shall come to pass in that day, that a man shall nourish a young cow, and two sheep;22 And it shall come to pass, for the abundance of milk that they shall give he shall eat butter: for butter and honey shall every one eat that is left in the land.23 And it shall come to pass in that day, that every place shall be, where there were a thousand vines at a thousand silverlings, it shall even be for briers and thorns.24 With arrows and with bows shall men come thither; because all the land shall become briers and thorns.25 And on all hills that shall be digged with the mattock, there shall not come thither the fear of briers and thorns: but it shall be for the sending forth of oxen, and for the treading of lesser cattle.