Aside from any doubts based purely on biology, there are many reasons to believe the "virginity" label was added as a second thought. In the time when Yeshua was born (First century BC), women had a very low standing in society. Menstruating and pregnant women were considered so impure that all gods of the time (including the Roman emperors who declared themselves divine) had virgin mothers. This includes the Buddha, Perseus, Hephaestus, Karna, Osiris, Horus, Ra, Bacchus, Hercules, etc.
The first mention of Mariam's virginity is from the Gospel of Matthew, written more than 70 years after Yeshua passed away. Neither Mark, John nor Paul mention a virgin birth. The Marian cult rose under Eusebius (260-340 AD) and Emperor Constantine (280-337 AD), when Christianity became a state religion and the Romans saw a political need to establish a competitor to the important Isis cult. Mother Goddess Mary replaced the Mother Goddess Isis. [source: The Court-Martial of Jesus by Weddig Fricke]
Yeshua's brothers and sisters are mentioned in several places of the New Testament: Matthew 13:55-56, John 7:1-10, Acts 1:14, and Galatians 1:19. Roman historian Josephus and Pope Clement I also confirm in their writings that Yakov, the brother of Yeshua, was the leader of the first assembly (ecclesia, or church) after Yeshua passed away.