i was asking myself if it would have been better for Berserk to start directly from The Golden Age, skipping the black swordsman's first three numbers.

No, it wouldn't have been better. The Black Swordsman arc lays out the foundation for the entire series. It explains every core concept in Berserk. It also gives the reader a reason to care about the character of Guts and the world he lives in.