The origin of human brain wrinkling remains an open fundamental problem, with implications to neurodevelopmental disorders. Studies in polymer gel models suggest that wrinkling emerge spontaneously due to the development of compression forces during differential swelling, however these ideas have not been tested in a living system. Here, we report the appearance of surface wrinkles during in vitro development and self-organization of human brain organoids, in a micro-fabricated compartment, which supports in situ imaging over weeks. By studying the cellular dynamics, we observed nuclear nematic ordering and compression during development. Convolutions emerged at a critical nuclear density, which is indicative of a mechanical instability. We identified two opposing forces which contribute to differential growth; cytoskeletal contraction at the organoid core, and nuclear expansion during cell-cycle at the organoid perimeter. The wrinkling wavelength exhibited linear scaling with tissue thickness, consistent with an equilibrium between bending and stretching energies. Finally, lissencephalic (smooth brain) organoids displayed reduced convulsions, linear scaling with an increased prefactor, and reduced elastic modulus.