Rifts that form at the fronts of floating ice shelves are fractures that cut through the entire thickness of the ice. They are believed to be the precursor of calving, which accounts for a significant part in the mass loss of present ice sheets. Here we investigate the formation of rifts in ice shelves and their evolution by combining laboratory-scale experiments of ice sheets together with theoretical modeling. Experimentally we model the deformation of ice using a thin film of non-Newtonian fluid that is driven axisymmetrically by buoyancy. The viscous fluid intrudes a bath of an inviscid, denser fluid that represents the ocean. Consequently, the circular symmetry of the propagating front breaks up near the grounding line into a set of tongues with a characteristic wavelength that coarsens over time, a pattern that is reminiscent of some ice rifts. Theoretically, we model the formation of rifts as a hydrodynamic instability of a power-law fluid. Our model resolves the formation of rifts and the coarsening of the characteristic wavelength, and predicts coarsening transition times that are consistent with our experimental measurements. We discuss the instability mechanism and its implications.