There has been an active debate whether global warming will result in a net gain or net loss for United States agriculture. With mounting evidence that climate is warming, we show that such warming will have substantial impacts on agricultural yields by the end of the century: yields of three major crops in the United States are predicted to decrease by 60 to 79% under the most rapid warming scenario. We use a 55-year panel of crop yields in the United States and pair it with a unique fine-scale weather data set that incorporates the whole distribution of temperatures between the minimum and maximum within each day and across all days in the growing season. The key contribution of our study is in identifying a highly non-linear and asymmetric relationship between temperature and yields. Yields increase in temperature until about 29° C for corn and soybeans and 33° C for cotton, but temperatures above these thresholds quickly become very harmful, and the slope of the decline above the optimum is significantly steeper than the incline below it. Previous studies average temperatures over a season, month, or day and thereby dilute this highly non-linear relationship. We use encompassing tests to compare our model with others in the literature and find its out-of-sample forecasts are significantly better. The stability of the estimated relationship across regions, crops, and time suggests it may be transferable to other crops and countries.