Changes in the global production of major crops are important drivers of food prices, food security and land use decisions. Average global yields for these commodities are determined by the performance of crops in millions of fields distributed across a range of management, soil and climate regimes. Despite the complexity of global food supply, here we show that simple measures of growing season temperatures and precipitation--spatial averages based on the locations of each crop--explain about 30% or more of year-to-year variations in global average yields for the world's six most widely grown crops. For wheat, maize, and barley, there is a clearly negative response of global yields to increased temperatures. Based on these sensitivities and observed climate trends, we estimate that warming since 1981 has resulted in annual combined losses of these three crops representing roughly 40 MT or $5 billion per year, as of 2002. While these impacts are small relative to the technological yield gains over the same period, the results demonstrate already occurring negative impacts of climate trends on crop yields at the global scale.