As part of their efforts to control water pollution, local governments in China introduced the river chief system, whereby a named individual undertakes responsibility for protecting a specified waterway. As one of the most prominent sources of water pollution, agricultural non-point-source (NPS) pollution is becoming increasingly serious. Determining whether the river chief system, an institutional reform in China's decentralized environmental regulation regime, is effective in alleviating NPS pollution is important for the realization of green development. The effect of the river chief system on reducing agricultural NPS pollution is explored in this study using panel data from 308 Chinese counties during the period from 2004 to 2015. The results reveal that the negative impact of manure output from animal breeding operations on surface water quality is reduced with the implementation of the river chief system. However, the river chief system is ineffective in dealing with the water pollution caused by fertilizer use. Furthermore, in the current system, cooperation among river chiefs only occurs within a province. Local governments should increase their efforts in reducing fertilizer source loads and preventing fertilizer loads from entering surface waters. In addition, the central government should improve cooperation among the river chiefs in upstream and downstream provinces.