Hourly measurements of water-soluble inorganic ionic species in ambient atmospheric particles were conducted at Shanghai, Hangzhou, and Guangzhou sampling sites in China during the period of 2009–2011. The relation between sulfate and nitrate in particulate matter (PM10 and PM2.5) was examined based on these measurements. Results showed that the mass fraction of sulfate was strongly negatively correlated with that of nitrate in atmospheric particles on most of the sampling days, especially when sulfate and nitrate made up the vast majority of the total soluble anions and cations (Na+, K+, Ca2+, and Mg2+) made a small contribution to the total water-soluble ions, revealing that the formation mechanisms of sulfate and nitrate in the atmosphere are highly correlated, and there exists a significant negative correlation trend between sulfate and nitrate mass fractions in the atmospheric particles. We found that local meteorological conditions presented opposite influences on the mass fractions of sulfate and nitrate. Further analysis indicated that the two mass fractions were modulated by the neutralizing level of atmospheric aerosols, and the negative correlation could be found in acidic atmospheric particles. Strong negative correlation was usually observed on clear days, hazy days, foggy days, and respirable particulate air pollution days, whereas poor negative correlation was often observed during cloud, rain, snow, dust storm, and suspended dust events. The results can help to better understand the formation mechanisms of atmospheric sulfate and nitrate during air pollution episodes and to better explain field results of atmospheric chemistry concerning sulfate and nitrate.