We investigated the effects of surface modification on the defect-related photoluminescence (PL) band in colloidal CdS quantum dots (QDs). A size-selective photoetching process and a surface modification technique with a Cd(OH)2 layer enabled the preparation of size-controlled CdS QDs with high PL efficiency. The Stokes shift of the defect-related PL band before and after the surface modification was ~1.0 eV and ~0.63 eV, respectively. This difference in the Stokes shifts suggests that the origin of the defect-related PL band was changed by the surface modification. Analysis by X-ray photoelectron spectroscopy revealed that the surface of the CdS QDs before and after the surface modification was S rich and Cd rich, respectively. These results suggest that Cd-vacancy acceptors and S-vacancy donors affect PL processes in CdS QDs before and after the surface modification, respectively.