The use of sewage sludge (SS) and solid waste composts in agriculture is considered as one of the best disposal options. In this study, impact of different composts and SS was evaluated on the soil quality, growth, and yield performance of rice and wheat crops. Four types of composts and SS were collected from different sources and applied in soil at 0.5% (5 g kg−1 soil) and 1.0% (10 g kg−1 soil) in comparison with chemical fertilizer (no SS/compost). The application of all types of composts at all the tested rates increased the straw and grain yields of both the crops compared with control. Nevertheless, application of Lahore compost and all SS (1.0%) resulted in a significant increase in the concentration of cadmium (Cd), copper (Cu), lead (Pb), and zinc (Zn) in plant parts. The Cd concentration in rice grains with Lahore compost and Kasur SS applied at the rate of 1.0% was above the safe limit. The trace elements in wheat grains with all composts and SS at all the tested rates remained within safe limits. The release of amendment bound trace elements in soil solution was highly dependent upon dissolved organic carbon and concentration of these trace elements especially for Cu and Zn. Application of SS at low level (0.5%) and compost (except Lahore compost) at both levels could be a better strategy to exploit their benefits in terms of crop performance and soil quality. However, application of SS and compost at high rates may lead to accumulation of trace elements in rice grains limiting its suitability for human consumption.