A general prediction model for seven heavy metals was established using the heavy metal contents of 207 soil samples measured by a portable X-ray fluorescence spectrometer (XRF) and six environmental factors as model correction coefficients. The eXtreme Gradient Boosting (XGBoost) model was used to fit the relationship between the content of heavy metals and environment characteristics to evaluate the soil ecological risk of the smelting site. The results demonstrated that the generalized prediction model developed for Pb, Cd, and As was highly accurate with fitted coefficients (R²) values of 0.911, 0.950, and 0.835, respectively. Topsoil presented the highest ecological risk, and there existed high potential ecological risk at some positions with different depths due to high mobility of Cd. Generally, the application of machine learning significantly increased the accuracy of pXRF measurements, and identified key environmental factors. The adapted potential ecological risk assessment emphasized the need to focus on Pb, Cd, and As in future site remediation efforts.