The graph-based representation of material structures, along with deep neural network models, often lacks locality and requires large datasets, which are seldom available in specialized materials research. To address this challenge, we developed a more data-efficient center−environment (CE) structure representation that incorporates a predefined attention-focused mechanism. This approach was applied in a machine learning (ML) study to examine the local alloying effects on the structural stability of Nb alloys. In the CE feature model, the atomic environment type (AET) method was utilized, which effectively describes the low-symmetry physical shell structures of neighboring atoms. The optimized ML-CE_AET models successfully predicted double-site substitution energies in Nb with a mean absolute error of 55.37 meV and identified Si−M pairs (where M = Ta, W, Re, and lanthanide rare-earth elements) as promising stabilizers for Nb. The ML-CE_AET model’s good transferability was further confirmed through accurate prediction of untrained alloying element Nb. Significantly, in cases involving small datasets, non-deep learning models with CE features outperformed deep learning models based on graph features reported in the literature.