A mesoscale modeling methodology is proposed to predict the strain induced abnormal grain growth in the annealing process of deformed aluminum alloys. Firstly, crystal plasticity finite element (CPFE) analysis is performed to calculate dislocation density and stored deformation energy distribution during the plastic deformation. A modified phase field (PF) model is then established by extending the continuum field method to consider both stored energy and local interface curvature as driving forces of grain boundary migration. An interpolation mapping approach is adopted to transfer the stored energy distribution from CPFE to PF efficiently. This modified PF model is implemented to a hypothetical bicrystal firstly for verification and then the coupled CPFE-PF framework is further applied to simulating the 2D synthetic polycrystalline microstructure evolution in annealing process of deformed AA3102 aluminum alloy. Results show that the nuclei with low stored energy embedded within deformed matrix tend to grow up, and abnormal large grains occur when the deformation is close to the critical plastic strain, attributing to the limited number of recrystallized nuclei and inhomogeneity of the stored energy.