Edge cracking is one of the most serious problems in the rolling process of magnesium alloy sheets, which limits its application. In this work, the edge cracking behavior of different initial AZ31 alloy sheets, including as-cast (AC), as-rolled (AR) and as-extruded (AE), was systematically investigated and compared under the online heating rolling (O-LHR) process with a single-pass reduction of 50% at 250 °C. The results show that both AC and AR sheets exhibit severe edge cracking behavior after the O-LHR. Among them, the AR sheet exhibits the severest edge cracking behavior on the rolling plane (RD-TD) and longitudinal section (RD-ND), which is attributed to the strong basal texture and extremely uneven microstructure with shear bands. While no visible edge crack appears in the AE rolled sheet, which is mainly related to the tilted texture and the more dynamic recrystallization during rolling process. Moreover, it is also found that the micro-cracks of the AC rolled sheet are mainly generated in the local fine-grained area and the twins where recrystallization occurs. In the AR rolled sheet, micro-cracks mainly develop inside the shear bands. Meanwhile, the micro-crack initiation mechanism of AC and AR rolled sheets was also discussed.