An abnormal texture with c axis of the grains parallel to extrusion direction (ED) was found in extruded Mg–Y-Sm-Zn-Zr alloy. The mechanisms for the formation of this abnormal texture were investigated based on the dynamic recrystallization (DRX) mechanisms and deformation modes during extrusion using electron backscatter diffraction (EBSD) and a viscoplastic self-consistent (VPSC) model. The microstructure evolution during extrusion indicated that the abnormal <0001>//ED texture was dominated by DRX grains. With the strain increasing, the intensity of this texture enhanced. Based on the EBSD results analysis, discontinuous dynamic recrystallization (DDRX) played the dominated role in nucleation of the new grains at the initial stage and then continuous dynamic recrystallization (CDRX) was activated at high strain. The formation of abnormal <0001>//ED texture was attributed to the activation of <c+a> slips and it could promote the rotation of c-axis of grains to ED, which contributed to the formation of <0001>//ED texture. The simulated texture predicated that the CRSS of basal slip was higher than that of pyramidal <c+a> slip, which played an important role in contributing the formation of the <0001>//ED texture component.