Arbuscular mycorrhizal (AM) fungi, a group of widespread fungal symbionts of crops, could be important in driving crop yield across crop rotations through plant–soil feedbacks (PSF). However, whether preceding crops have a legacy effect on the AM fungi of the subsequent crop is poorly known. We set up an outdoor mesocosm crop rotation experiment that consisted of a first phase growing either one of four pre-crops establishing AM and/or rhizobial symbiosis or not (spring barley, faba bean, lupine, canola), followed by an AM crop, winter barley. After the pre-crop harvest, carbon-rich organic substrates were applied to test whether it attenuated, accentuated or modified the effect of pre-crops. The pre-crop mycorrhizal status, but not its rhizobial status, affected the richness and composition of AM fungi, and this difference, in particular community composition, persisted and increased in the roots of winter barley. The effect of a pre-crop was driven by its single symbiotic group, not its mixed symbiotic group and/or by a crop-species-specific effect. This demonstrates that the pre-crop symbiotic group has lasting legacy effects on the AM fungal communities and may steer the AM fungal community succession across rotation phases. This effect was accentuated by sawdust amendment, but not wheat straw. Based on the previous observation of decreased crop yield after AM pre-crops, our findings suggest negative PSF at the level of the plant symbiotic group driven by a legacy effect of crop rotation history on AM fungal communities, and that a focus on crop symbiotic group offers additional understanding of PSF.