The conversion of tropical forests into agriculture reduces biodiversity dramatically. However, species might differ in their responses, depending on their habitat specialisation and geographic origin. In this study, we assess how butterfly assemblages differ between old-growth forests, forest fragments, forest-derived vanilla agroforests, fallow-derived vanilla agroforests, woody fallows, herbaceous fallows, and rice paddies in Madagascar. We recorded 88 butterfly species, of which 65 species are endemic to Madagascar. Land-use types with woody vegetation sustained many endemic (mean: 6.8 species) and forest butterfly species (mean: 4.8 species). Rice paddies and herbaceous fallows were richer in open-land species (mean: 7.6 species) and poorer in forest species (mean: 1.7 species) compared to other land-use types. Compared to herbaceous fallows, fallow-derived vanilla agroforests hosted more endemic (+164%) and forest (+239%) species. Richness of open-land species in forest-derived vanilla agroforests was six times higher than in forest fragments. Overall, 27% of species occurred exclusively in one land-use type and 19% of all species when old-growth forests were excluded. We found the highest number of exclusive species in forest-derived vanilla agroforests. We conclude that all studied land-use types contribute to butterfly conservation in our study region. Especially the woody vegetation in forest fragments, vanilla agroforests, and woody fallows supports a high butterfly diversity and is pivotal for maintaining a broad diversity of forest butterflies in the agricultural matrix. Our study highlights the importance of preserving the diversity of small-scale land-use types, including agroforestry, forests and fallow land in this tropical biodiversity hotspot.