We study the problem of classifying images into a given, pre-determined taxonomy. The task can be elegantly translated into the structured learning framework. Structured learning, however, is known for its memory consuming and slow training processes. The contribution of our paper is twofold: Firstly, we propose an efficient decomposition of the structured learning approach into an equivalent ensemble of local support vector machines (SVMs) which can be trained with standard techniques. Secondly, we combine the local SVMs to a global model by re-incorporating the taxonomy into the training process. Our empirical results on Caltech256 and VOC2006 data show that our local-global SVM effectively exploits the structure of the taxonomy and outperforms multi-class classification approaches.