The sparse representation-based classification algorithm has been used for human face recognition. But an image database was restricted to human frontal faces with only slight illumination and expression changes. Cropping and normalization of the face needs to be done beforehand. This paper uses a sparse representation-based algorithm for generic image classification with some intra-class variations and background clutter. A hierarchical framework based on the sparse representation is developed which flexibly combines different global and local features. Experiments with the hierarchical framework on 25 object categories selected from the Caltech101 dataset show that exploiting the advantage of local features with the hierarchical framework improves the classification performance and that the framework is robust to image occlusions, background clutter, and viewpoint changes.