Detecting the presence of hazardous materials in luggage is an important problem in aviation security. The current generation of inspection systems is based on X-ray computed tomography, followed by recognition systems to identify potential prohibited materials. As such, the image formation algorithms are designed independently of the recognition algorithms. In this paper, we present a new class of algorithms for processing the X-ray data by simultaneously forming images from the collected X-ray observations and identifying the underlying materials in the images. These algorithms exploit information about the possible materials in the image to modify the image reconstruction techniques, as well as material identification. We evaluate our joint algorithm on simulated phantoms using multi-spectral computed tomography, and compare our reconstruction and classification results with alternative state of the art approaches. Our experiments indicate that there are significant improvements in recognition performance possible through our joint approach.