Pneumonia is a severe inflammatory condition of the lungs that leads to the formation of pus and other liquids in the air sacs. The disease is reported to affect approximately 450 million people across the world, resulting in 2 million pediatric deaths every year. Chest X-ray (CXR) analysis is the most frequently performed radiographic examination for diagnosing the disease. Unlike pneumonia in adults, pediatric pneumonia is poorly studied. Computer-aided diagnostic (CADx) tools aim to improve disease diagnosis and supplement decision making while simultaneously bridging the gap in effective radiological interpretations during mobile field screening. These tools make use of handcrafted and/or convolutional neural networks (CNN) extracted image features for visual recognition. However, CNNs are perceived as black boxes since their performance lack explanations and poorly understood. The lack of transparency in the learned behavior of CNNs is a serious bottleneck in medical screening/diagnosis since poorly interpreted model behavior could unfavorably impact decision-making. Visualization tools are proposed to interpret and explain model predictions. In this study, we highlight the advantages of visualizing and explaining the activations and predictions of CNNs applied to the challenge of pneumonia detection in pediatric chest radiographs. We evaluate and statistically validate the models’ performance to reduce bias, overfitting, and generalization errors.