Both the visual and the haptic senses play important roles in the everyday perception of materials. However, how the two senses compare in material perception tasks and how they influence each other in the emergence of a common representation of materials is largely unknown. We have recently compared material property ratings following visual exploration of materials with ratings following haptic exploration and found that participantsâ€™ judgments of material properties are very similar, independent of the type of exploration (Baumgartner, Wiebel, & Gegenfurtner, 2013). However, when we asked our observers to categorize the different material samples, their performance was better with visual exploration rather than haptic exploration. To explore the effect visual experience has on material perception, we asked congenitally blind participants to explore different materials haptically and rate several material properties (roughness, elasticity, hardness, three-dimensionality, friction, orderliness, and temperature). In addition, we asked them to categorize our materials into one of eight categories (plastic, paper, fabric, fur, leather, stone, metal, and wood). Principal components analyses were conducted on sighted participantsâ€™ haptic rating data as well as blind participantsâ€™ haptic rating data. A procrustes analysis showed that the two principal component spaces were highly similar. Categorization performance was also comparable for the two groups. (sighted participants: 66.6%, congenitally blind participants: 68.6%). We conclude that the representational space of materials we have observed can be formed without the influence of visual experience.