Our goal in this work was to provide a principled approach to carry out kernel/metric learning in an unsupervised way, to take advantage of large datasets of unlabeled data. We investigated this research avenue by focusing mostly on histogram data (bags-of-features). Using a combination of 3 known approaches by Aitchison, Lebanon and Hinton, we were able to propose different algorithms which perform at state-of-the art level or directly outperform competing approaches.