Metric Learning with Adaptive Density Discrimination

Oren Rippel, Manohar Paluri, Piotr Dollar and Lubomir Bourdev

We propose an approach to address a number of subtle yet important issues which have stymied earlier distance metric learning algorithms. It maintains an explicit model of the distributions of the different classes in representation space. It employs this knowledge to adaptively assess similarity, and pursue local discrimination by penalizing class distribution overlap. This idea allows us to surpass existing approaches by a significant margin on a number of tasks such as classification, training efficiency and representation saliency.

Spectral Representations for Convolutional Neural Networks

Oren Rippel, Jasper Snoek, and Ryan P. Adams

We argue that, beyond its advantages for efficient convolution computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks. We propose spectral pooling, which preserves considerably more information per parameter than other pooling strategies and enables flexibility in the choice of pooling output dimensionality. We also demonstrate the effectiveness of complex-coefficient spectral parameterization of convolutional filters, which leads to significantly faster convergence during training.

Scalable Bayesian Optimization Using Deep Neural Networks

We explore the use of neural networks as an alternative to Gaussian processes to model distributions over functions. While this approach performs competitively with state-of-the-art GP-based approaches, it scales linearly with the number of data rather than cubically. This allows us to achieve a previously intractable degree of parallelism, which we use to rapidly search over large spaces of models.

MICMat Kernel Library for Xeon Phi

I developed and maintained the MICMat (MIC Matrix) kernel library, which enables interfacing with Intel's Xeon Phi Coprocessor directly from pure Python. It presents an extensive library of primitives, optimized for high performance computation, while allowing very convenient development.

Learning Ordered Representations with Nested Dropout

Oren Rippel, Michael Gelbart and Ryan P. Adams

We study ordered representations of data in which different dimensions have different degrees of importance. To learn these we introduce Nested Dropout. We rigorously show that the application of nested dropout enforces identifiability of the units. We use the ordering property to construct data structures that permit retrieval in time logarithmic in the database size and independent of the dimensionality of the representation. We also show that ordered representations are a promising way to learn adaptive compression for efficient online data reconstruction.

Avoiding Pathologies in Very Deep Networks

David Duvenaud, Oren Rippel, Ryan P. Adams and Zoubin Ghahramani

To help suggest better deep neural network architectures, we analyze the related problem of constructing useful priors on compositions of functions. We study deep Gaussian process, a type of infinitely-wide, deep neural net. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Finally, we characterize the model class you get if you do dropout on Gaussian processes.

High-Dimensional Probability Estimation with Deep Density Models

Oren Rippel and Ryan P. Adams

We introduce the deep density model for density estimation. We exploit insights from deep learning to construct a bijective map to a representation space, under which the transformation of the distribution of the data is approximately factorized and has identical and known marginal densities. The simplicity of the latent distribution under the model allows us to feasibly explore it, and the invertibility of the map to characterize contraction of measure across it. This enables us to compute normalized densities for out-of-sample data.