Elementary Estimators for Sparse Covariance Matrices and other Structured Moments

Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):397-405, 2014.

Abstract

We consider the problem of estimating distributional parameters that are expected values of given feature functions. We are interested in recovery under high-dimensional regimes, where the number of variables p is potentially larger than the number of samples n, and where we need to impose structural constraints upon the parameters. In a natural distributional setting for this problem, the feature functions comprise the sufficient statistics of an exponential family, so that the problem would entail estimating structured moments of exponential family distributions. A special case of the above involves estimating the covariance matrix of a random vector, and where the natural distributional setting would correspond to the multivariate Gaussian distribution. Unlike the inverse covariance estimation case, we show that the regularized MLEs for covariance estimation, as well as natural Dantzig variants, are \emphnon-convex, even when the regularization functions themselves are convex; with the same holding for the general structured moment case. We propose a class of elementary convex estimators, that in many cases are available in \emphclosed-form, for estimating general structured moments. We then provide a unified statistical analysis of our class of estimators. Finally, we demonstrate the applicability of our class of estimators on real-world climatology and biology datasets.

Related Material

@InProceedings{pmlr-v32-yangd14,
title = {Elementary Estimators for Sparse Covariance Matrices and other Structured Moments},
author = {Eunho Yang and Aurelie Lozano and Pradeep Ravikumar},
booktitle = {Proceedings of the 31st International Conference on Machine Learning},
pages = {397--405},
year = {2014},
editor = {Eric P. Xing and Tony Jebara},
volume = {32},
number = {2},
series = {Proceedings of Machine Learning Research},
address = {Bejing, China},
month = {22--24 Jun},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v32/yangd14.pdf},
url = {http://proceedings.mlr.press/v32/yangd14.html},
abstract = {We consider the problem of estimating distributional parameters that are expected values of given feature functions. We are interested in recovery under high-dimensional regimes, where the number of variables p is potentially larger than the number of samples n, and where we need to impose structural constraints upon the parameters. In a natural distributional setting for this problem, the feature functions comprise the sufficient statistics of an exponential family, so that the problem would entail estimating structured moments of exponential family distributions. A special case of the above involves estimating the covariance matrix of a random vector, and where the natural distributional setting would correspond to the multivariate Gaussian distribution. Unlike the inverse covariance estimation case, we show that the regularized MLEs for covariance estimation, as well as natural Dantzig variants, are \emphnon-convex, even when the regularization functions themselves are convex; with the same holding for the general structured moment case. We propose a class of elementary convex estimators, that in many cases are available in \emphclosed-form, for estimating general structured moments. We then provide a unified statistical analysis of our class of estimators. Finally, we demonstrate the applicability of our class of estimators on real-world climatology and biology datasets.}
}

%0 Conference Paper
%T Elementary Estimators for Sparse Covariance Matrices and other Structured Moments
%A Eunho Yang
%A Aurelie Lozano
%A Pradeep Ravikumar
%B Proceedings of the 31st International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2014
%E Eric P. Xing
%E Tony Jebara
%F pmlr-v32-yangd14
%I PMLR
%J Proceedings of Machine Learning Research
%P 397--405
%U http://proceedings.mlr.press
%V 32
%N 2
%W PMLR
%X We consider the problem of estimating distributional parameters that are expected values of given feature functions. We are interested in recovery under high-dimensional regimes, where the number of variables p is potentially larger than the number of samples n, and where we need to impose structural constraints upon the parameters. In a natural distributional setting for this problem, the feature functions comprise the sufficient statistics of an exponential family, so that the problem would entail estimating structured moments of exponential family distributions. A special case of the above involves estimating the covariance matrix of a random vector, and where the natural distributional setting would correspond to the multivariate Gaussian distribution. Unlike the inverse covariance estimation case, we show that the regularized MLEs for covariance estimation, as well as natural Dantzig variants, are \emphnon-convex, even when the regularization functions themselves are convex; with the same holding for the general structured moment case. We propose a class of elementary convex estimators, that in many cases are available in \emphclosed-form, for estimating general structured moments. We then provide a unified statistical analysis of our class of estimators. Finally, we demonstrate the applicability of our class of estimators on real-world climatology and biology datasets.

TY - CPAPER
TI - Elementary Estimators for Sparse Covariance Matrices and other Structured Moments
AU - Eunho Yang
AU - Aurelie Lozano
AU - Pradeep Ravikumar
BT - Proceedings of the 31st International Conference on Machine Learning
PY - 2014/01/27
DA - 2014/01/27
ED - Eric P. Xing
ED - Tony Jebara
ID - pmlr-v32-yangd14
PB - PMLR
SP - 397
DP - PMLR
EP - 405
L1 - http://proceedings.mlr.press/v32/yangd14.pdf
UR - http://proceedings.mlr.press/v32/yangd14.html
AB - We consider the problem of estimating distributional parameters that are expected values of given feature functions. We are interested in recovery under high-dimensional regimes, where the number of variables p is potentially larger than the number of samples n, and where we need to impose structural constraints upon the parameters. In a natural distributional setting for this problem, the feature functions comprise the sufficient statistics of an exponential family, so that the problem would entail estimating structured moments of exponential family distributions. A special case of the above involves estimating the covariance matrix of a random vector, and where the natural distributional setting would correspond to the multivariate Gaussian distribution. Unlike the inverse covariance estimation case, we show that the regularized MLEs for covariance estimation, as well as natural Dantzig variants, are \emphnon-convex, even when the regularization functions themselves are convex; with the same holding for the general structured moment case. We propose a class of elementary convex estimators, that in many cases are available in \emphclosed-form, for estimating general structured moments. We then provide a unified statistical analysis of our class of estimators. Finally, we demonstrate the applicability of our class of estimators on real-world climatology and biology datasets.
ER -