Category Archive

The first colloquium speaker at this semester, professor Wei Zheng from IUPUI, will give a talk on “Universally optimal designs for two interference models“. In this data explosive age, people are easy to get big data set, which renders people difficult to make inferences from such massive data. Since people usually think that with more data, they have more chance to get more useful information from them, lots of researchers are struggling to achieve methodological advancements under this setup. This is a very challenging research area and of course very important, which in my opinion needs the resurgence of mathematical statistics by borrowing great ideas from various mathematical fields. However, another great and classical statistical research area should come back again to help statistical inference procedures from the beginning stage of data analysis, collecting data by design of experiments so that we can control the data quality, usefulness and size. Thus it’s necessary for us to know what is optimal design of experiments. Here is an introduction to this interesting topic.

In statistics, we have to organize an experiment in order to gain some information about an object of interest. Fragments of this information can be obtained by making observations within some elementary experiments called trials. The set of all trials which can be incorporated in a prepared experiment will be denoted by , which we shall call the design space. The problem to be solved in experimental design is how to choose, say trials , called the support points of the design, or eventually how to choose the size of the design, to gather enough information about the object of interest. Optimum experimental design corresponds to the maximization, in some sense, of this information. In specific, the optimality of a design depends on the statistical model and is assessed with respect to a statistical criterion, which is related to the variance-matrix of the estimator. Specifying an appropriate model and specifying a suitable criterion function both require understanding of statistical theory and practical knowledge with designing experiments.

We shall restrict our attention to the parametric situation in the case of a regression model, the mean response function is then parameterized as

specifying for a particular with unknown parameter .

A design is specified by an initially arbitrary measure assigning design points to estimate the parameter vector. Here can be written as

where the design support points are elements of the design space , and the associated weights are nonnegative real numbers which sum to one. We make the usual second moment error assumptions leading to the use of least squares estimates. Then the corresponding Fisher information matrix associated with is given by

where and .

Now we have to propose the statistical criteria for the optimum. It is known that the least squares estimator minimizes the variance of mean-unbiased estimators (under the conditions of the Gauss–Markov theorem). In the estimation theory for statistical models with one real parameter, the reciprocal of the variance of an (“efficient”) estimator is called the “Fisher information” for that estimator. Because of this reciprocity, minimizing the variance corresponds to maximizing the information. When the statistical model has several parameters, however, the mean of the parameter-estimator is a vector and its variance is a matrix. The inverse matrix of the variance-matrix is called the “information matrix”. Because the variance of the estimator of a parameter vector is a matrix, the problem of “minimizing the variance” is complicated. Using statistical theory, statisticians compress the information-matrix using real-valued summary statistics; being real-valued functions, these “information criteria” can be maximized. The traditional optimality-criteria are invariants of the information matrix; algebraically, the traditional optimality-criteria are functionals of the eigenvalues of the information matrix.

A-optimality (“average” or trace)

One criterion is A-optimality, which seeks to minimize the trace of the inverse of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients.

D-optimality (determinant)

A popular criterion is D-optimality, which seeks to maximize the determinant of the information matrix of the design. This criterion results in maximizing the differential Shannon information content of the parameter estimates.

E-optimality (eigenvalue)

Another design is E-optimality, which maximizes the minimum eigenvalue of the information matrix.

T-optimality

This criterion maximizes the trace of the information matrix.

Other optimality-criteria are concerned with the variance of predictions:

G-optimality

A popular criterion is G-optimality, which seeks to minimize the maximum entry in the diagonal of the hat matrix. This has the effect of minimizing the maximum variance of the predicted values.

I-optimality (integrated)

A second criterion on prediction variance is I-optimality, which seeks to minimize the average prediction variance over the design space.

V-optimality (variance)

A third criterion on prediction variance is V-optimality, which seeks to minimize the average prediction variance over a set of m specific points.

Now back to our example, because the asymptotic covariance matrix associated with the LSE of is proportional to , the most popular regression design criterion is D-optimality, where designs are sought to minimize the determinant of . And the standardized predicted variance function, corresponding to the G-optimality, is

and G-optimality seeks to minimize .

A central result in the theory of optimal design, the General Equivalence Theorem, asserts that the design that is D-optimal is also G-optimal and that

where is the treatment assigned to the plot in the -th position of the -th block, and

is the general mean;

is the direct effect of treatment ;

and are respectively the left and right neighbor effects; that’s the interference effect of the treatment assigned to, respectively, the left and right neighbor plots and .

is the effect of the -th block; and

is the random error, .

We seed the optimal design among designs , the set of all designs with blocks of size and with treatments.

I am not going into the details of the derivation of the optimal design for the above interference model. I just sketch the outline here. First of all we can write down the information matrix for the direct treatment effect , say . Let be the set of all possible block sequences with replacement, which is the design space. Then we try to find the optimal measure among the set to maximize for a given function satisfying the following three conditions:

is concave;

for any permutation matrix ;

is nondecreasing in the scalar .

A measure which achieves the maximum of among for any satisfying the above three conditions is said to be universally optimal. Such measure is optimal under criteria of A, D, E, T, etc. Thus we could imagine that all of the analysis is just linear algebra.

There has been a Machine Learning (ML) reading list of books in hacker news for a while, where Professor Michael I. Jordan recommend some books to start on ML for people who are going to devote many decades of their lives to the field, and who want to get to the research frontier fairly quickly. Recently he articulated the relationship between CS and Stats amazingly well in his recent reddit AMA, in which he also added some books that dig still further into foundational topics. I just list them here for people’s convenience and my own reference.

Kreyszig, E. (1989). “Introductory Functional Analysis with Applications” Wiley.—Functional analysis is essentially linear algebra in infinite dimensions, and it’s necessary for kernel methods, for nonparametric Bayesian methods, and for various other topics.

Remarks from Professor Jordan: “not only do I think that you should eventually read all of these books (or some similar list that reflects your own view of foundations), but I think that you should read all of them three times—the first time you barely understand, the second time you start to get it, and the third time it all seems obvious.”

Last night, I had a discussion about the integrative data analysis (closely related with the discussion of AOAS 2014 paper from Dr Xihong Lin’s group and JASA 2014 paper from Dr. Hongzhe Li’s group) with my friend. If some biologist gave you the genetic variants (e.g. SNP) data and the phenotype (e.g. some trait) data, you were asked to do the association analysis to identify the genetic variants which is significantly associated with the trait. One year later, the biologist got some additional data such as gene expression data which are related with the two data sets given before, and you are now asked to calibrate your analysis to detect the association more efficiently and powerfully by integrating the three data sources. In this data rich age, it’s quite natural to get into this situation in practice. The question is how to come up with a natural and useful statistical framework to deal with such data integration.

For simplicity, we considered the problem that if you are first given two random variables, to study the association between them. Later on you are given another random variable to help to detect the significance association between and . We assume the following true model:

where is independent with . Now the question is what is the characteristic for to be helpful to raise the power for the detection.

What if and are uncorrelated? If they are uncorrelated, then what if and are uncorrelated?

What if and are correlated?

After thinking about these, you will find that for to be useful, it’s ideal that is uncorrelated with and is highly correlated with , i.e. highly correlated with the error term so that it can be used to explain more variation contained in to reduce the noise level.

In order to see why, first notice that the problem exactly depends on how to understand the following multiple linear regression problem:

where (see below for the proof). Thus in order to raise the signal to noise ratio, we hope that , i.e. or , which can keep the signal large. But in order to reduce the noise, we need . In summary, we need to have , which means that and are uncorrelated, and , which means that can be used to explain some variability contained in the noise.

Now please think about the question:

What is the difference between doing univariate regression one by one and doing multiple linear regression all at once?

Here is some hint: first we regress and both onto ,

And then on one hand we find that , and on the other hand we regress the residual onto the residual to get via

This procedure actually is explaining what is the multiple linear regression and what is the meaning for the coefficients (think about the meaning of from the above explanation).

A common theme in mathematics is to replace the study of an object with the study of some category that can be built from that object. For example, we can

replace the study of a group with the study of its category of linear representations,

replace the study of a ring with the study of its category of -modules,

replace the study of a topological space with the study of its category of sheaves,

and so forth. A general question to ask about this setup is whether or to what extent we can recover the original object from the category. For example, if is a finite group, then as a category, the only data that can be recovered from is the number of conjugacy classes of , which is not much information about . We get considerably more data if we also have the monoidal structure on , which gives us the character table of (but contains a little more data than that, e.g. in the associators), but this is still not a complete invariant of . It turns out that to recover we need the symmetric monoidal structure on ; this is a simple form of Tannaka reconstruction.

Python is great and I think will be also great. For pure mathematics, it has lots of symbol calculations, since pure mathematics is abstract and powerful, like differential geometry, commutative algebra, algebraic geometry, and so on. However, science is nothing but experiment and computation. We also need powerful computational software to help us to carry out the result by powerful computation. Sage is your choice ! Since Sage claims that

Gaussian free field and conformal field theory: In these expository lectures, it gives an elementary introduction to conformal field theory in the context of probability theory and complex analysis. It considers statistical fields, and defines Ward functionals in terms of their Lie derivatives. Based on this approach, it explains some equations of conformal field theory and outline their relation to SLE theory.

There will be a talk, “Landscape of Random Functions in Many Dimensions via Random Matrix Theory”, next week given by Antonio Auffinger from the University of Chicago.

Abstract: How many critical values a typical Morse function have on a high dimensional manifold? Could we say anything about the topology of its level sets? In this talk I will survey a joint work with Gerard Ben Arous and Jiri Cerny that addresses these questions in a particular but fundamental example. We investigate the landscape of a general Gaussian random smooth function on the N-dimensional sphere. These corresponds to Hamiltonians of well-known models of statistical physics, i.e spherical spin glasses. Using the classical Kac-Rice formula, this counting boils down to a problem in Random Matrix Theory. This allows us to show an interesting picture for the complexity of these random Hamiltonians, for the bottom of the energy landscape, and in particular a strong correlation between the index and the critical value. We also propose a new invariant for the possible transition between the so-called 1-step replica symmetry breaking and a Full Replica symmetry breaking scheme and show how the complexity function is related to the Parisi functional.

This topic is kind of a combination of my majors, differential geometry, probability and statistics. I am interested in this although I can imagine that it is hard.

Since I missed the whole summer, but during the summer, many interesting things happened, I have to make it up. So this post will be updated during the next few days. I will collect some posts from others here. I hope it would be helpful for you.

Algebraic statistics for binary random variables is concerned with highly structured algebraic varieties in the space of 2x2x…x2-tensors. We demonstrate the advantages of representing such varieties in the coordinate system of binary cumulants. Our primary focus lies on hidden subset models. Parametrizations and implicit equations in cumulants are derived for hyperdeterminants, for secant and tangential varieties of Segre varieties, and for certain context-specific independence models. Extending work of Rota and collaborators, we explore the polynomial inequalities satisfied by cumulants.