Consistent probabilistic outputs for protein function
prediction

Abstract

In predicting hierarchical protein function annotations, such as terms
in the Gene Ontology (GO), the simplest approach makes predictions for
each term independently. However, this approach has the unfortunate
consequence that the predictor may assign to a single protein a set of
terms that are inconsistent with one another; for example, the
predictor may assign a specific GO term to a given protein ('purine
nucleotide binding') but not assign the parent term ('nucleotide
binding'). Such predictions are difficult to interpret. In this work,
we focus on methods for calibrating and combining independent
predictions to obtain a set of probabilistic predictions that are
consistent with the topology of the ontology. We call this procedure
'reconciliation'. We begin with a baseline method for predicting GO
terms from a collection of data types using an ensemble of
discriminative classifiers. We apply the method to a previously
described benchmark data set, and we demonstrate that the resulting
predictions are frequently inconsistent with the topology of the
GO. We then consider 11 distinct reconciliation methods: three
heuristic methods; four variants of a Bayesian network; an extension
of logistic regression to the structured case; and three novel
projection methods - isotonic regression and two variants of a
Kullback-Leibler projection method. We evaluate each method in three
different modes - per term, per protein and joint - corresponding to
three types of prediction tasks. Although the principal goal of
reconciliation is interpretability, it is important to assess whether
interpretability comes at a cost in terms of precision and
recall. Indeed, we find that many apparently reasonable reconciliation
methods yield reconciled probabilities with significantly lower
precision than the original, unreconciled estimates. On the other
hand, we find that isotonic regression usually performs better than
the underlying, unreconciled method, and almost never performs worse;
isotonic regression appears to be able to use the constraints from the
GO network to its advantage. An exception to this rule is the high
precision regime for joint evaluation, where Kullback-Leibler
projection yields the best performance.