June 2008

In the 1980's, new learning algorithms for neural networks promised to
solve difficult classification tasks, like speech or object recognition,
by learning many layers of non-linear features. The results were
disappointing for two reasons: There was never enough labeled data to
learn millions of complicated features and the learning was much too slow
in deep neural networks with many layers of features. These problems can
now be overcome by learning one layer of features at a time and by
changing the goal of learning. Instead of trying to predict the labels,
the learning algorithm tries to create a generative model that produces
data which looks just like the unlabeled training data. These new neural
networks outperform other machine learning methods when labeled data is
scarce but unlabeled data is plentiful. An application to very fast
document retrieval will be described.

March 2007

Generative-discriminative Hybrid Technique
We plan to use a technique that combines two kinds of learning algorithms: discriminative and generative. We plan to employ Bayesian networks in the generative phase, and SVM in the discriminative phase.
Algorithms under the generative framework try to find a statistical model that best represents the data. The predictions are then based on the likelihood scores derived from the model. This category includes algorithms such as Hidden Markov Models (HMM) [1], Gaussian Mixture Models (GMM) [2] and more complicated graphical models such as Bayesian networks [3].