Conditional random fields (CRFs) are a class of statistical modeling method often applied
in pattern recognition and machine learning, where they are used for structured prediction.
Whereas an ordinary classifier predicts a label for a single sample without regard to "neighboring"
samples, a CRF can take context into account; e.g., the linear chain CRF popular in natural
language processing predicts sequences of labels for sequences of input samples.

While Conditional Random Fields can be seen as a generalization of Markov models, Hidden
Conditional Random Fields can be seen as a generalization of Hidden Markov Model Classifiers.
The (linear-chain) Conditional Random Field is the discriminative counterpart of the Markov model.
An observable Markov Model assumes the sequences of states y to be visible, rather than hidden.
Thus they can be used in a different set of problems than the hidden Markov models. Those models
are often used for sequence component labeling, also known as part-of-sequence tagging. After a model
has been trained, they are mostly used to tag parts of a sequence using the Viterbi algorithm.
This is very handy to perform, for example, classification of parts of a speech utterance, such as
classifying phonemes inside an audio signal.

// Now, we can use the Markov classifier to initialize a HCRFvar baseline = HiddenConditionalRandomField.FromHiddenMarkov(hmm);
// We can check that both are equivalent, although they have// formulations that can be learned with different methods:int[] predictedLabels = baseline.Decide(words);

// Now we can learn the HCRF using one of the best learning// algorithms available, Resilient Backpropagation learning:// Create the Resilient Backpropagation learning algorithmvar rprop = new HiddenResilientGradientLearning<double[]>()
{
Function = baseline.Function, // use the same HMM function
Iterations = 50,
Tolerance = 1e-5
};
// Run the algorithm and learn the modelsvar hcrf = rprop.Learn(words, labels);
// At this point, the HCRF should be successfully // able to distinguish between our three word classes:// int hc1 = hcrf.Decide(hello); // should be 0int hc2 = hcrf.Decide(car); // should be 1int hc3 = hcrf.Decide(wardrobe); // should be 2

The next example shows how to use the learning algorithms in a real-world dataset,
including training and testing in separate sets and evaluating its performance:

// Ensure we get reproducible results
Accord.Math.Random.Generator.Seed = 0;
// Download the PENDIGITS dataset from UCI ML repositoryvar pendigits = new Pendigits(path: localDownloadPath);
// Get and pre-process the training setdouble[][][] trainInputs = pendigits.Training.Item1;
int[] trainOutputs = pendigits.Training.Item2;
// Pre-process the digits so each of them is centered and scaled
trainInputs = trainInputs.Apply(Accord.Statistics.Tools.ZScores);
trainInputs = trainInputs.Apply((x) => x.Subtract(x.Min())); // make them positive// Create some prior distributions to help initialize our parametersvar priorC = new WishartDistribution(dimension: 2, degreesOfFreedom: 5);
var priorM = new MultivariateNormalDistribution(dimension: 2);
// Create a new learning algorithm for creating continuous hidden Markov model classifiersvar teacher1 = new HiddenMarkovClassifierLearning<MultivariateNormalDistribution, double[]>()
{
// This tells the generative algorithm how to train each of the component models. Note: The learning// algorithm is more efficient if all generic parameters are specified, including the fitting options
Learner = (i) => new BaumWelchLearning<MultivariateNormalDistribution, double[], NormalOptions>()
{
Topology = new Forward(5), // Each model will have a forward topology with 5 states// Their emissions will be multivariate Normal distributions initialized using the prior distributions
Emissions = (j) => new MultivariateNormalDistribution(mean: priorM.Generate(), covariance: priorC.Generate()),
// We will train until the relative change in the average log-likelihood is less than 1e-6 between iterations
Tolerance = 1e-6,
MaxIterations = 1000, // or until we perform 1000 iterations (which is unlikely for this dataset)// We will prevent our covariance matrices from becoming degenerate by adding a small // regularization value to their diagonal until they become positive-definite again:
FittingOptions = new NormalOptions()
{
Regularization = 1e-6
}
}
};
//// The following line is only needed to ensure reproducible results. Please remove it to enable full parallelization//teacher1.ParallelOptions.MaxDegreeOfParallelism = 1; // (Remove, comment, or change this line to enable full parallelism)// Use the learning algorithm to create a classifiervar hmmc = teacher1.Learn(trainInputs, trainOutputs);
// Create a new learning algorithm for creating HCRFsvar teacher2 = new HiddenResilientGradientLearning<double[]>()
{
Function = new MarkovMultivariateFunction(hmmc),
MaxIterations = 10
};
//// The following line is only needed to ensure reproducible results. Please remove it to enable full parallelization//teacher2.ParallelOptions.MaxDegreeOfParallelism = 1; // (Remove, comment, or change this line to enable full parallelism)// Use the learning algorithm to create a classifiervar hcrf = teacher2.Learn(trainInputs, trainOutputs);
// Compute predictions for the training setint[] trainPredicted = hcrf.Decide(trainInputs);
// Check the performance of the classifier by comparing with the ground-truth:var m1 = new GeneralConfusionMatrix(predicted: trainPredicted, expected: trainOutputs);
double trainAcc = m1.Accuracy; // should be 0.81532304173813608// Prepare the testing setdouble[][][] testInputs = pendigits.Testing.Item1;
int[] testOutputs = pendigits.Testing.Item2;
// Apply the same normalizations
testInputs = testInputs.Apply(Accord.Statistics.Tools.ZScores);
testInputs = testInputs.Apply((x) => x.Subtract(x.Min())); // make them positive// Compute predictions for the test setint[] testPredicted = hcrf.Decide(testInputs);
// Check the performance of the classifier by comparing with the ground-truth:var m2 = new GeneralConfusionMatrix(predicted: testPredicted, expected: testOutputs);
double testAcc = m2.Accuracy; // should be 0.77061649319455561