How To Work Through a Multi-Class Classification Project in Weka

The Weka machine learning workbench is so easy to use that working through a machine learning project can be a lot of fun.

In this post you will complete your first machine learning project using Weka, end-to-end. This gentle introduction to working through a project will tie together the key steps you need to complete when working through machine learning project in Weka.

After completing this project, you will know:

How to analyze a dataset and hypothesize data preparation and modeling algorithms that could be used.

How to spot check a suite of standard machine learning algorithms on a problem

Need more help with Weka for Machine Learning?

1. Load Dataset

In this tutorial, we will use the Iris Flowers Classification dataset.

Each instance in the iris dataset describes measurements of iris flowers and the task is to predict which species of 3 iris flower the observation belongs. There are 4 numerical input variables with the same units and generally the same scale. You can learn more about the datasets in the UCI Machine Learning Repository. Top results are in the order of 96% accuracy.

2. Analyze the Dataset

Reviewing the distribution of each attribute and the interactions between attributes may shed light on specific data transforms and specific modeling techniques that we could use.

Summary Statistics

Review the details about the dataset in the “Current relation” pane. We can notice a few things:

The dataset is called iris.

There are 150 instances. If we use 10-fold cross validation later to evaluate the algorithms, then each fold will be comprised of 15 instances, which is quite small. We may want to think about using 5-folds of 30 instances instead.

There are 5 attributes, 4 inputs and 1 output variable.

There are a small number of attributes and we could investigate further using feature selection methods.

Click on each attribute in the “Attributes” pane and review the summary statistics in the “Selected attribute” pane.

We can notice a few facts about our data:

There are no missing values for any of the attributes.

All inputs are numeric and have values in the same range between about 0 and about 8.

The last attribute is the output variable called class, it is nominal and has three values.

The classes are balanced, meaning that there is an equal number of instances in each class. If they were not balanced we may want to think about balancing them.

We may see some benefit from either normalizing or standardizing the data.

Attribute Distributions

Click the “Visualize All” button and lets review the graphical distribution of each attribute.

Weka Univariate Attribute Distribution Plots

We can notice a few things about the shape of the data:

We can see overlap but differing distributions for each of the class values on each of the attributes. This is a good sign as we can probably separate the classes.

It looks like sepalwidth has a Gaussian-like distribution. If we had a lot more data, perhaps it would be even more Gaussian.

It looks like the other 3 input attributes have nearly-Gaussian distributions with a skew or a large number of observations at the low end of the distribution. Again, it makes me think that the data may be Gaussian if we had an order of magnitude more examples.

We also get a visual indication that the classes are balanced.

Attribute Interactions

Click the “Visualize” tab and lets review some interactions between the attributes.

Increase the window size so all plots are visible.

Increase the “PointSize” to 3 to make the dots easier to see.

Click the “Update” button to apply the changes.

Weka Attribute Scatterplot Matrix

Looking across the graphs for the input variables, we can see good separation between the classes on the scatter plots. For example, petalwidth versus sepallength and petal width versus sepalwidth are good examples.

This suggest that linear methods and maybe decision trees and instance based methods may do well on this problem. It also suggest that we probably do not need to spend too much time tuning or using advanced modeling techniques and ensembles. It may be a straightforward modeling problem.

3. Evaluate Algorithms

Let’s design a small experiment to evaluate a suite of standard classification algorithms on the problem.

1. Close the Weka Explorer.

2. Click the “Experimenter” button on the Weka GUI Chooser to launch the Weka Experiment Environment.

Weka Experiment Environment

3. Click “New” to start a new experiment.

4. In the “Experiment Type” pane change the “Number of folds” from “10” to “5”.

We can see that all of the models have skill. Each model has a score that is better than ZeroR and the difference is statistically significant.

The results suggest both Logistic Regression and SVM achieved the highest accuracy. If we were to pick between the two, we would choose Logistic Regression if for no other reason that it is a much simpler model. Let’s compare all of the results to the Logistic Regression results as the test base.

12. Click “Select” for the “Test base”, select “functions.Logistic” and click the “Select” button to choose the new test base. Click the “Perform test” button again to perform the new analysis.

We now see a very different story. Although the results for Logistic look better, the analysis suggests that the difference between these results and the results from all of the other algorithms are not statistically significant.

From here we could choose an algorithm based on other criteria, like understandability or complexity. From this perspective Logistic Regression and Naive Bayes are good candidates.

We could also seek to further improve the results of one or more of these algorithms and see if we can achieve a significant improvement. If we change the “Significance” to less constraining values of 0.50, we can see that the tree and KNN algorithms start to drop away. This suggests we could spend more time on the remaining methods. Change “significance” back to “0.05”.

Let’s choose to stick with Logistic Regression. We can collect some numbers we can use to describe the performance of the model on unseen data.

7. Right click on the result item in the “Result list” and select “Save model”. Select a suitable location and type in a suitable name, such as “iris-logistic” for your model.

This model can then be loaded at a later time and used to make predictions on new flower measurements.

We can use the mean and standard deviation of the model accuracy collected in the last section to help quantify the expected variability in the estimated accuracy of the model on unseen data.

For example, we know that 95% of model accuracies will fall within two standard deviations of the mean model accuracy. Or, restated in a way we can explain to other people, we can generally expect that the performance of the model on unseen data will be 96.33% plus or minus 2 * 3.38 or 6.76, or between 87.57% and 100% accurate.

You can learn more about using the mean and standard deviation of a Gaussian distribution in the Wikipedia page titled 68–95–99.7 rule.

Summary

In this post you completed your first machine learning project end-to-end using the Weka machine learning workbench.

Specifically, you learned:

How to analyze your dataset and suggest at specific data transform and modeling techniques that may be useful.

How to spot check a suite of algorithms on the problem and analyze their results.

How to finalize the model for making predictions on new data and presenting the estimated accuracy of the model on unseen data.

Do you have any questions about running a machine learning project in Weka or about this post? Ask your questions in the comments and I will do my best to answer them.

23 Responses to How To Work Through a Multi-Class Classification Project in Weka

Could you please tell me if I can use Weka for Multi class classification of EEG signals? I have three datasets that contain brain-signals of a healthy person, and the rest are of patient’s signals with seizure and without seizure.
I did binary classification for the above mentioned analysis and used classes, ‘normal’ and ‘seizure’. Now I want to take it further by multi class classification. Which of the algorithms should I consider as I’m don’t have nay background for data mining and machine learning. Being a novice I would appreciate any suggestion or help.

Weka does support multi-class classification. I’m not sure about the specific of your problem, you may need to prepare data to make it a suitable supervised learning problem and organize the data into CSV or ARFF files.

thanks for this helpful article. Your Blog is great to get into Machine Learning.
The only thing in this article which i don’t understand is step 3.12 respectively the conclusion you draw from this. Could you explain this a bit more?

“We now see a very different story. Although the results for Logistic look better, the analysis suggests that the difference between these results and the results from all of the other algorithms are not statistically significant.”

How do i see that it is a complete different story? For me it looks very similar. Where do i see the statistical significance?

Sir,
I am new in machine learning techniques. I used Weka 3.9.0 for multi class classification and I got the result.I saved the trained model. But I actually need to use the trained model in my project (using Visual Studio C++). I have some images,each image is divided into different blocks.What I need to do is to select the first block and find out the class it belongs using the trained model, then select the next block and do the same and so on.
Sir, could you plz suggest a solution for this? I m totally stuck in this step. I searched a lot but didn’t get a solution.

Some ideas:
– perhaps use the Java API for Weka and integrate that into your Cpp project.
– perhaps use a cpp machine learning API instead.
– perhaps implement the model you want to use from scratch in your Cpp project.

I’m working on a project that rank resumes.I have used weka for machine learning i.e training of Resumes.
I made a model of 100 resumes initially for a single job type.
My class variable is suitablity that rank Resumes either good average or poor.
Now if I want to make a single model for different job types I will need multi class classification.??
How can I define more than one class label in Weka??

Weldone Dr. Jason. I found you post helpful to me as new to Machine Language. My problem is I tried to save the the result model but I can’t. Always it used to show me command java.io.FileNotFoundException: C:\Program File ………… (Access is denied). Sir, I am in need of your help for what to do.

Hi Jason, well explained article.
I wanted to ask whether it is possible to build a model from a dataset having only 1instance of every class (it is multi class classification problem) in weka?
Secondly, can we have a dataset with an instance having n attributes and another instance having less than or more than n attributes and build model from it?
And can you suggest a site where I can get a dataset mapping a set of symptoms with its associated disease? I’ve checked UCI ML Repository, kaggle but no luck.
Thanks in advance.

hi
i have a multiclass(3 classes) imbalanced data set and i want to use Weka to balance it. how can i do that? is it suitable for me to use one versus one or one versus all algorithm? or should i use smote or random undersampling algorithms. i am beginner in machine learning and weka. i will be so happy if you help me. thanks in advance