How to Prepare Data For Machine Learning

Machine learning algorithms learn from data. It is critical that you feed them the right data for the problem you want to solve. Even if you have good data, you need to make sure that it is in a useful scale, format and even that meaningful features are included.

In this post you will learn how to prepare data for a machine learning algorithm. This is a big topic and you will cover the essentials.

Data Preparation Process

The more disciplined you are in your handling of data, the more consistent and better results you are like likely to achieve. The process for getting data ready for a machine learning algorithm can be summarized in three steps:

Step 1: Select Data

Step 2: Preprocess Data

Step 3: Transform Data

You can follow this process in a linear manner, but it is very likely to be iterative with many loops.

Step 1: Select Data

This step is concerned with selecting the subset of all available data that you will be working with. There is always a strong desire for including all data that is available, that the maxim “more is better” will hold. This may or may not be true.

You need to consider what data you actually need to address the question or problem you are working on. Make some assumptions about the data you require and be careful to record those assumptions so that you can test them later if needed.

Below are some questions to help you think through this process:

What is the extent of the data you have available? For example through time, database tables, connected systems. Ensure you have a clear picture of everything that you can use.

What data is not available that you wish you had available? For example data that is not recorded or cannot be recorded. You may be able to derive or simulate this data.

What data don’t you need to address the problem? Excluding data is almost always easier than including data. Note down which data you excluded and why.

It is only in small problems, like competition or toy datasets where the data has already been selected for you.

Step 2: Preprocess Data

After you have selected the data, you need to consider how you are going to use the data. This preprocessing step is about getting the selected data into a form that you can work.

Three common data preprocessing steps are formatting, cleaning and sampling:

Formatting: The data you have selected may not be in a format that is suitable for you to work with. The data may be in a relational database and you would like it in a flat file, or the data may be in a proprietary file format and you would like it in a relational database or a text file.

Cleaning: Cleaning data is the removal or fixing of missing data. There may be data instances that are incomplete and do not carry the data you believe you need to address the problem. These instances may need to be removed. Additionally, there may be sensitive information in some of the attributes and these attributes may need to be anonymized or removed from the data entirely.

Sampling: There may be far more selected data available than you need to work with. More data can result in much longer running times for algorithms and larger computational and memory requirements. You can take a smaller representative sample of the selected data that may be much faster for exploring and prototyping solutions before considering the whole dataset.

It is very likely that the machine learning tools you use on the data will influence the preprocessing you will be required to perform. You will likely revisit this step.

Step 3: Transform Data

The final step is to transform the process data. The specific algorithm you are working with and the knowledge of the problem domain will influence this step and you will very likely have to revisit different transformations of your preprocessed data as you work on your problem.

Three common data transformations are scaling, attribute decompositions and attribute aggregations. This step is also referred to as feature engineering.

Scaling: The preprocessed data may contain attributes with a mixtures of scales for various quantities such as dollars, kilograms and sales volume. Many machine learning methods like data attributes to have the same scale such as between 0 and 1 for the smallest and largest value for a given feature. Consider any feature scaling you may need to perform.

Decomposition: There may be features that represent a complex concept that may be more useful to a machine learning method when split into the constituent parts. An example is a date that may have day and time components that in turn could be split out further. Perhaps only the hour of day is relevant to the problem being solved. consider what feature decompositions you can perform.

Aggregation: There may be features that can be aggregated into a single feature that would be more meaningful to the problem you are trying to solve. For example, there may be a data instances for each time a customer logged into a system that could be aggregated into a count for the number of logins allowing the additional instances to be discarded. Consider what type of feature aggregations could perform.

You can spend a lot of time engineering features from your data and it can be very beneficial to the performance of an algorithm. Start small and build on the skills you learn.

Summary

In this post you learned the essence of data preparation for machine learning. You discovered a three step framework for data preparation and tactics in each step:

Step 1: Data Selection Consider what data is available, what data is missing and what data can be removed.

Data preparation is a large subject that can involve a lot of iterations, exploration and analysis. Getting good at data preparation will make you a master at machine learning. For now, just consider the questions raised in this post when preparing data and always be looking for clearer ways of representing the problem you are trying to solve.

Resources

If you are looking to dive deeper into this subject, you can learn more in the resources below.

Hi Fraser, good question.
Indeed, it can difficult to know if data is bad and you may not always have a domain expert at hand to comment. Sometimes it is obvious though, like 0 values that are impossible in the domain like a blood pressure. I’ve also seen -999 used to signal “not provided”. In these cases we can mark attributes as missing and think about possible rules for imputing if we so desire.
Where do you draw the line though? Should severe outliers be marked as missing? Sometimes. I like to try a lot of stuff, for example, I would try removing instances with large outliers in one dimension and see what that did to my models, I’d also try removing instances with missing values and try models on variations of the data with imputed value. Almost always, modeling ground truth is not the goal, there are performance metrics like classification accuracy or AUC that we are being optimized.
You’re right though, sometimes the broken data can represent something very interesting – anomalies that signal something useful in and of themselves in the domain.

Yes, indeed. Is it an outlier, or a poorly encoded result, or a result with atypical calibration, or does it represent a distinct and real combination of natural conditions …

I work a lot with chemical concentration data in water and sediment and I run into censored data routinely. Mostly from the 1000 mg/L. Censored data of this particular type is handled differently by different people and as you suggest values need to be imputed (with an appropriate sampling distribution) if the rest of a multi-parameter time-sample result is to remain in the analysis.

For me this is what makes data analysis fun.

I just arrived at your site, and I see so many articles of interest. Thank you for making this available.

One issue that I run into is that the data sometimes lacks semantic integrity. This is not an issue of missing values, but just having improper values. When values are of different data types within a column, it is easy to detect and fix.

However, when the data type is the same but the meaning changes, then it’s much more difficult. For example, I’ve seen sales data where a column named ‘marketing plan code’ would have string data type denoting marketing plan codes, except in a few cases where the users put in vendor codes because they didn’t have any other field to record that information.

Jason, does it affect an algoritm if, during the preparation process I transform the list of rows (like tables, where the key columns repeats) to pivot tablee, where the key colums shows once and a lot of columns (say hundreds) have parcial sums or counts for the different conditions (let’s say sales of january in one column, sales of february in a second column and so on).
Does it would make muiltcorrelation as some columns could be aggregated to one?

I am currently working on a project on a government data set to find if an entity(person or an individual) were involved in a a positive or a negative way. I took a flat file containing some test data and prepared the code to perform sentiment analysis using Naive Bayes algorithm using NLTK python modules.

– In most cases we have a defined trained data set tagged as ‘positive’ or ‘negative’ (e.g movie reviews, twitter data set). In my case there is no existing trained government data set.
– The training data is available but I need to categorize the training data set as ‘positive’ or ‘negative’.
– My question here is, how do we go about classifying my government data as ‘positive’ or ‘negative’.

I’m looking forward on your advice on how to categorize my government training data as positive or negative. This is very important for me to get my sentiment analysis with best possible accuracy.

My current and first ML project has natural language as it’s input and I spent a huge chunk of time on preparing it.

I stopped once the data reached a “reasonable” level so that I could continue with the project, i.e. I’m dropping the hard to parse cases and might return to them later once the whole pipeline is ready for testing.

Hi Jason, thank you for the great effort and knowledge put into all these posts!
My question will probably be silly, but since I’m a complete n00b I’ll do it just the same.
Data prep, feature analysis and engineering will get you a set of data in a format completely different from original data. These data transformation steps may be very hard to do automatically. My problem is related to classification, I am using NN which may not be the best choice, but hey, humor me 😉
So, cutting short. Originally, I get raw data, I prep and transform it. The transformed data will train and test “my” NN. Now, the “real world” will challenge my model with raw data, presumably with the same format as my original training set, minus the classification ( of course…). Now, I suppose I’ll have to go through the same data transformation of the data before the trained model can be fed with it. Right? Doesn’t this mean extra care must be taken to make the data transformation process (at least ideally) automatic itself?
Sorry for the long question, hope to hear your thoughts on these points. And thank you once again!

Yes. Any data transformation performed on data used to fit your model must be performed on data when making predictions.

This means we need a very clear recipe for this transform, ideally automatic and also in the case of regression problems it must be reversible so that we can convert predictions back into their predictions scale for use.

I would like to offer that within your topic of “Select Data” you offer a bit more explicit guidance on the topic of assessing and characterizing data quality. It’s cliché, but garbage-in-garbage-out is a fundamental concept. I so often come across advanced analytic initiatives that have started out with Assumptions for quality of “selected” data and moved on – only to find out months later that everything has to reset to basic principles of data acquisition and management.

What transforms have been applies to source data by systems that precede the database you are selecting from?

If sensor data is involved, what formatting, precision, transformations, signal processing, etc. have been applied?

If data is being acquired from multiple, disparate systems what formatting, scale, and precision differences are being masked by the database system you are selecting from?

when I go through UCI Machine Learning Repository following doubts have occured:

1. in bike sharing dataset, I saw two .csv files(one is day.csv and another is hour.csv). So,i can’t understand how to make this dataset suitable for me to apply machine learning algorithm on it to make predictive model by splitting the whole dataset into train and test sets?

2. in this repository, I saw dataset characteristics as multivariate and univariate, what does this mean?

3. in this repository, whenever I explore any of the dataset, there is no statement present there to mention which is the feature to be predict by applying machine learning algorithms?

4. what if both numeric types(float as well as integer) values in any of the feature exist in a dataset? Should we scale the feature values(integer) to float in order to get good predictive model?

When I saw bike sharing dataset in UCI Machine Learning Dataset, UCI mention it’s dataset characteristics as univariate despite having total of 16 features(columns). Why is this so? Shouldn’t it be multivariate, instead.

Secondly, as you have recommended to join two .csv files into one, but when I use this dataset, I noticed that both of the files have same features(except hr(hour) available only in hour.csv file not in day.csv file) with different values in each of the same features available in both the files. In this kind of situation, if I join both the files, values get redundant and even features as well. So, what do you recommend, how to prepare my dataset in this type of situation?
Thanks for your quick response to previous question….

What happens if I use a data that does not have a normal distribution?
Are some ML algorithms only suitable for data that are assumed to be normal?
How can I identify whether an algorithm works with normal/non-normal or just normal data?

Thanks for your help. Can you please suggest me what is the best way to deal with a dataset that contains a lot of text columns. Also the values of these columns too have a huge set of different values.

Thank you for your work, I really appreciate your efforts in helping us.
I am a BIG fan.

First off all, I’m planning to use a LSMT-RNN in multivariate time series problem.

I’m beginning my studies in machine learning and probably my question is very silly, but to me is a big issue.

I have a time series database with 221 features not supervisioned yet, wich I would to transform to a input with to 6 up to 10 features. After this, I would like to supervise the output up to 10 time-steps with 1 feature.

Now I think I can choose my input data, but how? Should I pick from the same cluster that my output have affinity with output, or should I pick from other clusters that don’t have affinty with my output?

Another data processing technique that is commonly used today, particularly in computer vision, is data augmentation where basically we introduce small changes such as rotations, coloring, and translations to images in order to emulate different conditions.

hello,
Actually, I am new toML, I want to know that when we apply data preprocessing on a dataset, whether we have to change the existing dataset or we have to create a new dataset for the modified data? Means after preprocessing is done will we be having two datasets, one the actual dataset and the other preprocessed dataset or there will be only one dataset with preprocessed data?

Hi ,
I am vikash I want to know about the assumptions means about the pre-validation and post validation of data.for example for linear regression we have pre-validation or diagnosis like
1.normal distribution of data
2.No multicollinearity
3.Linear relationship
and
4.Missing values
for Post validation or diagnosis after creating the linear regression mode there are
1.Normality of errors
2.Homoschedasticity
3ouliers and levrges
5auto corelation.
these are the assumptions for Linear Regression .What about the rest of algorithms assumptions ? can you guide me the assumptions for other algorithms .

Hello,
I’m new in Machine Learning so I have a question. Input data have to be the same size?
I mean, I have 10 matrixs with data, but matrixs have size for example [60, 120], [60, 460], [60, 340] and so on. I want to use Tensorflow Engine.
I would be grateful if you could answer my question.
Regards!

i really like the site and there is a lot of really useful things here, i’m presented at the moment with a problem.

I’m attempting to classify a number of scanned PDFs based on the machine read text within them, i’ve got to the point where i have a relatively large test set.

The documents themselves have extremely predictable sentences which tie in very closely with the classification however all i’ve managed to really find on this is using the BoW model.

Would using a neural net to achieve this be a viable option? Also i’m having some problems with the pre-processing of the data. I’m not 100% on the best way to remove ‘\n’ characters and other punctuation from the large text strings.

hi Jason , i like a lot your way to explain machine learning.
i am working on combining machine learning techniques , and my question : there is ML problems where there are enough datasets to validate my work.

I have a question, I am writing a neural network from scratch (back propagation algorithm) using sigmoid function so I have scaled my data in a range between -1 and 1 ]-1,1[ but sigmoid function give results between 0 and 1. So I would like to know if I must scale my data in a range between 0 and 1 [0,1] for sigmoid function?. Or would DR Jason please make me clear if there is a recommended scale of data when using a sigmoid function? or what is the recommended scale for sigmoid function?

I’m trying to create a classification LSTM model. I have three categorical variables apart from my predictor variable. I have label encoded all the three variables. Do i need to scale the variables or I could use them as is .

I am working on Sentiment Analysis application for my MSc and I am pretty beginner in this feild
I have collected the data from twitter but I want to know shall I clean the data before or after Annotation ? will the order make a difference ?

Thanks again for a good read. In cases when we don’t have an inherent category/class backed up by literature, do you think its okay to use the mean value as a cut-off for classes?. For example, say we’re trying to separate between high performers and low performers in a workplace based on a survey outcome. Now that survey doesn’t have an exact cut-off saying anybody who gets above 10 out of 20 is high performer and below is low-performer. One thing that I guess we could do is just use clustering first to divide the dataset into two clusters and use that as classes. Other than that, would it be okay to calculate the mean score among all the participants and then use that as a cut-off to divide the sample to high and low class. And then use that for train/test? Does it make sense, you think? This is assuming that the data is normally distributed. If not, a percentile based approach might be good. Anyway, do you think it is okay to create classes based on the average score? If not, what might be some other ways to divide the classes based on a numerical value if there’s no inherent category? The reason this comes up is because I’m trying to convert a regression problem to a classification problem but I am not sure if classifying based on mean is a good idea.

Hi Jason,
How do you deal with missing data in your data set, do you just make them NA? I am using movie data and the variable that has missing data is the actors name. I put NA in this variable and it made 2829 NA out of 14800 records, I believe this could be a problem but wasn’t sure how to address it.