The sklearn.datasets package embeds some small toy datasets
as introduced in the Getting Started section.

This package also features helpers to fetch larger datasets commonly
used by the machine learning community to benchmark algorithms on data
that comes from the ‘real world’.

To evaluate the impact of the scale of the dataset (n_samples and
n_features) while controlling the statistical properties of the data
(typically the correlation and informativeness of the features), it is
also possible to generate synthetic data.

There are three main kinds of dataset interfaces that can be used to get
datasets depending on the desired type of dataset.

The dataset loaders. They can be used to load small standard datasets,
described in the Toy datasets section.

The dataset fetchers. They can be used to download and load larger datasets,
described in the Real world datasets section.

Both loaders and fetchers functions return a dictionary-like object holding
at least two items: an array of shape n_samples * n_features with
key data (except for 20newsgroups) and a numpy array of
length n_samples, containing the target values, with key target.

It’s also possible for almost all of these function to constrain the output
to be a tuple containing only the data and the target, by setting the
return_X_y parameter to True.

The datasets also contain a full description in their DESCR attribute and
some contain feature_names and target_names. See the dataset
descriptions below for details.

The dataset generation functions. They can be used to generate controlled
synthetic datasets, described in the Generated datasets section.

These functions return a tuple (X,y) consisting of a n_samples *
n_features numpy array X and an array of length n_samples
containing the targets y.

In addition, there are also miscellanous tools to load datasets of other
formats or from other locations, described in the Loading other datasets
section.

These datasets are useful to quickly illustrate the behavior of the
various algorithms implemented in scikit-learn. They are however often too
small to be representative of real world machine learning tasks.

Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.

The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken
from Fisher’s paper. Note that it’s the same as in R, but not as in the UCI
Machine Learning Repository, which has two wrong data points.

This is perhaps the best known database to be found in the
pattern recognition literature. Fisher’s paper is a classic in the field and
is referenced frequently to this day. (See Duda & Hart, for example.) The
data set contains 3 classes of 50 instances each, where each class refers to a
type of iris plant. One class is linearly separable from the other 2; the
latter are NOT linearly separable from each other.

Ten baseline variables, age, sex, body mass index, average blood
pressure, and six blood serum measurements were obtained for each of n =
442 diabetes patients, as well as the response of interest, a
quantitative measure of disease progression one year after baseline.

Data Set Characteristics:

Number of Instances:

442

Number of Attributes:

First 10 columns are numeric predictive values

Target:

Column 11 is a quantitative measure of disease progression one year after baseline

Attribute Information:

Age

Sex

Body mass index

Average blood pressure

S1

S2

S3

S4

S5

S6

Note: Each of these 10 feature variables have been mean centered and scaled by the standard deviation times n_samples (i.e. the sum of squares of each column totals 1).

The data set contains images of hand-written digits: 10 classes where
each class refers to a digit.

Preprocessing programs made available by NIST were used to extract
normalized bitmaps of handwritten digits from a preprinted form. From a
total of 43 people, 30 contributed to the training set and different 13
to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of
4x4 and the number of on pixels are counted in each block. This generates
an input matrix of 8x8 where each element is an integer in the range
0..16. This reduces dimensionality and gives invariance to small
distortions.

The data is the results of a chemical analysis of wines grown in the same
region in Italy by three different cultivators. There are thirteen different
measurements taken for different constituents found in the three types of
wine.

(1) S. Aeberhard, D. Coomans and O. de Vel,
Comparison of Classifiers in High Dimensional Settings,
Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of
Mathematics and Statistics, James Cook University of North Queensland.
(Also submitted to Technometrics).

The data was used with many others for comparing various
classifiers. The classes are separable, though only RDA
has achieved 100% correct classification.
(RDA : 100%, QDA 99.4%, LDA 98.9%, 1NN 96.1% (z-transformed data))
(All results using the leave-one-out technique)

(2) S. Aeberhard, D. Coomans and O. de Vel,
“THE CLASSIFICATION PERFORMANCE OF RDA”
Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of
Mathematics and Statistics, James Cook University of North Queensland.
(Also submitted to Journal of Chemometrics).

The mean, standard error, and “worst” or largest (mean of the three
largest values) of these features were computed for each image,
resulting in 30 features. For instance, field 3 is Mean Radius, field
13 is Radius SE, field 23 is Worst Radius.

Features are computed from a digitized image of a fine needle
aspirate (FNA) of a breast mass. They describe
characteristics of the cell nuclei present in the image.

Separating plane described above was obtained using
Multisurface Method-Tree (MSM-T) [K. P. Bennett, “Decision Tree
Construction Via Linear Programming.” Proceedings of the 4th
Midwest Artificial Intelligence and Cognitive Science Society,
pp. 97-101, 1992], a classification method which uses linear
programming to construct a decision tree. Relevant features
were selected using an exhaustive search in the space of 1-4
features and 1-3 separating planes.

The actual linear program used to obtain the separating plane
in the 3-dimensional space is that described in:
[K. P. Bennett and O. L. Mangasarian: “Robust Linear
Programming Discrimination of Two Linearly Inseparable Sets”,
Optimization Methods and Software 1, 1992, 23-34].

There are ten different images of each of 40 distinct subjects. For some
subjects, the images were taken at different times, varying the lighting,
facial expressions (open / closed eyes, smiling / not smiling) and facial
details (glasses / no glasses). All the images were taken against a dark
homogeneous background with the subjects in an upright, frontal position
(with tolerance for some side movement).

Data Set Characteristics:

Classes

40

Samples total

400

Dimensionality

4096

Features

real, between 0 and 1

The image is quantized to 256 grey levels and stored as unsigned 8-bit
integers; the loader will convert these to floating point values on the
interval [0, 1], which are easier to work with for many algorithms.

The “target” for this database is an integer from 0 to 39 indicating the
identity of the person pictured; however, with only 10 examples per class, this
relatively small dataset is more interesting from an unsupervised or
semi-supervised perspective.

The original dataset consisted of 92 x 112, while the version available here
consists of 64x64 images.

When using these images, please give credit to AT&T Laboratories Cambridge.

The 20 newsgroups dataset comprises around 18000 newsgroups posts on
20 topics split in two subsets: one for training (or development)
and the other one for testing (or for performance evaluation). The split
between the train and test set is based upon a messages posted before
and after a specific date.

In order to feed predictive or clustering models with the text data,
one first need to turn the text into vectors of numerical values suitable
for statistical analysis. This can be achieved with the utilities of the
sklearn.feature_extraction.text as demonstrated in the following
example that extract TF-IDF vectors of unigram tokens
from a subset of 20news:

It is easy for a classifier to overfit on particular things that appear in the
20 Newsgroups data, such as newsgroup headers. Many classifiers achieve very
high F-scores, but their results would not generalize to other documents that
aren’t from this window of time.

For example, let’s look at the results of a multinomial Naive Bayes classifier,
which is fast to train and achieves a decent F-score:

(The example Classification of text documents using sparse features shuffles
the training and test data, instead of segmenting by time, and in that case
multinomial Naive Bayes gets a much higher F-score of 0.88. Are you suspicious
yet of what’s going on inside this classifier?)

Let’s take a look at what the most informative features are:

>>> importnumpyasnp>>> defshow_top10(classifier,vectorizer,categories):... feature_names=np.asarray(vectorizer.get_feature_names())... fori,categoryinenumerate(categories):... top10=np.argsort(classifier.coef_[i])[-10:]... print("%s: %s"%(category," ".join(feature_names[top10])))...>>> show_top10(clf,vectorizer,newsgroups_train.target_names)alt.atheism: edu it and in you that is of to thecomp.graphics: edu in graphics it is for and of to thesci.space: edu it that is in and space to of thetalk.religion.misc: not it you in is that and to of the

You can now see many things that these features have overfit to:

Almost every group is distinguished by whether headers such as
NNTP-Posting-Host: and Distribution: appear more or less often.

Another significant feature involves whether the sender is affiliated with
a university, as indicated either by their headers or their signature.

The word “article” is a significant feature, based on how often people quote
previous posts like this: “In article [article ID], [name] <[e-mail address]>
wrote:”

Other features match the names and e-mail addresses of particular people who
were posting at the time.

With such an abundance of clues that distinguish newsgroups, the classifiers
barely have to identify topics from text at all, and they all perform at the
same high level.

For this reason, the functions that load 20 Newsgroups data provide a
parameter called remove, telling it what kinds of information to strip out
of each file. remove should be a tuple containing any subset of
('headers','footers','quotes'), telling it to remove headers, signature
blocks, and quotation blocks respectively.

This classifier lost over a lot of its F-score, just because we removed
metadata that has little to do with topic classification.
It loses even more if we also strip this metadata from the training data:

When evaluating text classifiers on the 20 Newsgroups data, you
should strip newsgroup-related metadata. In scikit-learn, you can do this by
setting remove=('headers','footers','quotes'). The F-score will be
lower because it is more realistic.

Each picture is centered on a single face. The typical task is called
Face Verification: given a pair of two pictures, a binary classifier
must predict whether the two images are from the same person.

An alternative task, Face Recognition or Face Identification is:
given the picture of the face of an unknown person, identify the name
of the person by referring to a gallery of previously seen pictures of
identified persons.

Both Face Verification and Face Recognition are tasks that are typically
performed on the output of a model trained to perform Face Detection. The
most popular model for Face Detection is called Viola-Jones and is
implemented in the OpenCV library. The LFW faces were extracted by this
face detector from various online websites.

scikit-learn provides two loaders that will automatically download,
cache, parse the metadata files, decode the jpeg and convert the
interesting slices into memmapped numpy arrays. This dataset size is more
than 200 MB. The first load typically takes more than a couple of minutes
to fully decode the relevant part of the JPEG files into numpy arrays. If
the dataset has been loaded once, the following times the loading times
less than 200ms by using a memmapped version memoized on the disk in the
~/scikit_learn_data/lfw_home/ folder using joblib.

The first loader is used for the Face Identification task: a multi-class
classification task (hence supervised learning):

The sklearn.datasets.fetch_lfw_pairs datasets is subdivided into
3 subsets: the development train set, the development test set and
an evaluation 10_folds set meant to compute performance metrics using a
10-folds cross validation scheme.

The samples in this dataset correspond to 30×30m patches of forest in the US,
collected for the task of predicting each patch’s cover type,
i.e. the dominant species of tree.
There are seven covertypes, making this a multiclass classification problem.
Each sample has 54 features, described on the
dataset’s homepage.
Some of the features are boolean indicators,
while others are discrete or continuous measurements.

Data Set Characteristics:

Classes

7

Samples total

581012

Dimensionality

54

Features

int

sklearn.datasets.fetch_covtype will load the covertype dataset;
it returns a dictionary-like object
with the feature matrix in the data member
and the target values in target.
The dataset will be downloaded from the web if necessary.

data:
The feature matrix is a scipy CSR sparse matrix, with 804414 samples and
47236 features. Non-zero values contains cosine-normalized, log TF-IDF vectors.
A nearly chronological split is proposed in [1]: The first 23149 samples are
the training set. The last 781265 samples are the testing set. This follows
the official LYRL2004 chronological split. The array has 0.16% of non zero
values:

>>> rcv1.data.shape(804414, 47236)

target:
The target values are stored in a scipy CSR sparse matrix, with 804414 samples
and 103 categories. Each sample has a value of 1 in its categories, and 0 in
others. The array has 3.15% of non zero values:

>>> rcv1.target.shape(804414, 103)

sample_id:
Each sample can be identified by its ID, ranging (with gaps) from 2286
to 810596:

>>> rcv1.sample_id[:3]array([2286, 2287, 2288], dtype=uint32)

target_names:
The target values are the topics of each sample. Each sample belongs to at
least one topic, and to up to 17 topics. There are 103 topics, each
represented by a string. Their corpus frequencies span five orders of
magnitude, from 5 occurrences for ‘GMIL’, to 381327 for ‘CCAT’:

>>> rcv1.target_names[:3].tolist()['E11', 'ECAT', 'M11']

The dataset will be downloaded from the rcv1 homepage if necessary.
The compressed size is about 656 MB.

The KDD Cup ‘99 dataset was created by processing the tcpdump portions
of the 1998 DARPA Intrusion Detection System (IDS) Evaluation dataset,
created by MIT Lincoln Lab [1]. The artificial data (described on the dataset’s
homepage) was
generated using a closed network and hand-injected attacks to produce a
large number of different types of attack with normal activity in the
background. As the initial goal was to produce a large training set for
supervised learning algorithms, there is a large proportion (80.1%) of
abnormal data which is unrealistic in real world, and inappropriate for
unsupervised anomaly detection which aims at detecting ‘abnormal’ data, ie

qualitatively different from normal data

in large minority among the observations.

We thus transform the KDD Data set into two different data sets: SA and SF.

-SA is obtained by simply selecting all the normal data, and a small
proportion of abnormal data to gives an anomaly proportion of 1%.

-SF is obtained as in [2]
by simply picking up the data whose attribute logged_in is positive, thus
focusing on the intrusion attack, which gives a proportion of 0.3% of
attack.

-http and smtp are two subsets of SF corresponding with third feature
equal to ‘http’ (resp. to ‘smtp’)

General KDD structure :

Samples total

4898431

Dimensionality

41

Features

discrete (int) or continuous (float)

Targets

str, ‘normal.’ or name of the anomaly type

SA structure :

Samples total

976158

Dimensionality

41

Features

discrete (int) or continuous (float)

Targets

str, ‘normal.’ or name of the anomaly type

SF structure :

Samples total

699691

Dimensionality

4

Features

discrete (int) or continuous (float)

Targets

str, ‘normal.’ or name of the anomaly type

http structure :

Samples total

619052

Dimensionality

3

Features

discrete (int) or continuous (float)

Targets

str, ‘normal.’ or name of the anomaly type

smtp structure :

Samples total

95373

Dimensionality

3

Features

discrete (int) or continuous (float)

Targets

str, ‘normal.’ or name of the anomaly type

sklearn.datasets.fetch_kddcup99 will load the kddcup99 dataset; it
returns a dictionary-like object with the feature matrix in the data member
and the target values in target. The dataset will be downloaded from the
web if necessary.

The target variable is the median house value for California districts.

This dataset was derived from the 1990 U.S. census, using one row per census
block group. A block group is the smallest geographical unit for which the U.S.
Census Bureau publishes sample data (a block group typically has a population
of 600 to 3,000 people).

Both make_blobs and make_classification create multiclass
datasets by allocating each class one or more normally-distributed clusters of
points. make_blobs provides greater control regarding the centers and
standard deviations of each cluster, and is used to demonstrate clustering.
make_classification specialises in introducing noise by way of:
correlated, redundant and uninformative features; multiple Gaussian clusters
per class; and linear transformations of the feature space.

make_multilabel_classification generates random samples with multiple
labels, reflecting a bag of words drawn from a mixture of topics. The number of
topics for each document is drawn from a Poisson distribution, and the topics
themselves are drawn from a fixed random distribution. Similarly, the number of
words is drawn from Poisson, with words drawn from a multinomial, where each
topic defines a probability distribution over words. Simplifications with
respect to true bag-of-words mixtures include:

Per-topic word distributions are independently drawn, where in reality all
would be affected by a sparse base distribution, and would be correlated.

For a document generated from multiple topics, all topics are weighted
equally in generating its bag of words.

Documents without labels words at random, rather than from a base
distribution.

make_regression produces regression targets as an optionally-sparse
random linear combination of random features, with noise. Its informative
features may be uncorrelated, or low rank (few features account for most of the
variance).

Other regression generators generate functions deterministically from
randomized features. make_sparse_uncorrelated produces a target as a
linear combination of four features with fixed coefficients.
Others encode explicitly non-linear relations:
make_friedman1 is related by polynomial and sine transforms;
make_friedman2 includes feature multiplication and reciprocation; and
make_friedman3 is similar with an arctan transformation on the target.

The default coding of images is based on the uint8 dtype to
spare memory. Often machine learning algorithms work best if the
input is converted to a floating point representation first. Also,
if you plan to use matplotlib.pyplpt.imshow don’t forget to scale to the range
0 - 1 as done in the following example.

scikit-learn includes utility functions for loading
datasets in the svmlight / libsvm format. In this format, each line
takes the form <label><feature-id>:<feature-value><feature-id>:<feature-value>.... This format is especially suitable for sparse datasets.
In this module, scipy sparse CSR matrices are used for X and numpy arrays are used for y.

To fully specify a dataset, you need to provide a name and a version, though
the version is optional, see Dataset Versions below.
The dataset contains a total of 1080 examples belonging to 8 different
classes:

The DESCR contains a free-text description of the data, while details
contains a dictionary of meta-data stored by openml, like the dataset id.
For more details, see the OpenML documentation The data_id of the mice protein dataset
is 40966, and you can use this (or the name) to get more information on the
dataset on the openml website:

A dataset is uniquely specified by its data_id, but not necessarily by its
name. Several different “versions” of a dataset with the same name can exist
which can contain entirely different datasets.
If a particular version of a dataset has been found to contain significant
issues, it might be deactivated. Using a name to specify a dataset will yield
the earliest version of a dataset that is still active. That means that
fetch_openml(name="miceprotein") can yield different results at different
times if earlier versions become inactive.
You can see that the dataset with data_id 40966 that we fetched above is
the version 1 of the “miceprotein” dataset:

>>> mice.details['version']'1'

In fact, this dataset only has one version. The iris dataset on the other hand
has multiple versions:

Specifying the dataset by the name “iris” yields the lowest version, version 1,
with the data_id 61. To make sure you always get this exact dataset, it is
safest to specify it by the dataset data_id. The other dataset, with
data_id 969, is version 3 (version 2 has become inactive), and contains a
binarized version of the data:

>>> np.unique(iris_969.target)array(['N', 'P'], dtype=object)

You can also specify both the name and the version, which also uniquely
identifies the dataset:

scikit-learn works on any numeric data stored as numpy arrays or scipy sparse
matrices. Other types that are convertible to numeric arrays such as pandas
DataFrame are also acceptable.

Here are some recommended ways to load standard columnar data into a
format usable by scikit-learn:

pandas.io
provides tools to read data from common formats including CSV, Excel, JSON
and SQL. DataFrames may also be constructed from lists of tuples or dicts.
Pandas handles heterogeneous data smoothly and provides tools for
manipulation and conversion into a numeric array suitable for scikit-learn.

scipy.io
specializes in binary formats often used in scientific computing
context such as .mat and .arff

scikit-learn’s datasets.load_svmlight_file for the svmlight or libSVM
sparse format

scikit-learn’s datasets.load_files for directories of text files where
the name of each directory is the name of each category and each file inside
of each directory corresponds to one sample from that category

For some miscellaneous data such as images, videos, and audio, you may wish to
refer to:

Note: if you manage your own numerical data it is recommended to use an
optimized file format such as HDF5 to reduce data load times. Various libraries
such as H5Py, PyTables and pandas provides a Python interface for reading and
writing data in that format.