smallDataMatrix is an abstract class,
which provides as abstraction to an in memory matrix of data.
It should not be subclassed by concrete subclasses.
Instead,
the subclass anySizeDataMatrix,
can be subclassed with concrete subclasses,
which will provide abstractions to dataMatrices stored in particular file formats,
such as pcl files.

Internally,
all data are read into memory,
and the actual data matrix is stored as a 2 dimensional array.
The indexes of the array which are still valid are stored in internal hashes,
such that subsequent manipulations of the data only consider the data that have not been filtered out.
As rows or columns are filtered out by some of the methods,
the entries for these rows or columns are deleted from the hashes that track valid data.
Thus when data are redumped to a file,
only those data that have not been filtered are printed out.

As smallDataMatrix is an abstract class,
it has no constructor.
However,
the subclass,
anySizeDataMatrix,
once it has determined that a matrix will indeed fit into memory,
MUST call the _init method,
which will result in all data being read into memory.

This protected method will read in all the data for the matrix,
and store it in memory.
It MUST be called during initialization of a subclass object,
before any other methods can be called by the client.

This private method filters out rows that do not have a count for some particular property above or equal to a threshold. It accepts a hash reference, that hashes the row number to a count, and a threshold value. Note that not all rows are necessarily entered into the hash, so this method iterates over all rows, and checks each valid one for its count in the hash, then invalidates those with too low a count.

This private method calculates the standard deviations for each valid column, and returns references to two hashes. Both have the column index as the key, and one has the standard deviation as the values, the other has the column means as the values.

This private setter method receives a pointer to a hash of hashes that stores the percentiles of the data. The first key in the hash is the row from which an element of data came, and the second is the column. The value is the percentile for that piece of data in the column in which it is found.

This private mutator method makes a row invalid. The invalidation is actually done by the the super class, but here, to save memory, we delete the data from the in memory matrix itself. This method is not undoable, because the invalidation also deletes the data for the row.

Note: These methods provide the backend nuts and bolts for a transformation or filtering. They should only be called by the immediate subclass, anySizeDataMatrix, and not directly by the concrete subclasses of anySizeDataMatrix. In addition, note that the companion bigDataMatrix must (and does) provide identical interfaces to these methods (obviously with different underlying implementations), such that anySizeDataMatrix can call the methods without regard to the size of the underlying matrix.

This protected method actually centers the row data, by calculating the average (mean or median, depending on what was requested) for each row, and then subtracting that value from each valid datapoint in the row.

This protected method filters out rows based on their column percentile, when all data are known to be in memory, and optionally allows for the percentiles of each datapoint to be displayed in the output file.