What we are really doing in statistical learning is to find a approximator for the data.

Suppose we have a model for some data , which describes the data as

where is some function of data and is the prediction. In principle, it can be any function since data can be very complicated. However, Taylor expansion tells us that

Think about linear regression. It’s just an approximator that neglects the higher orders. In principle we could put in higher orders for the data. These models with higher orders are called polynomial regression.

In general, we could think of other forms, such as fourier series, wavelets, step functions, etc.