Menu

Spell Out Convolution 1D (in CNN’s)

I’m working on a text analysis problem and got slightly better results using a CNN than RNN. The CNN is also (much) faster than a recurrent neural net. I wanted to tune it further but had difficulties understanding the Conv1D on the nuts and bolts level. There are multiple great resources explaining 2D convolutions, see for example CS231n Convolutional Neural Networks for Visual Recognition, but I couldn’t find a really simple 1D example. So here it is.

The input data is four items. The 1D convolution slides a size two window across the data without padding. Thus, the result is an array of three values. In Keras/Tensorflow terminology I believe the input shape is (1, 4, 1) i.e. one sample of four items, each item having one channel (feature). The Convolution1D shape is (2, 1) i.e. one filter of size 2.

The Same 1D Convolution Using Keras

Set up a super simple model with some toy data. The convolution weights are initialized to random values. After fitting, the convolution weights should be the same as above, i.e. [1, 2].

1D Convolution and Channels

Let’s add another dimension: ‘channels’. In the beginning this was confusing me. Why is it 1D conv if input data is 2D? In 2D convolutions (e.g. image classification CNN’s) the channels are often R, G, and B values for each pixel. In 1D text case the channels could be e.g. word embedding dimension: a 300-dimensional word embedding would introduce 300 channels in the data and the input shape for single ten words long sentence would be (1, 10, 300).

Converges slowly, and looks like it found another fitting solution to the problem.

1D Convolution and Multiple Filters

Another dimension to consider is the number of filters that the conv1d layer will use. Each filter will create a separate output. The neural net should learn to use one filter to recognize edges, another filter to recognize curves, etc. Or that’s what they’ll do in the case of images. This excercise resulted from me thinking that it would be nice to figure out what the filters recognize in the 1D text data.

14 Comments

How do you define batch_size in fit if you have a shape e.g. (1,40,1) ? how is this working ? are you just taking batch_size 1 than ? if yes is the model taking for a kernel of 5, 9 times a range of 5 and is putting it through the convolution ?

In the toy example, I have only one sample. Thus the batch size is one, even if it’s not defined explicitly. The batch size for shape [1, 40, 1] would also be one, as there’s only one sample of length 40.

kernel_size=5, stride=1 would move the kernel 1 step at a time across the length 40. The number of times depends on the padding parameter. I think with padding=valid you’d get 9 times like you wrote.

Thank you very much for the clear explanation. Just what I needed to understand how the different parameters work. I started to toy around with Tensorflow JS recently, and reproduced all your examples and results – this was the first time I could represent an expected results and have a clear understanding of why and what is happening.

I have a example, where my dataset have 18 colums and 1000 rows. The output is the last colum, and I convert labels in [0 0 1] or [ 0 1 0] . How I input my values in Convolutuion1D and yours parameters?

Valtet: sounds like your input shape is (1000, 17, 1) and your output shape is (1000, 3). You’d probably want to have a dense layer after the convolution in your neural net architecture in order to optimize your convolution flexibly while keeping the output shape constant. If you only want a convolutional layer, you’d need to adjust the filter size (K), padding (P) and stride (S). The output shape of a convolution layer is O = (W – K + 2P)/S + 1. You can vary the values of K, P, and S so that O becomes 3 when W is 17.

Hi
I am doing a time series analysis using conv1D. I am having 9 columns and 20075 rows. Out of this, I am using 16075 for training the model and remaining for testing the model. I am confused with Kera’s documentation of Conv 1D. What I understood from the Keras doc is
A filter is a number of times we want to extract the features from the input.
A kernel is the width of 1 filter.
Stride is by how many positions we want to move the filter.
If my explanation is right (please correct me if I am wrong) please explain the following:
I am taking the last 5 days of all 9 columns as input and the 6th day of the 9 th column is my output i.e. my input shape is (16070,1,45) and the output shape is 16070,1).
Now I want to take 128 filters, kernel size 10 and stride =5 but it is showing an error.
Please help me what wrong I am doing??

Hello
I am using conv1D. I am having 45 columns and 77292 rows. Out of this, I am using 70% for training the model and remaining for testing the model. I am confused with Kera’s documentation of Conv 1D.
if i take the input shape is (77292 ,1,45) and the output shape is 77292 ,1).
Now I dont know what size i have to put for filter which will be helpful for this amount of data
Please help me regarding this ?
I will be wait for your kind reply Sir.

I searched this kind of explanation for months. I wish I read this blog before.
This is undoubtedly the most clear explanation of conv1d.
I have to investigate channel’s effect on the custom ResNet developed for timeseries,
do you have any advice, like saliency map for timeseries.
good luck in your studies.