Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.

On Jan 27, 8:43 am, "Steve Sachay" <core...@gmail.com> wrote:
> Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.

The human eye is a very accurate peak detector. It automatically
discards the noise and concentrates on the peaks behind the noise.
To emulate that with a computer algorithm is not always easy.
One way is to low pass filter to remove the noise first.
Then derivative-based solutions have a better chance of success.
I use orthogonal wavelet decomposition for low-pass filtering, but
there are other methods like Butterworth.

"Steve Sachay" wrote in message <ihptg8$h6g$1@fred.mathworks.com>...
> Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.

"Paulo Silva" wrote in message <ihq67j$joa$1@fred.mathworks.com>...
> "Steve Sachay" wrote in message <ihptg8$h6g$1@fred.mathworks.com>...
> > Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.
>
> Can you share the data with us or just part of it?

Sure. This is a 4124x2 matrix with seconds (starting at 1177) in column 1 and data in column 2. There are 'rises' in there, the first one is easy, and the next three are a bit trickier. I want to get the points where the 'rise' begins to be at 1257, 2155, 3953 and 4854s respectively.

"Steve Sachay" wrote in message <ihque0$gra$1@fred.mathworks.com>...
> "Paulo Silva" wrote in message <ihq67j$joa$1@fred.mathworks.com>...
> > "Steve Sachay" wrote in message <ihptg8$h6g$1@fred.mathworks.com>...
> > > Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.
> >
> > Can you share the data with us or just part of it?
>
> Sure. This is a 4124x2 matrix with seconds (starting at 1177) in column 1 and data in column 2. There are 'rises' in there, the first one is easy, and the next three are a bit trickier. I want to get the points where the 'rise' begins to be at 1257, 2155, 3953 and 4854s respectively.
>
> http://www.filedropper.com/examplerisefind

Thanks very much for the effort, but this seem to find values closer to the peaks than the beginning of the rise. I'll play with the parameters and see what I can do, but it was very helpful to see your code.

"Steve Sachay" wrote in message <ihptg8$h6g$1@fred.mathworks.com>...
> Of course it doesn't have to be perfect, a good estimation is fine.

let's see...

"Steve Sachay" wrote in message <ihs6o0$arn$1@fred.mathworks.com>...

> Thanks very much for the effort, but this seem to find values closer to the peaks than the beginning of the rise. I'll play with the parameters and see what I can do, but it was very helpful to see your code.

So this simple and free code isn't good enough for your needs, that's great, maybe you can do much better or just wait for another fool like me to give you a miracle solution, with the sample data my code detected the points with a maximum error of 22 points from your "eye" values but still not good enough, I must remember to stop wasting my time trying to help solving other peoples problems.

In article <ihs6o0$arn$1@fred.mathworks.com>,
Steve Sachay <coregis@gmail.com> wrote:
>Thanks very much for the effort, but this seem to find values closer to
>the peaks than the beginning of the rise. I'll play with the parameters
>and see what I can do, but it was very helpful to see your code.

If your example is truly representative of your data, my approach
would be to do the following:

1) find the global data maximum, at time tpeak
2) find the mean value and standard deviation (or range) of the
baseline preceding this peak within a time window tpeak-d1..tpeak-d2
3) find the first point (if any) that occurs above this baseline mean
plus some threshold derived from the amplitude of the baseline noise
(the time of this point is what you are seeking)
4) blank off the data between tpeak-b1 and tpeak+b2
5) repeat steps 1-4 until all the data is blanked and/or no more
'first points' are found

This algorithm/heuristic should be fairly straightforward to implement
in MATLAB code.

> So this simple and free code isn't good enough for your needs, that's great, maybe you can do much better or just wait for another fool like me to give you a miracle solution, with the sample data my code detected the points with a maximum error of 22 points from your "eye" values but still not good enough, I must remember to stop wasting my time trying to help solving other peoples problems.

I am very sorry if you are offended, I meant no ill will. I did not ask anyone to write code for me, just make suggestions of how to proceed beyond my simple initial pass that was not working. My entire rise and fall lasts about 100 points, so within 22 points was not the resolution I am after. I'm sorry if that was unclear.

I have incorporated some of Francis' ideas and now I can obtain the beginning of the rise within 3 points, which works well for my application. Thank you all for your help.

In article <ihsqhn$m68$1@fred.mathworks.com>,
Steve Sachay <coregis@gmail.com> wrote:
>I have incorporated some of Francis' ideas and now I can obtain the
>beginning of the rise within 3 points, which works well for my
>application. Thank you all for your help.

Below is some code I wrote to implement some of the ideas I
outlined in my other post (actually simplified). Both data
and difference data are filtered by running medians of window
size 5 and 3 which doesn't smooth the data in the same way as
a linear filter would but has the useful effect of getting rid
of brief transients. It's a way of reducing noise without
introducing a significant 'lag' into the main transients.

The function takes two threshold arguments which may be tweaked
as necessary:

>> rt=FindPeakRiseTimes(examp, 10, 4)

rt =

2158 4854 3952 1257

which is fairly close to what you required. The times are
listed in order of peak 'energy', but you can of course do a
sort(rt).

Note that I haven't tested this code at all other than to run
it on your data. Also, I am sure that some of it (e.g. the
cummag function) could be expressed much more concisely with
'vector' code.

> Below is some code I wrote to implement some of the ideas I
> outlined in my other post (actually simplified). Both data
> and difference data are filtered by running medians of window
> size 5 and 3 which doesn't smooth the data in the same way as
> a linear filter would but has the useful effect of getting rid
> of brief transients. It's a way of reducing noise without
> introducing a significant 'lag' into the main transients.
>
> The function takes two threshold arguments which may be tweaked
> as necessary:

Thank you again Franicis. The mean filter is a good idea for this, I did find that lowpass filtering was moving the peaks slightly so it was not ideal. I took a similar approach at attacking the derivative but your solution is more elegant (and my code much sloppier :) Thank you for posting this so others may find it in the future.