Hi,
I have a set of reference values. After a particular experiment I get a
set of result.
How can I identify a point after which it deviated above a threshold
value?

An example: suppose the reference is 20,19,18,17,16,15,14,13,12,11.
After an experiment the values are changed to :
20,19,18,17,16,11,10,9,8,7.
At the 6th point the value deviated from reference by a value of 5.
After what operations can I get the answer as "6"?

Please help

March 10th 2011, 02:26 AM

Ackbeet

I'm not sure I understand what you're about. At the 6th point, the value has changed by 4, not 5: 15 - 11 = 4.

Quote:

After what operations can I get the answer as "6"?

What are the sorts of "operations" that you're allowed to have? Are these data points results from an experiment? If so, you're not going to change the values except by changing the experiment.

March 10th 2011, 02:46 AM

mailranjithr

Hi Adrian,
Thanks for the reply.
At the 6th point, the value has changed by 4. Not 5. A typo error;

The results are the outcome of an experiment. we can have any sort of operations but finally have to get the point from it deviated. the values may not be the same but the experiment remains the same.

March 10th 2011, 05:06 AM

Ackbeet

I still have no clue what you're trying to do. Let me illustrate by quoting you:

My point is that there has to be a universe of discourse of operations from which you can pick ones that are valid (obviously, some of the ones I listed above are not valid on a list of numbers, while others are). What is your universe of discourse here? That is, please list, concretely, all the possible operations which you are considering using on these lists to manipulate the data.

Quote:

but finally have to get the point from it deviated.

This is not correct English: it doesn't parse. Can you please rephrase using complete sentences? If you can also illustrate with a picture, that'd also help.

March 10th 2011, 07:37 PM

mailranjithr

2 Attachment(s)

Thanks for the reply.
What I meant by "Any sort of operations" is that we can perform mathematical operations(eg. differentiation) to the data to get the intended results.
I am attaching 2 images, one is the reference and other after an expt.
Please go through it..

March 11th 2011, 01:51 AM

Ackbeet

Quote:

Originally Posted by mailranjithr

Thanks for the reply.
What I meant by "Any sort of operations" is that we can perform mathematical operations(eg. differentiation) to the data to get the intended results.

(Drily) Not exactly the scientific method to alter data after obtaining it, except in a well-documented fashion and for a specific purpose. Also, you can't technically ever differentiate measured data, because all measured data is necessarily discrete. You can maybe try some sort of discrete derivative, though, like a difference.

Quote:

I am attaching 2 images, one is the reference and other after an expt.
Please go through it..

I went through both images, and found that my understanding did not increase at all.

Let me ask you this question: what is your goal here? What kind of statement should be the answer to your problem?

March 11th 2011, 02:07 AM

mailranjithr

In that expt.jpg we can see a sharp decline by a value > 4 after the point 32 in X-axis. This is not seen in the ref.jpg. Both ref and expt values are sampled and we see the data in CSV format. So using the data and doing some operations, say in MATLAB, we should get that point as a peak. You can suggest me a way to get that peak...
Thanks

March 11th 2011, 02:19 AM

Ackbeet

Now we're getting somewhere. You want to automatically detect the location of a steep drop in the signal. Is that correct? What assumptions can you make about the signal on either side of this drop? In particular, can you assume that the data is pretty much a straight line on either side of the drop?

March 11th 2011, 02:25 AM

mailranjithr

Yes. I want to automatically detect the location of a steep drop in the signal....
The data on both side of the drop is much like a straight line like in the image expt.jpg.

March 11th 2011, 02:41 AM

Ackbeet

Ok. One option is the following:

1. Take the discrete differential of the signal. That is, from the list of data points where construct the new list

for Notice you will necessarily lose one point in this process, because of the fence-post principle.

2. The drop you're looking for will then appear as a negative peak in the differential signal. If your peak-finding algorithm prefers positive peaks, then simply multiply your differential signal by -1.

And that's it. Your peak-finding algorithm might be as simple as finding the maximum point, if you're sure that there will only be one such large drop.

On the other hand, the operation of taking the differential is highly sensitive to noise. (Integration smooths functions out, and is not sensitive to noise; differentiation makes functions more angular, and is sensitive to noise). So if you tend to get a lot of noise in the signal, you might get a large drop that isn't "real" - it isn't the drop you're looking for. One way to compensate for this is to run your data through some sort of running average filter before taking the differential. Positive: makes your algorithm more robust to noise. Negative: this operation will tend to spread your sudden drop so that it "takes longer". That is, your final result is not likely to be as accurate as it might be.

These are trade-offs you'll have to consider. Hope this helps.

March 11th 2011, 02:46 AM

mailranjithr

Thanks Adrian,
I will try the method

March 11th 2011, 02:47 AM

Ackbeet

You're welcome. Let me know how it goes.

March 11th 2011, 04:39 AM

Ackbeet

Incidentally, I absolutely love problems like this: finding features such as corners, peaks, regions of concavity, etc., in real-world signals. If you have more problems like this, please do pass them on.