How important is detection time?

I received an email recently (an anonymous email!) which questioned the effectiveness of those FOD detection systems which have a detection time greater than the mean time between aircraft movements. The question in the email was this, “Did you neglect the timing requirement [detection time] when you did your system design???[sic]“ The suggestion was that detection time should be the fundamental design consideration when building a FOD detection system. Personally I believe that reducing the risk from FOD should be the key design consideration. The idea that detection time is the key is based on the following argument:

If a system takes 6 minutes to detect an item of FOD, and the next aircraft is due in 4 minutes, then the FOD detection system is completely ineffective at reducing risk.

It’s the sort of argument that people in marketing dream of, not only does it appear to make a lot of sense, but FOD detection time is easy to quantify, and therefore it’s easy to compare across the various systems. Unfortunately, it does not stand up to any form of rigorous analysis. So, let’s take at look at this in more detail, the first thing we need to do is to define the risk from FOD.

What is risk?

It’s often very difficult to quantify risk in any meaningful way, but with FOD it’s actually quite easy. The risk posed by FOD is proportional to the time the FOD object spends on the runway surface (or other airfield surfaces). Of course, the risk from an item of FOD will also depend on exactly where the FOD is found, and what the object is, but, with a large enough data set these two factors should become less important, and then you’re left with the risk from FOD simply being proportional to the time the object spends on the airfield surface.

For example, a single item of FOD on the runway for 1 hour represents the same risk as two items of FOD on the runway for 30 minutes each. It’s even possible to give units, and I’m going to call the units of FOD risk “FOD minutes” (as it’s simply the number of FOD items multiplied by the time on the surface)

An example

Let’s consider a very simple example of a single item of FOD that is on the runway for 3 hrs, and there is no FOD detection system installed. (I’ve assumed the runway is manually checked every 6 hrs, so the mean time FOD will spend on the runway is 3hrs)

The risk posed by this one item is 1 x 180 mins = 180 FOD minutes

Now let’s add a FOD detection system and see what happens, let’s assume that the FOD detection system has a detection time of 2 minutes.

The risk posed by this one item is 1 x 2 mins = 2 FOD minutes

But, it gets even better, if there is an aircraft movement every 4 minutes then for any detection time less than 4 min the FOD risk will actually be zero! Unfortunately this hypothetical case is so simple it’s essentially meaningless, this examples makes 3 assumptions:

All FOD items originate from the last aircraft that used the runway

The risk from FOD is proportional to the detection time

The FOD system detects all FOD

Lets deal with each one of these in turn:

Assuming that all FOD originates from the previous aircraft is unrealistic, it assumes that there are no other sources of FOD, such as items from ground vehicles, wildlife, litter, broken concrete or tarmac, stones, tools, luggage etc.

The risk from FOD is proportional to the time the FOD spends on the surface, this is the detection time plus the retrieval time. The FOD does not magically disappear after it has been detected.

No FOD system is 100% effective. Each will have a probability of detection. So there will still be some FOD on the runway which will be found during manual checks.

A better example

Let’s create a more realistic example that includes the probability of detection, retrieval time, and assumes that there are sources of FOD other than the previous aircraft. First we have to generate a hypothetical airport, this airport has the following parameters:

300 Items of FOD are found at the airport each year.

It takes, on average, 5 mins to retrieve an item of FOD.

There are 6 hrs between manual FOD checks.

The average time between aircraft movements is 4 mins.

50% of all FOD found came from the previous aircraft.

Now let us also introduce two FOD detection systems that differ only in their probability of detection and their detection time.

System A: Detection time = 1 min. Probability of detection = 90%

System B: Detection time = 7 mins. Probability of detection = 95%

So which system is most effective at reducing the risk due to FOD? Well, the probability of detection differs by just 5%, but the detection time for system B is 600% greater than for system A. Not only is it 600% greater, but it’s also greater than the the mean time between aircraft movements.

So, the answer is obvious, isn’t it? Actually, each system reduces the risk from FOD by almost exactly the same amount, down to around 10% of the original risk when no FOD detection system was installed (System A reduces the risk to 13%, and system B to 11.3%). More information is included in the image below.

Click to Zoom

But what if both systems had the same probability of detection and only differed in their detection time? In this case, where both systems have a 95% probability of detection, system A has now reduced the risk from FOD to just 8.2%, and system B remains unchanged at 11.3%. So yes, detection time is a factor. But, a massive reduction in detection time results in a very small reduction in risk.

So, why don’t all the FOD detection systems just reduce their detection time to less than the mean time between aircraft movements?

The fact is that you don’t get something for nothing, and this is true for detection systems. And this is where it starts to get interesting. Each of the systems can probably reduce their detection times, if they wished, but, it would come at a cost, and that cost could be a reduction in probability of detection . And we’ve just seen an example where a system that had a smaller detection time (by 600%), had that advantage wiped out by a system that had a better probability of detection, by just 5%!

This is best explained with an example. Consider a detection system (FOD or not, it doesn’t matter) based on a visible camera system. The system makes detections based on information it receives, and that information is in the form of light entering the camera. Less light, less information, reduced probability of detection. So, what happens when it gets dark? One of two things can happen:

The camera collects less light, and the probability of detection reduces

The camera stares for longer, i.e. it collects light for longer, and the probability of detection remains constant.

There is a FOD detection system on the market that is based on a visible camera system, and the detection time does indeed increase as it gets dark. Why did they choose to follow option 2 above? why not just follow option 1 and allow the probability of detection to decrease. It’s simple, they realise that probability of detection is far more important that detection time.

So, does this mean that systems with a longer detection time are better?

No, I wish it were that simple, but it’s impossible to compare the trade-off between probability of detection and detection time across systems that are based on different technologies. It’s quite possible that a system based on one form of technology will have a better probability of detection, and a shorter detection time.

Does a reduction in detection time always result in a decrease in probability of detection?

No, it’s possible to reduce detection time while keeping probability of detection constant. When you reduce detection time you’re essentially degrading the quality of the data used to determine the detection, you can still maintain the probability of detection by increasing the system’s sensitivity, but again, you don’t get something for nothing. Increasing the sensitivity will lead to an increase in false alarms, and we’re not talking about a few percent. An increase in the false alarm rate is in itself not a problem, if it means maintaining the probability of detection then you’re still reducing the risk on the runway, which of course is the goal. But, if the false alarm rate is so high that the operator starts to ignore the alarms, that’s when the risk on the runway starts to increase again.

So which system suffers from these trade offs between detection time, probability of detection and false alarm rate?

It doesn’t matter if the FOD detection system uses cameras, radar or lidar, the trade-offs still exist. Actually it doesn’t matter if it’s a FOD detection system or not, it’s fundamental to all detection systems. Radar, cameras (visible, infrared), lidar, human beings, sniffer dogs, or squirrels looking for nuts on the forest floor, they are all detection systems and they all suffer these issues. Lets consider the current FOD detection system used in every airport today, the human being, he drives down the runway at a particular speed trying to detect FOD. Let’s reduce his detection time, it’s easy, ask him to drive twice as fast, unfortunately he now has 1/2 as long to look at each object. So what’s the trade-off in this situation? Typically he will find less FOD (the probability of detection will fall), but what if he has been given strict instructions to find the same amount of FOD (i.e. maintain the probability of detection), well, he can do that, but the amount of false alarms will rise. It’s a silly example, but it proves the point.

So why do some vendors still insist that their system is better just because they have a shorter detection time?

Unfortunately, it’s our fault. We like comparing numbers, it’s easy, system A has a shorter detection time than system B, therefore I’m going to buy system A. The best analogy I can make is with digital cameras. The number of megapixels a camera has is still used by many people as the factor that most influences which camera they buy. It’s easy to compare this parameter across various cameras. But, much like detection systems, there is a trade off, if the digital camera sensor size remains constant, then as the number of pixels is increased, each pixel has to be made smaller, and unfortunately this tends to increase the noise in the system, and this can result in reduced image quality. And comparing the image quality from different cameras is much harder than comparing the number of megapixels they have. It’s very similar to FOD detection systems, comparing detection time is easy, but comparing the probability of detection, or false alarm rate is not.

To sum up, I’m not saying that detection time is not important, I’m just saying that it’s not as important as it would first appear (and there are far more important parameters to consider). Always remember that the goal of a FOD detection system is to reduce the risk from FOD, it’s not to have the shortest detection time, or the lowest number of false alarms. And to the person (who didn’t leave their name, position, company, or a usable email address) who asked “Did you neglect the timing requirement[detection time] when you did your system design???[sic]”, I have one question for you, “did you only consider the detection time when you designed/bought your system?” if the answer is Yes, then all I can say to you is “Oops”

I disagree with you, how do I comment? Is an anonymous email the best method?

Ummm….let me think about this….no, an anonymous email is not the preferred method. I’m aware that not everyone is familiar with a blog style website, so here are some basic instructions for anyone who wishes to comment on this article.

At the bottom of this article (and every article on this website) is a section entitled Leave a Reply, this is where comments should be left. It’s public, it allows others to see the comments, which I believe is useful, especially if they have similar comments. If someone has an issue with a particular assumption I’ve made, such as “50% of FOD originates from the previous aircraft”, please remember that this is not supposed to represent an average value, or a value for a particular airfield. If you have some real data for the airport values used above then please use the spreadsheet (link at the end of this article), do the calculation for your particular airport, include your system’s detection time and probability of detection and include your results in the Leave a Reply section.

If you discover an error in the calculation itself then let me know in the Leave a Reply section, I’m human, mistakes happen. The spreadsheet is available for anyone to download, it’s completely open, there are no hidden or protected fields. Please feel free to check the calculations yourself (build on the original calculation if you wish, add other airfield surfaces, change the risk for each surface, add a probability of detection for the manual checks etc). I have nothing to hide, and I’m a fan of transparency. If you are really uncomfortable with your comments being public then feel free to use the Contact Form, but my personal preference is that we keep any discussions public.

The Spreadsheet

Since starting to write this article the spreadsheet has grown and become more involved than was originally planned. It now includes the ability to input data for 3 FOD detection systems, and also allows the user to select which action is taken when FOD is detected, i.e. to close the runway and retrieve, or to continue operations and retrieve. The spreadsheet is provided, as is.

7 Responses to “How important is detection time?”

I appreciate your insightful articles on FOD detection systems. While FOD detection systems are no substitute to a physical inspection, they can provide an additional layer of safety in the effort to manage potential damaging FOD and reduce the risk of a damaging event. However, one aspect of your article that I found absent was the importance of managing FOD data and the benefits of acting on the FOD data’s analysis.

Your article argues the facts that FOD risk reductions are based on the accuracy and frequency of a system’s “sweep time and sensitivity to detecting FOD”. However, it doesn’t discuss the critical nature of a systems ability to collect data and to analyze and manage the data it collects. Any FOD detection process, whether human or automated, must not only reliably detect potential damaging FOD and cause the removal of same but should integrate and record FOD data and resolution into its system.

Safety Risk Management will become a key fundamental of any airports Safety Management System (SMS), soon to become regulatory for certificated airports and air carriers. SMS, like an airports Pavement Management Systems (another FAA requirement for airports), depends upon the ability to not only collect data but to provide meaningful analysis of that data that gives airport decision makers the means to justify spending more (or less) on FOD management and other linked safety efforts. In the long run without knowing what you’re finding, where it was located, where it came from and what you did with it (simple key performance indicators) you simply have a very expensive radar system that “sees FOD” but does nothing about it. And in that case you’re no smarter for the money spent or time taken to use the technology.

I couldn’t agree with you more. The reason I don’t discuss the risk reduction that can be achieved through analysis of FOD data is only because the sole purpose of the article was to look at a system’s detection time (it was designed to bust a particular myth that seems to exist currently).

The Tarsier Toolbox (a product I developed while at QinetiQ) is designed to do exactly what you suggest. It allows analysis of the FOD data, both spatially and temporally, for example, it allows the generation of heat maps that can be overlaid onto a map of the runway, showing problem areas. It contains details not only of the location and time of a FOD find, but also an image of the FOD item and the weather conditions when the FOD was detected. The sole purpose of the Tarsier Toolbox is to enable better (more focused) preventative measures to be taken. But, the ability to catalog and analyze FOD data has always existed, FOD detection is new. I don’t think that any “new” preventative measures, however well informed, can reduce the risk from FOD to around 10% (which is what a FOD detection system can do). The current FOD detection systems on the market (all of them) represent a step-change in the ability to detect FOD and therefore reduce risk. The systems do not just “see FOD”, they enable that FOD to be removed up to 6 hours before it would have been removed without the detection system.

A FOD detection system will also result in a larger, and better populated database of FOD items. Larger, as it will find more FOD than is found via manual inspections alone, and better, as it will find FOD that won’t be found (or cataloged) by manual inspections. For example, animals that are on the runway at night (either to eat, or to keep warm on the surface) won’t be found via a manually inspection, but will be detected by a FOD detection system.

In summary, I completely agree with you i.e. that any FOD detection system has to have a database with the parameters that you describe. And that’s why, while in my previous role as Tarsier Product Manager, the first thing I did was to build one.

I actually have a presentation on exactly this subject which I plan on posting to Slideshare.net sometime soon.

First thing, I agree with you that detection time is not the “only” parameter for a FOD detection system and reducing FOD risk. It is obviously an important factor, together with probability of detection, False Alarm rate, visual interrogation capabilities, System availability in different weather conditions, etc. All these are all covered in the FAA’s AC, so nothing new here.

However, I have a technical comment regarding your analysis. Detection time is not only a sensitivity issue. That is, you claim that decreasing detection time would decrease detection sensitivity (or increase false alarm rate). I claim that detection time is also a function of the coverage area. The smaller the coverage area is, the faster it can be scanned. That is, a system using more sensors to scan the runway, can perform faster without impacting the detection performance. Thus, in your example, you should assume (in some cases) that both slow and fast systems can detect with probability of 95%.

“Thus, in your example, you should assume (in some cases) that both slow and fast systems can detect with probability of 95%.”

Yes, I agree with you. I do try and make it clear that it is quite possible for systems based on different technologies to have a shorter detection time and a better probability of detection, when I say :

“So, does this mean that systems with a longer detection time are better?
No, I wish it were that simple, but it’s impossible to compare the trade-off between probability of detection and detection time across systems that are based on different technologies. It’s quite possible that a system based on one form of technology will have a better probability of detection, and a shorter detection time.”

Maybe I should have been clearer and stated that as well as the technology used, the configuration is also a factor. The aim of the article was not really to describe every factor that can be used to determine the absolute value of the key parameters, its aim was to demonstrate that detection time should not be the main design aim. Is having a short detection time good? absolutely yes, and if you can achieve this while maintaining a high probability of detection (by increasing the number of sensors, increasing radar power output, adding lights to camera based systems etc) then it is a very worthwhile pursuit.

The Risk Calculator spreadsheet (and now also a web page) takes this into account by allowing detection time and probability of detection to be entered as independent variables.

“In summary, I completely agree with you i.e. that any FOD detection system has to have a database with the parameters that you describe. And that’s why, while in my previous role as Tarsier Product Manager, the first thing I did was to build one.

I actually have a presentation on exactly this subject which I plan on posting to Slideshare.net sometime soon.”

Did you ever post this presentation or do you have anymore information on this topic please?

I think the detection time is rather important in enhancing the profitability of an airport. Assuming the four systems approved by FAA have the same FOD detection capability, a shorter detection time will allow the airport management to increase the frequency of aircraft take offs and landings, thereby enhancing the appeal and ultimately the profitability of that airport.

The bottom line of an FOD Detection System is to ensure the airport is safe for aircraft take offs and landings but a shorter detection will definitely help in enhancing the profitability of that airport