Engineering, Science, and Society

Did my little piece for good science and engineering today

2016 December 16

by Daniel Lakeland

I had a call from a client who is dealing with a quality control issue in the repair and re-installation of a large number of windows at a hotel. My recommendation came down to the only way to ensure consistency through time in installation was to randomly test as things were repaired. In particular, no you can't say that there were 200 windows installed in the last few months, but only 20 of them were eligible for testing on the day you went out, and of the 20, 7 were randomly selected for testing, and all of the 7 passed, and then extrapolate to what the condition of the full 1000 windows will be at the end of the project a year from now.

Unlike a random number generator whose consistency is mathematically guaranteed by algorithm design and testing, in the real world windows are installed through time, in different temperatures, different amounts of rain, with materials coming from different suppliers, with crews that get sick, or have vacations, with materials that are delivered and sit out in the sun, or don't sit out the sun, with windows installed on different faces of the building getting different amounts of heat, or wind, different amounts of dust on the surfaces that the sealant has to adhere to, different crews who do or do not know about how to properly clean the surfaces...

Unlike a computer RNG there is no future guarantee of consistency in real-world conditions.

I also argued for choosing how much resources to allocate to testing by balancing the cost of testing against the expected risk cost of having to re-repair windows done wrong, rather than say relying on some formal NHST power calculation or the like (these questions typically come in the form "how many windows would we have to test to have 95% confidence?" which isn't a complete question, but even if you flesh it out so that it has an answer, it's still the wrong way to think about the problem.). I find it makes sense to reorienting the question towards something like: "what is the best number of windows to test to keep our total cost low including the cost of both testing, and of re-repairing things when we later discover they were done wrong?"