Machine learning competitions and holdout sets

Guys what’s your view on the below? The widely used holdout method involves splitting an underlying data set into two separate sets. But most of us often forget something important when applying the classic holdout method. The holdout score gives an accurate estimate of the true performance of the model on the underlying distribution from which the data were drawn. However, this is only the case when the model is independent of the holdout data! In contrast, in a competition the model generally incorporates previously observed feedback from the holdout set. Competitors work adaptively and iteratively with the feedback they receive. An improved score for one submission might convince the team to tweak their current approach, while a lower score might cause them to try out a different strategy. But…