I am training a random forest classifier in a setting with such a low sample size that I cannot afford setting aside validation and test sets. I train the hyperparameters via cross-validation and utilize the out-of-bag error of the final model (retrained on the whole dataset with the hyperparameters learned via CV) as an estimator for the generalization error when the model is applied to new data. Is this a sound procedure or is the OOB error overly optimistic, since it is based on the data which has also been used for training the hyperparameters?

$\begingroup$Thank you. I guess this is a typical example of data leakage. I compared the results of the described approach with nested cross validation, using leave-one-out CV in the outer loop and the original CV in the inner loop, which should provide an unbiased estimate. The results are considerably worse using this approach.$\endgroup$
– MarchhareApr 11 at 7:27