May I suggest a conventional alternative? I you get rid of the very dark and bright regions first, you may be able to simply rely on the most frequent value of the hue component calculated from the histogram.

Mind that the borders of the numbers will never be absolutely exact, since colours are similar in the surrounding.

Furthrmore, you could select the maximum blob only (according to size) to suppress remaining small blobs outside.