The present paper discusses a nonsupervised multicategory problem in terms of nonparametric learning. An algorithm for seeking modes of an unknown multidimensional probability density function (pdf) is considered by employing a hypercubic window function. The convergence proof of the algorithm is also presented. The discriminant function for multicategory problems is constructed by using the estimates of the modes of the multimodal pdf. An application of the mode estimation algorithm to nonparametric signal detection is described. The analytical result shows that our machine nearly converges to the optimal machine without supervision.