This paper presents a fast top-down visual attention method to downsize the search space of template matching. Such a method first generates patterns representing the local structures, and then calculates the pattern distributions representing the template and its surroundings. From here two separate operations are performed: the "pattern weight" is first introduced, which describes how well a certain pattern is correlated to the template, and then weights of all patterns are calculated for later reference. This is the "off-line" operation, and in comparison the "on-line" operation only calculates the pattern of each pixel, whose weights can be indexed conveniently from the off-line results. With all pixels' pattern weights calculated, the weight image is ready, from which we can extract the region of interest for subsequent matching. Experiments showed that our method obtained at least 6.21 times speed-ups over the state-of-the-art methods with little or no loss in performance.