In ES 5.x, numerics use a new datastructure (BKD tree). This allows better compression, faster numeric operations and lower memory usage... but it is not ideal for "point lookups" like a term query. E.g. it is designed for numeric style operations like ranges, but not single value lookups.

If that field is really an identifier of some type, where you mainly do point lookups, you should re-map it as a keyword. Keyword fields are optimized for single-term lookups like you're doing.

To dive a bit more into technicals, the BKD datastructure doesn't support sorted iteration, so it has to collect all matches, sort the array and then return an iterator to that sorted array (paraphrasing). That process happens during the build_scorer step, which is why it is slow and you see it showing up in the profiler output. This process isn't bad when dealing with numeric ranges, but can get expensive when asking for a bunch of individual points.