Unfortunately expressions (in Elasticsearch) are hard coded to only expose a single value (for a multi valued field, the first value is exposed). However, you can access all values of a field through painless. I suggest you try that. The performance should be roughly the same.

Thanks @rjernst
Since the order of items is important for me (I'm putting a vector there), seems like multi value field is not the right container type for vectors. I'm gonna look into how to write ES plugins. In the mean time I will just put each element of the vector as its own field in ES document which is really ugly...