How can I get them without making thousands of API calls?
And also without downloading the WHOLE Wikidata database, which is probably huge.
Is there a file that contains just the Wikivoyage article name to Wikidata identifier mapping?

I just put LIMIT clause. I'm not sure I understand your question correctly: do you want to get all Wikiwoyage articles about Barcelona (in English, Spanish etc.), or do you want to get Wikidata entities for all Wikivoyage articles (e. g. for all en.wikivoyage.org articles)? Try this query.
– Stanislav KralinJun 29 '17 at 7:34

@StanislavKralin: Sorry I did not remember my own question, I should have read it carefully. Indeed I ws asking for the identifiers of all articles, so your query is perfect. Could you please post it as an answer? Thanks!
– Nicolas RaoulJun 29 '17 at 7:45