In a group of aged Croatian tourists (mostly zagrebački, if this information could help), there was a little jam at the entrance of a bar and one of them shouted "neka, neka!" to another tourist, menacing him (friendly) with the hand. What on earth told him to do or not do do?

Neka! or neka, neka!, literally 'let go' or 'let', as an exclamation is used to denote 1) speaker's reluctant acceptance of the situation: 'I'll let you go away with it this time', 'let them have it for now', or 2) a praise to someone else that he did it well: Neka si ga udario, zaslužio je! ('You did well to punch him, he deserved it'). So, only the actors could possibly know what had instigated it.

Duya, to me, "Neka, neka!" in this case seems similar to colloquial Bosnian "Ohani, bolan". I personally have used "Neka, ba, smiri" and the like before, with the meaning "Relax / Take it easy / Calm down" (with a slightly menacing overtone).