This is actually by design. I see your point about the green + being misleading, though. The best solution would be to have a modifier key which when held down would cause the drag to replace the current queue, with addition occurring otherwise.