Python, Seaborn: Logarithmic Swarmplot has unexpected gaps in the swarm

Let's look at a swarmplot, made with Python 3.5 and Seaborn on some data (which is stored in a pandas dataframe df with column lables stored in another class. This does not matter for now, just look at the plot):

Now the data is more readable if plotted in log scale on the y-axis because it goes over some decades.
So let's change the scaling to logarithmic:

ax.set_yscale("log")
ax.set_ylim(bottom = 5*10**-10)

Well I have a problem with the gaps in the swarms. I guess they are there because they have been there when the plot is created with a linear axis in mind and the dots should not overlap there. But now they look kind of strange and there is enough space to from 4 equal looking swarms.
My question is: How can I force Seaborn to recalculate the position of the dots to create better looking swarms?