Traffic shaper limit throughput not stable

I've been trying to setup some basic limiters for testing before I move onto more advanced QoS. I am using pfSense in a VM on ESXi.

I have simple limiters of 200Mbit and 10Mbit. When I do a download test without limiters its 100% 230Mbit stable throughput. When I turn on the limiter of 200Mbit down, I get approx. 100 to 170 Mbit. Additionally, the download spikes up and down sometimes below 70Mbit.

My processor or memory isn't constrained. Does anyone have any idea why the limiter isn't limiting to a stable/constant 200Mbit? I've tried all kinds of things except physical hardware and I think this could be a virtual/FreeBSD thing.

Is it my understanding of what the limter are? I'm not expecting any schedulers or algorithms on the pipe, so should I be expecting a hard limit or a scheduler of some sort to help with bufferbloat and therefore not 100% limit.

I notice when I run 'ipfw pipe list', the pipe is FIFO, but I am seeing a sched using WF2Q+, when I execute 'ipfw sched list'.

Is there a queue depth set? I don't know about limiters, but shapers default to 50 packets, which is not enough and causes dropped packets at high rates which reduces throughput.

Thank you Harvy66, this appears to have resolved the issue. I do however need to set this to 400 to get constant throughput, what would you expect this to be and can it be bad having too high a number?

Edit: so basically bufferbloat if the number is too high. So I need to understand the settings at this point and possibly move onto Codel or something else.

If you're worried about bufferbloat, you could look into setting up fq_Codel on your limiters. Otherwise, a good rule of thumb is to aim for about 10ms-50ms of buffer. How to calculate this. The average backbone packet size is about 600bytes. 600 bytes time 8 = 4800bits.

(Bandwidth*10ms)/4800bits = queue size

(200Mbit*10ms)/4800bits = 416 packets.

I guess your 400 packets is about right.

Keep an eye on this if you don't want to manually setup fq_Codel. https://redmine.pfsense.org/issues/6620 They're planning on adding it to the UI at some point. I assume it may be as easy as a check box on limiters.

It worked perfectly for me, including giving me top-notch VOIP while bandwidth is fully saturated with upload or download traffic, and perfectly dividing the bandwidth between multiple users.

The most salient sections from that post follow.

Fixed Limiters

These are the more commonly discussed limits from what I've seen. Fixed limits are used when a network operator wants to permit only a very specific upper bound of bandwidth to be used by an individual device, no matter what. Use cases might include public WiFi scenarios, where a network operator wants to discourage people from relying on it being a top quality connection to avoid attracting people who camp out on their network consuming maximum bandwidth all day.

Create a new Pipe
Name: Upload
Bandwidth: 256kbps
Schedule: None
Mask: Source addresses (no need to type a number into either of the numeric field boxes in this section)
Create a new Pipe
Name: Download
Bandwidth: 1mbps
Schedule: None
Mask: Destination addresses (no need to type a number into either of the numeric field boxes in this section)

The configuration in the applicable LAN-side firewall rule:

Advanced > In/Out: Upload / Download

Flexible Limits

These are less common, and I didn't realize it was actually possible to do this with pfSense until I got Steve's feedback (forum discussions allude to it, but I haven't seen a correct config fully described anywhere yet). The purpose of flexible limits is to allow pfSense to enforce a total cap on user traffic and to dynamically manage the connections based on real network conditions – allocating more bandwidth per device when the network is quiet and less bandwidth per device when many clients are chatting at the same time. In my case, I've seen users report pleasantly usable network conditions consistently even while the network link was 100% saturated -- this is a very good tool to have in your kit for overloaded Internet uplinks (in one case, I've got a large download capacity but a very small upload capacity, and the users would completely overload the upload, resulting in poor conditions for everyone until I implemented this flexible limiter).