On Wed, Mar 25, 2009 at 02:30:49PM +0100, Joakim Tjernlund wrote:
> >>From 1c2f23b1f37f4818c0fd0217b93eb38ab6564840 Mon Sep 17 00:00:00 2001> From: Joakim Tjernlund <Joakim.Tjernlund@transmode.se>> Date: Tue, 24 Mar 2009 10:19:27 +0100> Subject: [PATCH] ucc_geth: Move freeing of TX packets to NAPI context.> Also increase NAPI weight somewhat.> This will make the system alot more responsive while> ping flooding the ucc_geth ethernet interaface.
Some time ago I've tried a similar thing for this driver, but during
tcp (or udp I don't quite remember) netperf tests I was getting tx
watchdog timeouts after ~2-5 minutes of work. I was testing with a
gigabit and 100 Mbit link, with 100 Mbit link the issue was not
reproducible.
Though, I recalling I was doing a bit more than your patch: I was
also clearing the TX events in the ucce register before calling
ucc_geth_tx, that way I was trying to avoid stale interrupts. That
helped to increase an overall performance (not only responsiveness),
but as I said my approach didn't pass the tests.
I don't really think that your patch may cause this, but can you
try netperf w/ this patch applied anyway? And see if it really
doesn't cause any issues under stress?
Thanks,

From: Joakim Tjernlund <Joakim.Tjernlund@transmode.se>
Date: Wed, 25 Mar 2009 16:16:24 +0100
> UCC_GETH_DEV_WEIGHT needs to be a bit bigger than the number of RX> HW buffers avaliable, otherwise one won't be able to drain the whole> queue in one go. Changing weight to something bigger made a big> difference.
You're not supposed to "drain the whole queue in one go", that
is not the goal of the weight value.
The goal of the weight value is that it is low enough such that
other devices also scheduled for NAPI on the current processor
can get some fair time to process packets too.

On Thu, Mar 26, 2009 at 1:51 AM, Joakim Tjernlund
<Joakim.Tjernlund@transmode.se> wrote:
> Anton Vorontsov <avorontsov@ru.mvista.com> wrote on 25/03/2009 15:25:40:>> On Wed, Mar 25, 2009 at 02:30:49PM +0100, Joakim Tjernlund wrote:>> > >>From 1c2f23b1f37f4818c0fd0217b93eb38ab6564840 Mon Sep 17 00:00:00> 2001>> > From: Joakim Tjernlund <Joakim.Tjernlund@transmode.se>>> > Date: Tue, 24 Mar 2009 10:19:27 +0100>> > Subject: [PATCH] ucc_geth: Move freeing of TX packets to NAPI context.>> > Also increase NAPI weight somewhat.>> > This will make the system alot more responsive while>> > ping flooding the ucc_geth ethernet interaface.>>>> Some time ago I've tried a similar thing for this driver, but during>> tcp (or udp I don't quite remember) netperf tests I was getting tx>> watchdog timeouts after ~2-5 minutes of work. I was testing with a>> gigabit and 100 Mbit link, with 100 Mbit link the issue was not>> reproducible.>>>> Though, I recalling I was doing a bit more than your patch: I was>> also clearing the TX events in the ucce register before calling>> ucc_geth_tx, that way I was trying to avoid stale interrupts. That>> helped to increase an overall performance (not only responsiveness),>> but as I said my approach didn't pass the tests.>>>> I don't really think that your patch may cause this, but can you>> try netperf w/ this patch applied anyway? And see if it really>> doesn't cause any issues under stress?>> Does the line(in ucc_geth_tx()) look OK to you:> if ((bd == ugeth->txBd[txQ]) && (netif_queue_stopped(dev) == 0))> break;>> Sure does look fishy to me.
There are two cases when txBd=ConfBd: the BD ring is full or empty.
The condition used here ensures that it is the empty case. Because in
hard_start_xmit, the queue will be stopped when the BD ring is full.
Maybe some comment is needed here.
- Leo

pku.leo@gmail.com wrote on 30/03/2009 10:34:47:
> > On Thu, Mar 26, 2009 at 1:51 AM, Joakim Tjernlund> <Joakim.Tjernlund@transmode.se> wrote:> > Anton Vorontsov <avorontsov@ru.mvista.com> wrote on 25/03/2009
15:25:40:
> >> On Wed, Mar 25, 2009 at 02:30:49PM +0100, Joakim Tjernlund wrote:> >> > >>From 1c2f23b1f37f4818c0fd0217b93eb38ab6564840 Mon Sep 17 00:00:00> > 2001> >> > From: Joakim Tjernlund <Joakim.Tjernlund@transmode.se>> >> > Date: Tue, 24 Mar 2009 10:19:27 +0100> >> > Subject: [PATCH] ucc_geth: Move freeing of TX packets to NAPI
context.
> >> > Also increase NAPI weight somewhat.> >> > This will make the system alot more responsive while> >> > ping flooding the ucc_geth ethernet interaface.> >>> >> Some time ago I've tried a similar thing for this driver, but during> >> tcp (or udp I don't quite remember) netperf tests I was getting tx> >> watchdog timeouts after ~2-5 minutes of work. I was testing with a> >> gigabit and 100 Mbit link, with 100 Mbit link the issue was not> >> reproducible.> >>> >> Though, I recalling I was doing a bit more than your patch: I was> >> also clearing the TX events in the ucce register before calling> >> ucc_geth_tx, that way I was trying to avoid stale interrupts. That> >> helped to increase an overall performance (not only responsiveness),> >> but as I said my approach didn't pass the tests.> >>> >> I don't really think that your patch may cause this, but can you> >> try netperf w/ this patch applied anyway? And see if it really> >> doesn't cause any issues under stress?> >> > Does the line(in ucc_geth_tx()) look OK to you:> > if ((bd == ugeth->txBd[txQ]) && (netif_queue_stopped(dev) ==
0))
> > break;> >> > Sure does look fishy to me.> > There are two cases when txBd=ConfBd: the BD ring is full or empty.> The condition used here ensures that it is the empty case. Because in> hard_start_xmit, the queue will be stopped when the BD ring is full.> Maybe some comment is needed here.
But how do you know that the queue hasn't been stopped by someone else
than
the driver?
If it is stopped by higher layers, the if stmt will fail.
Jocke

On Mon, Mar 30, 2009 at 5:21 PM, Joakim Tjernlund
<Joakim.Tjernlund@transmode.se> wrote:
> pku.leo@gmail.com wrote on 30/03/2009 10:34:47:>>>> On Thu, Mar 26, 2009 at 1:51 AM, Joakim Tjernlund>> <Joakim.Tjernlund@transmode.se> wrote:>> > Anton Vorontsov <avorontsov@ru.mvista.com> wrote on 25/03/2009> 15:25:40:>> >> On Wed, Mar 25, 2009 at 02:30:49PM +0100, Joakim Tjernlund wrote:>> >> > >>From 1c2f23b1f37f4818c0fd0217b93eb38ab6564840 Mon Sep 17 00:00:00>> > 2001>> >> > From: Joakim Tjernlund <Joakim.Tjernlund@transmode.se>>> >> > Date: Tue, 24 Mar 2009 10:19:27 +0100>> >> > Subject: [PATCH] ucc_geth: Move freeing of TX packets to NAPI> context.>> >> > Also increase NAPI weight somewhat.>> >> > This will make the system alot more responsive while>> >> > ping flooding the ucc_geth ethernet interaface.>> >>>> >> Some time ago I've tried a similar thing for this driver, but during>> >> tcp (or udp I don't quite remember) netperf tests I was getting tx>> >> watchdog timeouts after ~2-5 minutes of work. I was testing with a>> >> gigabit and 100 Mbit link, with 100 Mbit link the issue was not>> >> reproducible.>> >>>> >> Though, I recalling I was doing a bit more than your patch: I was>> >> also clearing the TX events in the ucce register before calling>> >> ucc_geth_tx, that way I was trying to avoid stale interrupts. That>> >> helped to increase an overall performance (not only responsiveness),>> >> but as I said my approach didn't pass the tests.>> >>>> >> I don't really think that your patch may cause this, but can you>> >> try netperf w/ this patch applied anyway? And see if it really>> >> doesn't cause any issues under stress?>> >>> > Does the line(in ucc_geth_tx()) look OK to you:>> > if ((bd == ugeth->txBd[txQ]) && (netif_queue_stopped(dev) ==> 0))>> > break;>> >>> > Sure does look fishy to me.>>>> There are two cases when txBd=ConfBd: the BD ring is full or empty.>> The condition used here ensures that it is the empty case. Because in>> hard_start_xmit, the queue will be stopped when the BD ring is full.>> Maybe some comment is needed here.>> But how do you know that the queue hasn't been stopped by someone else> than> the driver?> If it is stopped by higher layers, the if stmt will fail.
It looks like from existing code that only the driver can legally stop
the queue. I'm not 100% sure though. Correct me if I'm wrong.
- Leo

On Mon, Mar 30, 2009 at 6:01 PM, Joakim Tjernlund
<Joakim.Tjernlund@transmode.se> wrote:
> pku.leo@gmail.com wrote on 30/03/2009 11:36:36:>>>> On Mon, Mar 30, 2009 at 5:21 PM, Joakim Tjernlund>> <Joakim.Tjernlund@transmode.se> wrote:>> > pku.leo@gmail.com wrote on 30/03/2009 10:34:47:>> >>>> >> On Thu, Mar 26, 2009 at 1:51 AM, Joakim Tjernlund>> >> <Joakim.Tjernlund@transmode.se> wrote:>> >> > Anton Vorontsov <avorontsov@ru.mvista.com> wrote on 25/03/2009>> > 15:25:40:>> >> >> On Wed, Mar 25, 2009 at 02:30:49PM +0100, Joakim Tjernlund wrote:>> >> >> > >>From 1c2f23b1f37f4818c0fd0217b93eb38ab6564840 Mon Sep 17> 00:00:00>> >> > 2001>> >> >> > From: Joakim Tjernlund <Joakim.Tjernlund@transmode.se>>> >> >> > Date: Tue, 24 Mar 2009 10:19:27 +0100>> >> >> > Subject: [PATCH] ucc_geth: Move freeing of TX packets to NAPI>> > context.>> >> >> > Also increase NAPI weight somewhat.>> >> >> > This will make the system alot more responsive while>> >> >> > ping flooding the ucc_geth ethernet interaface.>> >> >>>> >> >> Some time ago I've tried a similar thing for this driver, but> during>> >> >> tcp (or udp I don't quite remember) netperf tests I was getting tx>> >> >> watchdog timeouts after ~2-5 minutes of work. I was testing with a>> >> >> gigabit and 100 Mbit link, with 100 Mbit link the issue was not>> >> >> reproducible.>> >> >>>> >> >> Though, I recalling I was doing a bit more than your patch: I was>> >> >> also clearing the TX events in the ucce register before calling>> >> >> ucc_geth_tx, that way I was trying to avoid stale interrupts. That>> >> >> helped to increase an overall performance (not only> responsiveness),>> >> >> but as I said my approach didn't pass the tests.>> >> >>>> >> >> I don't really think that your patch may cause this, but can you>> >> >> try netperf w/ this patch applied anyway? And see if it really>> >> >> doesn't cause any issues under stress?>> >> >>> >> > Does the line(in ucc_geth_tx()) look OK to you:>> >> > if ((bd == ugeth->txBd[txQ]) && (netif_queue_stopped(dev) ==>> > 0))>> >> > break;>> >> >>> >> > Sure does look fishy to me.>> >>>> >> There are two cases when txBd=ConfBd: the BD ring is full or empty.>> >> The condition used here ensures that it is the empty case. Because> in>> >> hard_start_xmit, the queue will be stopped when the BD ring is full.>> >> Maybe some comment is needed here.>> >>> > But how do you know that the queue hasn't been stopped by someone else>> > than>> > the driver?>> > If it is stopped by higher layers, the if stmt will fail.>>>> It looks like from existing code that only the driver can legally stop>> the queue. I'm not 100% sure though. Correct me if I'm wrong.>> I don't know. But the question you should ask is: Does the networking> code promise this now and for the future?
Right. But it's beyond my knowledge to answer this question. If not,
adding a device specific flag is not very costing.
Hi Dave,
Can we assume that the netif_stop_queue() and netif_wake_queue() are
only used by the netdev driver? And the queue state will not be
changed by other part of the networking subsystem?
- Leo

From: Joakim Tjernlund <Joakim.Tjernlund@transmode.se>
Date: Mon, 30 Mar 2009 12:01:33 +0200
> I don't know. But the question you should ask is: Does the networking> code promise this now and for the future? If not, you should> fix the driver not to relay on netif_queue_stopped() here.
Stop this nonsense talk.
If the driver can't be the one and only controller of the TX queue
state, everything would essentially explode.
The fact is, it does.